diff options
| author | Christian Cleberg <[email protected]> | 2026-08-02 15:29:32 -0500 |
|---|---|---|
| committer | Christian Cleberg <[email protected]> | 2026-08-02 15:29:32 -0500 |
| commit | e7ec619ff25af162e464c1118571edb28b536a48 (patch) | |
| tree | 4e7820e951ed9c7ef1fc2fffbe36db43d7cca831 | |
| parent | 5d258a14f09f16b40f9007755b07ce123b524dd8 (diff) | |
| download | hutch-stats-main.tar.gz hutch-stats-main.tar.bz2 hutch-stats-main.zip | |
| -rw-r--r-- | API.txt (renamed from API.md) | 0 | ||||
| -rw-r--r-- | LICENSE | 12 | ||||
| -rw-r--r-- | README.md | 367 | ||||
| -rw-r--r-- | README.nfo | 86 |
4 files changed, 98 insertions, 367 deletions
@@ -0,0 +1,12 @@ +Copyright (C) 2026 krazy warez + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/README.md b/README.md deleted file mode 100644 index 6f862e6..0000000 --- a/README.md +++ /dev/null @@ -1,367 +0,0 @@ -# srht-contrib - -`srht-contrib` is a small Python service that polls SourceHut activity, normalizes it into one internal event model, stores it in SQLite, and exposes a contribution-calendar JSON API that an iOS app can render directly. - -The current V1 is intentionally narrow and production-oriented: - -- FastAPI JSON API only -- SQLite-backed persistence -- polling-based ingestion -- complete `todo.sr.ht` ingestion path -- practical `git.sr.ht` commit ingestion with automatic repository discovery -- public read-only contribution endpoints plus API-key protection for mutating/admin routes -- Alembic-managed schema migrations - -## What It Does - -The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs recent incremental polling, backfills the most recent 365 days for newly requested actors, and prunes older activity from storage. - -Example use cases: - -- render a GitHub-style contribution grid in an iOS app -- show total score and streak stats for a SourceHut user -- poll recent activity on a schedule or trigger polling manually - -## Architecture Overview - -The code is split into small, testable layers: - -- `src/srht_contrib/config.py`: environment-driven settings and event weights -- `src/srht_contrib/db.py`: SQLAlchemy engine/session setup and app-scoped DB access -- `src/srht_contrib/models.py`: ORM models for normalized events, sync state, aliases, tracked actors, and backfill state -- `src/srht_contrib/services/srht_client.py`: generic SourceHut GraphQL client with error handling and simple retries -- `src/srht_contrib/services/todo.py`: `todo.sr.ht` ingestion and normalization -- `src/srht_contrib/services/git.py`: `git.sr.ht` tracked-repository commit ingestion -- `src/srht_contrib/services/aggregator.py`: per-day aggregation and streak/stat calculations -- `src/srht_contrib/jobs/poller.py`: repeated-safe polling and idempotent persistence -- `src/srht_contrib/api/`: FastAPI routes, auth dependencies, and repository management -- `alembic/`: schema migration environment and versioned migrations - -## Supported sr.ht Services - -### Implemented - -- `todo.sr.ht` -- `git.sr.ht` - -Current normalized event types: - -- `ticket_created` -- `ticket_comment` -- `ticket_closed` -- `commit` - -`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories. - -## Canonical Event Model - -All ingestion services normalize external activity into this shape: - -- `service` -- `event_type` -- `actor` -- `repo_name` -- `resource_id` -- `external_uid` -- `occurred_at` -- `weight` -- `raw_payload_json` - -The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat. - -Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories. - -## Configuration - -Environment variables: - -- `API_KEY`: required header token for mutating/admin routes via `X-API-Key` -- `ENABLE_SCHEDULER`: defaults to `false`; enables in-process polling when set to `true` -- `SRHT_TOKEN`: bearer token for SourceHut GraphQL -- `TODO_SRHT_ENDPOINT`: defaults to `https://todo.sr.ht/query` -- `GIT_SRHT_ENDPOINT`: defaults to `https://git.sr.ht/query` -- `DATABASE_URL`: defaults to `sqlite:///./srht_contrib.db` -- `DEFAULT_ACTOR`: actor used by the scheduled poll job -- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds -- `DISCOVERY_BATCH_SIZE`: max number of due actors to process per scheduler pass -- `INDEXED_ACTOR_REPOLL_SECONDS`: how long to wait before re-polling an already indexed actor -- `DISCOVERY_ERROR_BACKOFF_SECONDS`: base retry delay after a failed scheduled poll -- `DISCOVERY_ERROR_BACKOFF_MAX_SECONDS`: maximum retry delay after repeated scheduled poll failures -- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping -- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling - -Example `.env`: - -```env -API_KEY=replace-me -ENABLE_SCHEDULER=false -SRHT_TOKEN=replace-me -TODO_SRHT_ENDPOINT=https://todo.sr.ht/query -GIT_SRHT_ENDPOINT=https://git.sr.ht/query -DATABASE_URL=sqlite:///./srht_contrib.db -DEFAULT_ACTOR=~your-user -POLL_INTERVAL_SECONDS=300 -DISCOVERY_BATCH_SIZE=20 -INDEXED_ACTOR_REPOLL_SECONDS=21600 -DISCOVERY_ERROR_BACKOFF_SECONDS=3600 -DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600 -ACTOR_ALIASES_JSON={"~your-user":["[email protected]","Your Name"]} -GIT_TRACKED_REPOSITORIES=["your-repo","~your-user/your-site"] -``` - -## Local Run Instructions - -### 1. Create a virtual environment and install dependencies - -Using `uv`: - -```bash -uv venv -source .venv/bin/activate -uv pip install -e ".[dev]" -``` - -Using `pip`: - -```bash -python3.12 -m venv .venv -source .venv/bin/activate -pip install -e ".[dev]" -``` - -### 2. Configure environment - -```bash -cp .env.example .env -``` - -Set at least: - -- `API_KEY` -- `SRHT_TOKEN` -- `DEFAULT_ACTOR` -- `GIT_TRACKED_REPOSITORIES` if you want to force-include extra repositories beyond the actor's owned repos - -### 3. Run database migrations - -```bash -alembic upgrade head -``` - -### 4. Run the API - -```bash -uvicorn srht_contrib.main:app --reload -``` - -## Manual Polling - -Manual polling is exposed as an API endpoint: - -```bash -curl -X POST "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \ - -H "X-API-Key: replace-me" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "inserted_events": 3, - "services": ["todo", "git"] -} -``` - -Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, runs one poll immediately at startup, and public contribution reads register additional actors for later background polling and one-year backfill. - -The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to `DISCOVERY_BATCH_SIZE` per run, performs a fast first indexing pass before bounded one-year backfill work, then reschedules indexed actors with `INDEXED_ACTOR_REPOLL_SECONDS` and failed actors with capped backoff based on `DISCOVERY_ERROR_BACKOFF_SECONDS`. - -Clients can explicitly signal that a public contribution read is for the signed-in user's own graph by sending `prioritize_self=true` on the read request. That temporarily boosts the actor to the front of the due queue for the next indexing pass, then clears the boost after the poll completes. - -## Bulk Enqueue Without Immediate Indexing - -To durably queue a large username list without polling it immediately: - -```bash -srht-enqueue-actors srht_usernames.txt --stagger-seconds 60 -``` - -This command stores usernames in `tracked_actors`, marks them queued, and spaces out their first eligible poll time. With `--stagger-seconds 60`, a file of 15,771 users will be spread across roughly 11 days before becoming due for first poll. - -For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either: - -- `"Hutch"` for a repository owned by `DEFAULT_ACTOR` -- `"~your-user/your-site"` for an explicit owner/repository pair - -## API Endpoints - -### Health - -```bash -curl "http://127.0.0.1:8000/health" -``` - -Response: - -```json -{"status":"ok"} -``` - -### Contribution Calendar by Year - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026" -``` - -### Contribution Calendar by Date Range - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-01-01&to=2026-03-30" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "from": "2026-01-01", - "to": "2026-03-30", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "days": [ - {"date": "2026-03-28", "count": 3, "score": 3.5}, - {"date": "2026-03-29", "count": 0, "score": 0.0}, - {"date": "2026-03-30", "count": 7, "score": 8.25} - ] -} -``` - -### Contribution Stats - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?year=2026" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "from": "2026-01-01", - "to": "2026-12-31", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "total_events": 42, - "total_score": 37.5, - "active_days": 18, - "longest_streak": 5, - "current_streak": 2 -} -``` - -### Tracked Repositories - -List tracked repositories: - -```bash -curl "http://127.0.0.1:8000/api/repositories?actor=~your-user" \ - -H "X-API-Key: replace-me" -``` - -Create a tracked repository: - -```bash -curl -X POST "http://127.0.0.1:8000/api/repositories" \ - -H "X-API-Key: replace-me" \ - -H "Content-Type: application/json" \ - -d '{"actor":"~your-user","repo_name":"your-repo"}' -``` - -Get, update, and delete a tracked repository: - -```bash -curl "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" - -curl -X PATCH "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" \ - -H "Content-Type: application/json" \ - -d '{"repo_name":"~your-user/your-site"}' - -curl -X DELETE "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" -``` - -## Event Weighting - -Weights live in `src/srht_contrib/config.py` so they are easy to tune without touching aggregation code: - -- `commit`: `1.0` -- `ticket_created`: `1.0` -- `ticket_comment`: `0.5` -- `ticket_closed`: `0.75` -- `build_started`: `0.25` -- `build_passed`: `0.25` - -## Testing - -Run the test suite with: - -```bash -pytest -``` - -Covered areas: - -- health endpoint -- public read-only contribution endpoints -- API key enforcement for mutating/admin routes -- calendar aggregation -- zero-filled ranges -- stats calculations -- invalid date handling -- idempotent ingestion -- todo feed fallback traversal -- repository CRUD and normalization -- SourceHut error mapping -- git commit alias normalization -- Alembic upgrade path - -## SourceHut Schema Assumptions - -The SourceHut-specific assumptions are isolated to the service modules: - -- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery. -- `src/srht_contrib/services/git.py` discovers owned repositories for an actor, polls each repository `log(cursor)`, and attributes commits through the configured alias map. -- the service only retains the most recent 365 days of contribution history and periodically prunes older rows - -## Known Limitations - -- `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API -- scheduled polling runs in-process, so it is not a distributed scheduler -- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs -- the service is intentionally limited to a rolling one-year history window; older activity is not retained -- one-year backfill still runs in bounded batches, so a newly requested actor may take multiple scheduler passes before their visible graph is fully filled in -- alias management is config-driven; there is no alias CRUD API yet -- current deployment model is trusted-operator V1, not a public multi-tenant service - -## Deployment Notes - -- Add a `.dockerignore` when building container images so local secrets and SQLite files are never sent to the build context. -- For production, prefer exposing the service behind a reverse proxy instead of publishing the application port directly to the internet. -- Set `ENABLE_SCHEDULER=true` only for single-instance deployments where this service should own polling. - -## Recommended Next Steps - -1. Add alias-management APIs or seed files for stronger actor identity mapping. -2. Add more SourceHut services such as `builds.sr.ht` and `lists.sr.ht`. -3. Move scheduled polling into an external worker if the deployment grows past a single process. diff --git a/README.nfo b/README.nfo new file mode 100644 index 0000000..788d8ec --- /dev/null +++ b/README.nfo @@ -0,0 +1,86 @@ +┌──────────────────────────────────────────────────────────────┐ +│ H U T C H - S T A T S [ KRZ ] krz.sh │ +└──────────────────────────────────────────────────────────────┘ + +WHAT + srht-contrib. python service. polls sourcehut activity, normalizes + it into one event model, stores it in sqlite, and serves a + contribution-calendar json api an ios app renders directly. + + v1 scope: + - fastapi json api + - sqlite persistence + - polling ingestion + - full todo.sr.ht path + - git.sr.ht commit ingestion with repo auto-discovery + - public read-only endpoints, api-key auth for mutating routes + - alembic migrations + +DOES + collects activity from sr.ht graphql services, aggregates by day, + and returns zero-filled ranges so the client never patches missing + dates. polls incrementally, backfills the last 365 days for new + actors, prunes older activity. + +SERVICES + implemented: todo.sr.ht, git.sr.ht. + event types: ticket_created, ticket_comment, ticket_closed, commit. + +ENV + API_KEY=replace-me + ENABLE_SCHEDULER=false + SRHT_TOKEN=replace-me + TODO_SRHT_ENDPOINT=https://todo.sr.ht/query + GIT_SRHT_ENDPOINT=https://git.sr.ht/query + DATABASE_URL=sqlite:///./srht_contrib.db + DEFAULT_ACTOR=~your-user + POLL_INTERVAL_SECONDS=300 + + more knobs (discovery batch size, repoll and backoff intervals, + actor aliases, tracked repositories) live in config.py. + +RUN + uv venv + source .venv/bin/activate + uv pip install -e ".[dev]" + cp .env.example .env + alembic upgrade head + uvicorn srht_contrib.main:app --reload + +POLL + manual poll: + + curl -X POST \ + "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \ + -H "X-API-Key: replace-me" + + scheduled polling runs only when ENABLE_SCHEDULER=true. it drains + due actors in batches, indexes fast, then backfills a year in + bounded passes. + + bulk queue without immediate indexing: + + srht-enqueue-actors srht_usernames.txt --stagger-seconds 60 + +ENDPOINTS + calendar by year, calendar by date range, stats, and repository + crud. full request and response shapes are in API.txt. + +WEIGHTS + commit 1.0, ticket_created 1.0, ticket_comment 0.5, + ticket_closed 0.75, build_started 0.25, build_passed 0.25. + +TESTS + pytest + +LIMITS + - git polling assumes repos are discoverable via graphql + - the scheduler runs in-process, not distributed + - new actors index asynchronously; the first read may be empty + - rolling one-year window; older activity is not kept + - alias management is config-driven, no crud api yet + - trusted-operator v1, not a public multi-tenant service + +┌──────────────────────────────────────────────────────────────┐ +│ krz.sh │ +└──────────────────────────────────────────────────────────────┘ |
