diff options
| author | Christian Cleberg <[email protected]> | 2026-08-02 15:29:32 -0500 |
|---|---|---|
| committer | Christian Cleberg <[email protected]> | 2026-08-02 15:29:32 -0500 |
| commit | e7ec619ff25af162e464c1118571edb28b536a48 (patch) | |
| tree | 4e7820e951ed9c7ef1fc2fffbe36db43d7cca831 /README.md | |
| parent | 5d258a14f09f16b40f9007755b07ce123b524dd8 (diff) | |
| download | hutch-stats-main.tar.gz hutch-stats-main.tar.bz2 hutch-stats-main.zip | |
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 367 |
1 files changed, 0 insertions, 367 deletions
diff --git a/README.md b/README.md deleted file mode 100644 index 6f862e6..0000000 --- a/README.md +++ /dev/null @@ -1,367 +0,0 @@ -# srht-contrib - -`srht-contrib` is a small Python service that polls SourceHut activity, normalizes it into one internal event model, stores it in SQLite, and exposes a contribution-calendar JSON API that an iOS app can render directly. - -The current V1 is intentionally narrow and production-oriented: - -- FastAPI JSON API only -- SQLite-backed persistence -- polling-based ingestion -- complete `todo.sr.ht` ingestion path -- practical `git.sr.ht` commit ingestion with automatic repository discovery -- public read-only contribution endpoints plus API-key protection for mutating/admin routes -- Alembic-managed schema migrations - -## What It Does - -The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs recent incremental polling, backfills the most recent 365 days for newly requested actors, and prunes older activity from storage. - -Example use cases: - -- render a GitHub-style contribution grid in an iOS app -- show total score and streak stats for a SourceHut user -- poll recent activity on a schedule or trigger polling manually - -## Architecture Overview - -The code is split into small, testable layers: - -- `src/srht_contrib/config.py`: environment-driven settings and event weights -- `src/srht_contrib/db.py`: SQLAlchemy engine/session setup and app-scoped DB access -- `src/srht_contrib/models.py`: ORM models for normalized events, sync state, aliases, tracked actors, and backfill state -- `src/srht_contrib/services/srht_client.py`: generic SourceHut GraphQL client with error handling and simple retries -- `src/srht_contrib/services/todo.py`: `todo.sr.ht` ingestion and normalization -- `src/srht_contrib/services/git.py`: `git.sr.ht` tracked-repository commit ingestion -- `src/srht_contrib/services/aggregator.py`: per-day aggregation and streak/stat calculations -- `src/srht_contrib/jobs/poller.py`: repeated-safe polling and idempotent persistence -- `src/srht_contrib/api/`: FastAPI routes, auth dependencies, and repository management -- `alembic/`: schema migration environment and versioned migrations - -## Supported sr.ht Services - -### Implemented - -- `todo.sr.ht` -- `git.sr.ht` - -Current normalized event types: - -- `ticket_created` -- `ticket_comment` -- `ticket_closed` -- `commit` - -`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories. - -## Canonical Event Model - -All ingestion services normalize external activity into this shape: - -- `service` -- `event_type` -- `actor` -- `repo_name` -- `resource_id` -- `external_uid` -- `occurred_at` -- `weight` -- `raw_payload_json` - -The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat. - -Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories. - -## Configuration - -Environment variables: - -- `API_KEY`: required header token for mutating/admin routes via `X-API-Key` -- `ENABLE_SCHEDULER`: defaults to `false`; enables in-process polling when set to `true` -- `SRHT_TOKEN`: bearer token for SourceHut GraphQL -- `TODO_SRHT_ENDPOINT`: defaults to `https://todo.sr.ht/query` -- `GIT_SRHT_ENDPOINT`: defaults to `https://git.sr.ht/query` -- `DATABASE_URL`: defaults to `sqlite:///./srht_contrib.db` -- `DEFAULT_ACTOR`: actor used by the scheduled poll job -- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds -- `DISCOVERY_BATCH_SIZE`: max number of due actors to process per scheduler pass -- `INDEXED_ACTOR_REPOLL_SECONDS`: how long to wait before re-polling an already indexed actor -- `DISCOVERY_ERROR_BACKOFF_SECONDS`: base retry delay after a failed scheduled poll -- `DISCOVERY_ERROR_BACKOFF_MAX_SECONDS`: maximum retry delay after repeated scheduled poll failures -- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping -- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling - -Example `.env`: - -```env -API_KEY=replace-me -ENABLE_SCHEDULER=false -SRHT_TOKEN=replace-me -TODO_SRHT_ENDPOINT=https://todo.sr.ht/query -GIT_SRHT_ENDPOINT=https://git.sr.ht/query -DATABASE_URL=sqlite:///./srht_contrib.db -DEFAULT_ACTOR=~your-user -POLL_INTERVAL_SECONDS=300 -DISCOVERY_BATCH_SIZE=20 -INDEXED_ACTOR_REPOLL_SECONDS=21600 -DISCOVERY_ERROR_BACKOFF_SECONDS=3600 -DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600 -ACTOR_ALIASES_JSON={"~your-user":["[email protected]","Your Name"]} -GIT_TRACKED_REPOSITORIES=["your-repo","~your-user/your-site"] -``` - -## Local Run Instructions - -### 1. Create a virtual environment and install dependencies - -Using `uv`: - -```bash -uv venv -source .venv/bin/activate -uv pip install -e ".[dev]" -``` - -Using `pip`: - -```bash -python3.12 -m venv .venv -source .venv/bin/activate -pip install -e ".[dev]" -``` - -### 2. Configure environment - -```bash -cp .env.example .env -``` - -Set at least: - -- `API_KEY` -- `SRHT_TOKEN` -- `DEFAULT_ACTOR` -- `GIT_TRACKED_REPOSITORIES` if you want to force-include extra repositories beyond the actor's owned repos - -### 3. Run database migrations - -```bash -alembic upgrade head -``` - -### 4. Run the API - -```bash -uvicorn srht_contrib.main:app --reload -``` - -## Manual Polling - -Manual polling is exposed as an API endpoint: - -```bash -curl -X POST "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \ - -H "X-API-Key: replace-me" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "inserted_events": 3, - "services": ["todo", "git"] -} -``` - -Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, runs one poll immediately at startup, and public contribution reads register additional actors for later background polling and one-year backfill. - -The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to `DISCOVERY_BATCH_SIZE` per run, performs a fast first indexing pass before bounded one-year backfill work, then reschedules indexed actors with `INDEXED_ACTOR_REPOLL_SECONDS` and failed actors with capped backoff based on `DISCOVERY_ERROR_BACKOFF_SECONDS`. - -Clients can explicitly signal that a public contribution read is for the signed-in user's own graph by sending `prioritize_self=true` on the read request. That temporarily boosts the actor to the front of the due queue for the next indexing pass, then clears the boost after the poll completes. - -## Bulk Enqueue Without Immediate Indexing - -To durably queue a large username list without polling it immediately: - -```bash -srht-enqueue-actors srht_usernames.txt --stagger-seconds 60 -``` - -This command stores usernames in `tracked_actors`, marks them queued, and spaces out their first eligible poll time. With `--stagger-seconds 60`, a file of 15,771 users will be spread across roughly 11 days before becoming due for first poll. - -For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either: - -- `"Hutch"` for a repository owned by `DEFAULT_ACTOR` -- `"~your-user/your-site"` for an explicit owner/repository pair - -## API Endpoints - -### Health - -```bash -curl "http://127.0.0.1:8000/health" -``` - -Response: - -```json -{"status":"ok"} -``` - -### Contribution Calendar by Year - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026" -``` - -### Contribution Calendar by Date Range - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-01-01&to=2026-03-30" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "from": "2026-01-01", - "to": "2026-03-30", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "days": [ - {"date": "2026-03-28", "count": 3, "score": 3.5}, - {"date": "2026-03-29", "count": 0, "score": 0.0}, - {"date": "2026-03-30", "count": 7, "score": 8.25} - ] -} -``` - -### Contribution Stats - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?year=2026" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "from": "2026-01-01", - "to": "2026-12-31", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "total_events": 42, - "total_score": 37.5, - "active_days": 18, - "longest_streak": 5, - "current_streak": 2 -} -``` - -### Tracked Repositories - -List tracked repositories: - -```bash -curl "http://127.0.0.1:8000/api/repositories?actor=~your-user" \ - -H "X-API-Key: replace-me" -``` - -Create a tracked repository: - -```bash -curl -X POST "http://127.0.0.1:8000/api/repositories" \ - -H "X-API-Key: replace-me" \ - -H "Content-Type: application/json" \ - -d '{"actor":"~your-user","repo_name":"your-repo"}' -``` - -Get, update, and delete a tracked repository: - -```bash -curl "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" - -curl -X PATCH "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" \ - -H "Content-Type: application/json" \ - -d '{"repo_name":"~your-user/your-site"}' - -curl -X DELETE "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" -``` - -## Event Weighting - -Weights live in `src/srht_contrib/config.py` so they are easy to tune without touching aggregation code: - -- `commit`: `1.0` -- `ticket_created`: `1.0` -- `ticket_comment`: `0.5` -- `ticket_closed`: `0.75` -- `build_started`: `0.25` -- `build_passed`: `0.25` - -## Testing - -Run the test suite with: - -```bash -pytest -``` - -Covered areas: - -- health endpoint -- public read-only contribution endpoints -- API key enforcement for mutating/admin routes -- calendar aggregation -- zero-filled ranges -- stats calculations -- invalid date handling -- idempotent ingestion -- todo feed fallback traversal -- repository CRUD and normalization -- SourceHut error mapping -- git commit alias normalization -- Alembic upgrade path - -## SourceHut Schema Assumptions - -The SourceHut-specific assumptions are isolated to the service modules: - -- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery. -- `src/srht_contrib/services/git.py` discovers owned repositories for an actor, polls each repository `log(cursor)`, and attributes commits through the configured alias map. -- the service only retains the most recent 365 days of contribution history and periodically prunes older rows - -## Known Limitations - -- `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API -- scheduled polling runs in-process, so it is not a distributed scheduler -- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs -- the service is intentionally limited to a rolling one-year history window; older activity is not retained -- one-year backfill still runs in bounded batches, so a newly requested actor may take multiple scheduler passes before their visible graph is fully filled in -- alias management is config-driven; there is no alias CRUD API yet -- current deployment model is trusted-operator V1, not a public multi-tenant service - -## Deployment Notes - -- Add a `.dockerignore` when building container images so local secrets and SQLite files are never sent to the build context. -- For production, prefer exposing the service behind a reverse proxy instead of publishing the application port directly to the internet. -- Set `ENABLE_SCHEDULER=true` only for single-instance deployments where this service should own polling. - -## Recommended Next Steps - -1. Add alias-management APIs or seed files for stronger actor identity mapping. -2. Add more SourceHut services such as `builds.sr.ht` and `lists.sr.ht`. -3. Move scheduled polling into an external worker if the deployment grows past a single process. |
