From e7ec619ff25af162e464c1118571edb28b536a48 Mon Sep 17 00:00:00 2001 From: Christian Cleberg Date: Sun, 2 Aug 2026 15:29:32 -0500 Subject: convert readme to nfo; convert docs to txt; add 0bsd license --- API.md | 560 ------------------------------------------------------------- API.txt | 560 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ LICENSE | 12 ++ README.md | 367 ---------------------------------------- README.nfo | 86 ++++++++++ 5 files changed, 658 insertions(+), 927 deletions(-) delete mode 100644 API.md create mode 100644 API.txt create mode 100644 LICENSE delete mode 100644 README.md create mode 100644 README.nfo diff --git a/API.md b/API.md deleted file mode 100644 index 8c810ab..0000000 --- a/API.md +++ /dev/null @@ -1,560 +0,0 @@ -# API Reference - -`srht-contrib` exposes a small HTTP JSON API for health checks, contribution calendar reads, manual polling, and optional tracked repository management. - -Base URL examples: - -- Local development: `http://127.0.0.1:8000` -- Deployed example: `https://hutch-stats.example.com` - -Content type: - -- Request bodies: `application/json` -- Response bodies: `application/json` - -Authentication: - -- Public endpoints: - - `GET /health` - - `GET /api/contributions/{actor}` - - `GET /api/contributions/{actor}/stats` -- Protected endpoints require `X-API-Key`: - - `POST /api/contributions/poll` - - all `/api/repositories*` - -Example protected header: - -```http -X-API-Key: your-api-key -``` - -## Common Conventions - -Actors: - -- Actors are SourceHut canonical names such as `~your-user`. -- Actor aliases may resolve to the canonical actor through configured alias mappings. - -Dates: - -- Query date format is `YYYY-MM-DD`. -- `year` and `from`/`to` are mutually exclusive on contribution endpoints. - -Contribution ranges: - -- Contribution read endpoints return zero-filled days, so clients do not need to patch missing dates. -- Public contribution reads also register the actor for background indexing. A first lookup may therefore return an empty graph while the scheduler catches up. -- Clients may add `prioritize_self=true` on contribution read endpoints to explicitly request temporary indexing priority for the signed-in user's own graph. -- Incremental indexing and one-year backfill are separate. An actor can be recently indexed before the retained one-year window is fully filled in. -- The service only retains and backfills the most recent 365 days of activity. - -Background polling: - -- When `ENABLE_SCHEDULER=true`, the service runs one poll immediately at startup and then continues polling on `POLL_INTERVAL_SECONDS`. -- The scheduler always seeds `DEFAULT_ACTOR` as a known actor. -- Public contribution reads register additional actors for later background polling. -- The scheduler only processes due actors, up to `DISCOVERY_BATCH_SIZE` per pass. -- Scheduled first indexing skips bounded one-year backfill work, then drains backfill in later scheduled passes so newly requested actors become indexed sooner. -- Manual polling remains available through `POST /api/contributions/poll`. - -Repository names: - -- Repository create/update accepts either: - - shorthand `repo-name` - - canonical `~owner/repo-name` -- Stored repository names are normalized to canonical `~owner/repo-name` form. -- Git polling auto-discovers repositories owned by the actor through SourceHut. -- Tracked repositories are optional force-includes for git polling; they are not required for normal owned-repository discovery. - -## Health - -### `GET /health` - -Returns a basic service health response. - -Auth: - -- Public - -Response `200 OK`: - -```json -{ - "status": "ok" -} -``` - -## Contributions - -### `GET /api/contributions/{actor}` - -Returns a contribution calendar for an actor over a year or explicit date range. - -Auth: - -- Public - -Path parameters: - -- `actor` string: SourceHut actor, for example `~your-user` - -Query parameters: - -- `year` integer, optional -- `from` string `YYYY-MM-DD`, optional -- `to` string `YYYY-MM-DD`, optional -- `prioritize_self` boolean, optional - -Rules: - -- Provide either `year` -- Or provide both `from` and `to` -- Do not combine `year` with `from`/`to` - -Behavior notes: - -- This endpoint resolves aliases to a canonical actor before querying data. -- This endpoint also registers the actor for background indexing and updates the actor's `last_requested_at` timestamp. -- When `prioritize_self=true`, registration also applies a temporary scheduler boost so that due polls for that actor run ahead of the normal due queue. -- The response is always immediate; it does not wait for SourceHut polling to finish. -- One-year backfill runs in bounded background batches and may take multiple scheduler passes to complete. - -Example by year: - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026" -``` - -Example by range: - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-03-01&to=2026-04-15" -``` - -Response `200 OK`: - -```json -{ - "actor": "~your-user", - "from": "2026-03-01", - "to": "2026-04-15", - "is_indexed": false, - "last_polled_at": null, - "indexing_state": "pending", - "is_recent_window_backfilled": false, - "recent_backfill_state": "in_progress", - "recent_backfill_completed_at": null, - "days": [ - { "date": "2026-03-01", "count": 0, "score": 0.0 }, - { "date": "2026-03-02", "count": 3, "score": 2.5 } - ] -} -``` - -Response fields: - -- `actor` string: canonical actor after alias resolution -- `from` string: inclusive start date -- `to` string: inclusive end date -- `is_indexed` boolean: whether the service has already completed at least one successful recent/incremental poll for this actor -- `last_polled_at` string or `null`: most recent successful poll time, if any -- `indexing_state` string: one of `pending`, `indexed`, or `error` -- `is_recent_window_backfilled` boolean: whether the service has finished filling the retained one-year history window -- `recent_backfill_state` string: one of `pending`, `in_progress`, `completed`, or `error` -- `recent_backfill_completed_at` string or `null`: when one-year backfill completed, if it has -- `days` array: - - `date` string `YYYY-MM-DD` - - `count` integer contribution count for the day - - `score` float weighted score for the day - -Indexing state semantics: - -- `pending`: the actor is known but has not completed a successful poll yet -- `indexed`: at least one successful poll has completed for the actor -- `error`: the most recent poll attempt for the actor failed - -Recent backfill semantics: - -- the service only retains and backfills the most recent 365 days of activity -- `pending`: the actor has not started one-year backfill yet -- `in_progress`: one-year backfill is actively progressing in bounded background batches -- `completed`: the retained one-year window is fully backfilled -- `error`: the most recent backfill attempt failed - -Retention notes: - -- activity older than 365 days is not retained -- scheduled polling periodically prunes contribution rows older than the retained window - -Possible errors: - -- `400 Bad Request` for invalid or conflicting date input - -Example `400`: - -```json -{ - "detail": "Provide `year` or both `from` and `to`." -} -``` - -### `GET /api/contributions/{actor}/stats` - -Returns aggregated stats for the same date selection rules as the calendar endpoint. - -Auth: - -- Public - -Path parameters: - -- `actor` string - -Query parameters: - -- `year` integer, optional -- `from` string `YYYY-MM-DD`, optional -- `to` string `YYYY-MM-DD`, optional -- `prioritize_self` boolean, optional - -Behavior notes: - -- This endpoint has the same actor-registration and alias-resolution behavior as the calendar endpoint. -- When `prioritize_self=true`, registration also applies the same temporary scheduler boost as the calendar endpoint. -- This endpoint returns immediately and does not block on SourceHut polling. -- This endpoint also reflects whether the retained one-year history window has been fully backfilled yet. - -Example: - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?from=2026-03-01&to=2026-04-15" -``` - -Response `200 OK`: - -```json -{ - "actor": "~your-user", - "from": "2026-03-01", - "to": "2026-04-15", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "total_events": 126, - "total_score": 116.75, - "active_days": 14, - "longest_streak": 5, - "current_streak": 0 -} -``` - -Response fields: - -- `actor` string -- `from` string -- `to` string -- `is_indexed` boolean -- `last_polled_at` string or `null` -- `indexing_state` string -- `is_recent_window_backfilled` boolean -- `recent_backfill_state` string -- `recent_backfill_completed_at` string or `null` -- `total_events` integer -- `total_score` float -- `active_days` integer -- `longest_streak` integer -- `current_streak` integer - -Possible errors: - -- `400 Bad Request` for invalid or conflicting date input - -### `POST /api/contributions/poll` - -Triggers a manual SourceHut poll for the given actor and stores any newly discovered events. - -Auth: - -- Requires `X-API-Key` - -Query parameters: - -- `actor` string: SourceHut actor to poll - -Example: - -```bash -curl -X POST \ - -H "X-API-Key: your-api-key" \ - "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" -``` - -Response `200 OK`: - -```json -{ - "actor": "~your-user", - "inserted_events": 57, - "services": ["todo", "git"] -} -``` - -Response fields: - -- `actor` string: canonical actor after alias resolution -- `inserted_events` integer: number of newly inserted normalized events -- `services` array of strings: currently `["todo", "git"]` - -Behavior notes: - -- Manual polling also updates the actor's indexing metadata. -- Manual polling also advances one-year backfill by bounded batches for each supported service. -- Git polling auto-discovers the actor's owned repositories and unions in any configured tracked repositories. - -Possible errors: - -- `401 Unauthorized` if the API key is missing or invalid -- `502 Bad Gateway` if polling SourceHut fails - -Example `401`: - -```json -{ - "detail": "Invalid API key." -} -``` - -Example `502`: - -```json -{ - "detail": "SourceHut polling failed: HTTP error from SourceHut: 502" -} -``` - -## Tracked Repositories - -All repository endpoints are protected and require `X-API-Key`. - -Tracked repositories are optional force-includes for git polling. Each repository is associated with an actor and stored in canonical `~owner/repo` form. - -### `GET /api/repositories` - -Lists tracked git repositories. - -Auth: - -- Requires `X-API-Key` - -Query parameters: - -- `actor` string, optional: filter to a canonical actor or alias - -Example: - -```bash -curl \ - -H "X-API-Key: your-api-key" \ - "http://127.0.0.1:8000/api/repositories?actor=~your-user" -``` - -Response `200 OK`: - -```json -[ - { - "id": 1, - "service": "git", - "actor": "~your-user", - "repo_name": "~your-user/your-repo" - } -] -``` - -### `GET /api/repositories/{repository_id}` - -Fetches one tracked repository by numeric ID. - -Auth: - -- Requires `X-API-Key` - -Path parameters: - -- `repository_id` integer - -Example: - -```bash -curl \ - -H "X-API-Key: your-api-key" \ - "http://127.0.0.1:8000/api/repositories/1" -``` - -Response `200 OK`: - -```json -{ - "id": 1, - "service": "git", - "actor": "~your-user", - "repo_name": "~your-user/your-repo" -} -``` - -Possible errors: - -- `404 Not Found` if the repository ID does not exist - -### `POST /api/repositories` - -Creates a tracked repository entry. - -Auth: - -- Requires `X-API-Key` - -Request body: - -```json -{ - "actor": "~your-user", - "repo_name": "your-repo" -} -``` - -Example: - -```bash -curl -X POST \ - -H "X-API-Key: your-api-key" \ - -H "Content-Type: application/json" \ - -d '{"actor":"~your-user","repo_name":"your-repo"}' \ - "http://127.0.0.1:8000/api/repositories" -``` - -Response `201 Created`: - -```json -{ - "id": 1, - "service": "git", - "actor": "~your-user", - "repo_name": "~your-user/your-repo" -} -``` - -Possible errors: - -- `401 Unauthorized` if the API key is missing or invalid -- `409 Conflict` if the normalized repository already exists for that actor -- `422 Unprocessable Content` if `actor` or `repo_name` is blank or malformed - -### `PATCH /api/repositories/{repository_id}` - -Updates an existing tracked repository. - -Auth: - -- Requires `X-API-Key` - -Path parameters: - -- `repository_id` integer - -Request body: - -```json -{ - "actor": "~your-user", - "repo_name": "~your-user/your-other-repo" -} -``` - -Body rules: - -- At least one of `actor` or `repo_name` must be present - -Example: - -```bash -curl -X PATCH \ - -H "X-API-Key: your-api-key" \ - -H "Content-Type: application/json" \ - -d '{"repo_name":"~your-user/your-other-repo"}' \ - "http://127.0.0.1:8000/api/repositories/1" -``` - -Response `200 OK`: - -```json -{ - "id": 1, - "service": "git", - "actor": "~your-user", - "repo_name": "~your-user/your-other-repo" -} -``` - -Possible errors: - -- `404 Not Found` -- `409 Conflict` -- `422 Unprocessable Content` - -### `DELETE /api/repositories/{repository_id}` - -Deletes a tracked repository. - -Auth: - -- Requires `X-API-Key` - -Path parameters: - -- `repository_id` integer - -Example: - -```bash -curl -X DELETE \ - -H "X-API-Key: your-api-key" \ - "http://127.0.0.1:8000/api/repositories/1" -``` - -Response `204 No Content` - -Possible errors: - -- `404 Not Found` - -## Error Summary - -Common status codes: - -- `200 OK` successful read or manual poll -- `201 Created` successful repository creation -- `204 No Content` successful repository deletion -- `400 Bad Request` invalid date parameters -- `401 Unauthorized` missing or invalid API key -- `404 Not Found` missing repository record -- `409 Conflict` duplicate repository after normalization -- `422 Unprocessable Content` invalid repository payload -- `502 Bad Gateway` upstream SourceHut failure during poll - -## OpenAPI - -FastAPI also serves an OpenAPI document at: - -```text -/openapi.json -``` - -If interactive docs are enabled by your deployment, the standard FastAPI docs may also be available at: - -```text -/docs -``` diff --git a/API.txt b/API.txt new file mode 100644 index 0000000..8c810ab --- /dev/null +++ b/API.txt @@ -0,0 +1,560 @@ +# API Reference + +`srht-contrib` exposes a small HTTP JSON API for health checks, contribution calendar reads, manual polling, and optional tracked repository management. + +Base URL examples: + +- Local development: `http://127.0.0.1:8000` +- Deployed example: `https://hutch-stats.example.com` + +Content type: + +- Request bodies: `application/json` +- Response bodies: `application/json` + +Authentication: + +- Public endpoints: + - `GET /health` + - `GET /api/contributions/{actor}` + - `GET /api/contributions/{actor}/stats` +- Protected endpoints require `X-API-Key`: + - `POST /api/contributions/poll` + - all `/api/repositories*` + +Example protected header: + +```http +X-API-Key: your-api-key +``` + +## Common Conventions + +Actors: + +- Actors are SourceHut canonical names such as `~your-user`. +- Actor aliases may resolve to the canonical actor through configured alias mappings. + +Dates: + +- Query date format is `YYYY-MM-DD`. +- `year` and `from`/`to` are mutually exclusive on contribution endpoints. + +Contribution ranges: + +- Contribution read endpoints return zero-filled days, so clients do not need to patch missing dates. +- Public contribution reads also register the actor for background indexing. A first lookup may therefore return an empty graph while the scheduler catches up. +- Clients may add `prioritize_self=true` on contribution read endpoints to explicitly request temporary indexing priority for the signed-in user's own graph. +- Incremental indexing and one-year backfill are separate. An actor can be recently indexed before the retained one-year window is fully filled in. +- The service only retains and backfills the most recent 365 days of activity. + +Background polling: + +- When `ENABLE_SCHEDULER=true`, the service runs one poll immediately at startup and then continues polling on `POLL_INTERVAL_SECONDS`. +- The scheduler always seeds `DEFAULT_ACTOR` as a known actor. +- Public contribution reads register additional actors for later background polling. +- The scheduler only processes due actors, up to `DISCOVERY_BATCH_SIZE` per pass. +- Scheduled first indexing skips bounded one-year backfill work, then drains backfill in later scheduled passes so newly requested actors become indexed sooner. +- Manual polling remains available through `POST /api/contributions/poll`. + +Repository names: + +- Repository create/update accepts either: + - shorthand `repo-name` + - canonical `~owner/repo-name` +- Stored repository names are normalized to canonical `~owner/repo-name` form. +- Git polling auto-discovers repositories owned by the actor through SourceHut. +- Tracked repositories are optional force-includes for git polling; they are not required for normal owned-repository discovery. + +## Health + +### `GET /health` + +Returns a basic service health response. + +Auth: + +- Public + +Response `200 OK`: + +```json +{ + "status": "ok" +} +``` + +## Contributions + +### `GET /api/contributions/{actor}` + +Returns a contribution calendar for an actor over a year or explicit date range. + +Auth: + +- Public + +Path parameters: + +- `actor` string: SourceHut actor, for example `~your-user` + +Query parameters: + +- `year` integer, optional +- `from` string `YYYY-MM-DD`, optional +- `to` string `YYYY-MM-DD`, optional +- `prioritize_self` boolean, optional + +Rules: + +- Provide either `year` +- Or provide both `from` and `to` +- Do not combine `year` with `from`/`to` + +Behavior notes: + +- This endpoint resolves aliases to a canonical actor before querying data. +- This endpoint also registers the actor for background indexing and updates the actor's `last_requested_at` timestamp. +- When `prioritize_self=true`, registration also applies a temporary scheduler boost so that due polls for that actor run ahead of the normal due queue. +- The response is always immediate; it does not wait for SourceHut polling to finish. +- One-year backfill runs in bounded background batches and may take multiple scheduler passes to complete. + +Example by year: + +```bash +curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026" +``` + +Example by range: + +```bash +curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-03-01&to=2026-04-15" +``` + +Response `200 OK`: + +```json +{ + "actor": "~your-user", + "from": "2026-03-01", + "to": "2026-04-15", + "is_indexed": false, + "last_polled_at": null, + "indexing_state": "pending", + "is_recent_window_backfilled": false, + "recent_backfill_state": "in_progress", + "recent_backfill_completed_at": null, + "days": [ + { "date": "2026-03-01", "count": 0, "score": 0.0 }, + { "date": "2026-03-02", "count": 3, "score": 2.5 } + ] +} +``` + +Response fields: + +- `actor` string: canonical actor after alias resolution +- `from` string: inclusive start date +- `to` string: inclusive end date +- `is_indexed` boolean: whether the service has already completed at least one successful recent/incremental poll for this actor +- `last_polled_at` string or `null`: most recent successful poll time, if any +- `indexing_state` string: one of `pending`, `indexed`, or `error` +- `is_recent_window_backfilled` boolean: whether the service has finished filling the retained one-year history window +- `recent_backfill_state` string: one of `pending`, `in_progress`, `completed`, or `error` +- `recent_backfill_completed_at` string or `null`: when one-year backfill completed, if it has +- `days` array: + - `date` string `YYYY-MM-DD` + - `count` integer contribution count for the day + - `score` float weighted score for the day + +Indexing state semantics: + +- `pending`: the actor is known but has not completed a successful poll yet +- `indexed`: at least one successful poll has completed for the actor +- `error`: the most recent poll attempt for the actor failed + +Recent backfill semantics: + +- the service only retains and backfills the most recent 365 days of activity +- `pending`: the actor has not started one-year backfill yet +- `in_progress`: one-year backfill is actively progressing in bounded background batches +- `completed`: the retained one-year window is fully backfilled +- `error`: the most recent backfill attempt failed + +Retention notes: + +- activity older than 365 days is not retained +- scheduled polling periodically prunes contribution rows older than the retained window + +Possible errors: + +- `400 Bad Request` for invalid or conflicting date input + +Example `400`: + +```json +{ + "detail": "Provide `year` or both `from` and `to`." +} +``` + +### `GET /api/contributions/{actor}/stats` + +Returns aggregated stats for the same date selection rules as the calendar endpoint. + +Auth: + +- Public + +Path parameters: + +- `actor` string + +Query parameters: + +- `year` integer, optional +- `from` string `YYYY-MM-DD`, optional +- `to` string `YYYY-MM-DD`, optional +- `prioritize_self` boolean, optional + +Behavior notes: + +- This endpoint has the same actor-registration and alias-resolution behavior as the calendar endpoint. +- When `prioritize_self=true`, registration also applies the same temporary scheduler boost as the calendar endpoint. +- This endpoint returns immediately and does not block on SourceHut polling. +- This endpoint also reflects whether the retained one-year history window has been fully backfilled yet. + +Example: + +```bash +curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?from=2026-03-01&to=2026-04-15" +``` + +Response `200 OK`: + +```json +{ + "actor": "~your-user", + "from": "2026-03-01", + "to": "2026-04-15", + "is_indexed": true, + "last_polled_at": "2026-04-11T18:05:00Z", + "indexing_state": "indexed", + "is_recent_window_backfilled": true, + "recent_backfill_state": "completed", + "recent_backfill_completed_at": "2026-04-11T18:02:00Z", + "total_events": 126, + "total_score": 116.75, + "active_days": 14, + "longest_streak": 5, + "current_streak": 0 +} +``` + +Response fields: + +- `actor` string +- `from` string +- `to` string +- `is_indexed` boolean +- `last_polled_at` string or `null` +- `indexing_state` string +- `is_recent_window_backfilled` boolean +- `recent_backfill_state` string +- `recent_backfill_completed_at` string or `null` +- `total_events` integer +- `total_score` float +- `active_days` integer +- `longest_streak` integer +- `current_streak` integer + +Possible errors: + +- `400 Bad Request` for invalid or conflicting date input + +### `POST /api/contributions/poll` + +Triggers a manual SourceHut poll for the given actor and stores any newly discovered events. + +Auth: + +- Requires `X-API-Key` + +Query parameters: + +- `actor` string: SourceHut actor to poll + +Example: + +```bash +curl -X POST \ + -H "X-API-Key: your-api-key" \ + "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" +``` + +Response `200 OK`: + +```json +{ + "actor": "~your-user", + "inserted_events": 57, + "services": ["todo", "git"] +} +``` + +Response fields: + +- `actor` string: canonical actor after alias resolution +- `inserted_events` integer: number of newly inserted normalized events +- `services` array of strings: currently `["todo", "git"]` + +Behavior notes: + +- Manual polling also updates the actor's indexing metadata. +- Manual polling also advances one-year backfill by bounded batches for each supported service. +- Git polling auto-discovers the actor's owned repositories and unions in any configured tracked repositories. + +Possible errors: + +- `401 Unauthorized` if the API key is missing or invalid +- `502 Bad Gateway` if polling SourceHut fails + +Example `401`: + +```json +{ + "detail": "Invalid API key." +} +``` + +Example `502`: + +```json +{ + "detail": "SourceHut polling failed: HTTP error from SourceHut: 502" +} +``` + +## Tracked Repositories + +All repository endpoints are protected and require `X-API-Key`. + +Tracked repositories are optional force-includes for git polling. Each repository is associated with an actor and stored in canonical `~owner/repo` form. + +### `GET /api/repositories` + +Lists tracked git repositories. + +Auth: + +- Requires `X-API-Key` + +Query parameters: + +- `actor` string, optional: filter to a canonical actor or alias + +Example: + +```bash +curl \ + -H "X-API-Key: your-api-key" \ + "http://127.0.0.1:8000/api/repositories?actor=~your-user" +``` + +Response `200 OK`: + +```json +[ + { + "id": 1, + "service": "git", + "actor": "~your-user", + "repo_name": "~your-user/your-repo" + } +] +``` + +### `GET /api/repositories/{repository_id}` + +Fetches one tracked repository by numeric ID. + +Auth: + +- Requires `X-API-Key` + +Path parameters: + +- `repository_id` integer + +Example: + +```bash +curl \ + -H "X-API-Key: your-api-key" \ + "http://127.0.0.1:8000/api/repositories/1" +``` + +Response `200 OK`: + +```json +{ + "id": 1, + "service": "git", + "actor": "~your-user", + "repo_name": "~your-user/your-repo" +} +``` + +Possible errors: + +- `404 Not Found` if the repository ID does not exist + +### `POST /api/repositories` + +Creates a tracked repository entry. + +Auth: + +- Requires `X-API-Key` + +Request body: + +```json +{ + "actor": "~your-user", + "repo_name": "your-repo" +} +``` + +Example: + +```bash +curl -X POST \ + -H "X-API-Key: your-api-key" \ + -H "Content-Type: application/json" \ + -d '{"actor":"~your-user","repo_name":"your-repo"}' \ + "http://127.0.0.1:8000/api/repositories" +``` + +Response `201 Created`: + +```json +{ + "id": 1, + "service": "git", + "actor": "~your-user", + "repo_name": "~your-user/your-repo" +} +``` + +Possible errors: + +- `401 Unauthorized` if the API key is missing or invalid +- `409 Conflict` if the normalized repository already exists for that actor +- `422 Unprocessable Content` if `actor` or `repo_name` is blank or malformed + +### `PATCH /api/repositories/{repository_id}` + +Updates an existing tracked repository. + +Auth: + +- Requires `X-API-Key` + +Path parameters: + +- `repository_id` integer + +Request body: + +```json +{ + "actor": "~your-user", + "repo_name": "~your-user/your-other-repo" +} +``` + +Body rules: + +- At least one of `actor` or `repo_name` must be present + +Example: + +```bash +curl -X PATCH \ + -H "X-API-Key: your-api-key" \ + -H "Content-Type: application/json" \ + -d '{"repo_name":"~your-user/your-other-repo"}' \ + "http://127.0.0.1:8000/api/repositories/1" +``` + +Response `200 OK`: + +```json +{ + "id": 1, + "service": "git", + "actor": "~your-user", + "repo_name": "~your-user/your-other-repo" +} +``` + +Possible errors: + +- `404 Not Found` +- `409 Conflict` +- `422 Unprocessable Content` + +### `DELETE /api/repositories/{repository_id}` + +Deletes a tracked repository. + +Auth: + +- Requires `X-API-Key` + +Path parameters: + +- `repository_id` integer + +Example: + +```bash +curl -X DELETE \ + -H "X-API-Key: your-api-key" \ + "http://127.0.0.1:8000/api/repositories/1" +``` + +Response `204 No Content` + +Possible errors: + +- `404 Not Found` + +## Error Summary + +Common status codes: + +- `200 OK` successful read or manual poll +- `201 Created` successful repository creation +- `204 No Content` successful repository deletion +- `400 Bad Request` invalid date parameters +- `401 Unauthorized` missing or invalid API key +- `404 Not Found` missing repository record +- `409 Conflict` duplicate repository after normalization +- `422 Unprocessable Content` invalid repository payload +- `502 Bad Gateway` upstream SourceHut failure during poll + +## OpenAPI + +FastAPI also serves an OpenAPI document at: + +```text +/openapi.json +``` + +If interactive docs are enabled by your deployment, the standard FastAPI docs may also be available at: + +```text +/docs +``` diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..419813e --- /dev/null +++ b/LICENSE @@ -0,0 +1,12 @@ +Copyright (C) 2026 krazy warez + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/README.md b/README.md deleted file mode 100644 index 6f862e6..0000000 --- a/README.md +++ /dev/null @@ -1,367 +0,0 @@ -# srht-contrib - -`srht-contrib` is a small Python service that polls SourceHut activity, normalizes it into one internal event model, stores it in SQLite, and exposes a contribution-calendar JSON API that an iOS app can render directly. - -The current V1 is intentionally narrow and production-oriented: - -- FastAPI JSON API only -- SQLite-backed persistence -- polling-based ingestion -- complete `todo.sr.ht` ingestion path -- practical `git.sr.ht` commit ingestion with automatic repository discovery -- public read-only contribution endpoints plus API-key protection for mutating/admin routes -- Alembic-managed schema migrations - -## What It Does - -The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs recent incremental polling, backfills the most recent 365 days for newly requested actors, and prunes older activity from storage. - -Example use cases: - -- render a GitHub-style contribution grid in an iOS app -- show total score and streak stats for a SourceHut user -- poll recent activity on a schedule or trigger polling manually - -## Architecture Overview - -The code is split into small, testable layers: - -- `src/srht_contrib/config.py`: environment-driven settings and event weights -- `src/srht_contrib/db.py`: SQLAlchemy engine/session setup and app-scoped DB access -- `src/srht_contrib/models.py`: ORM models for normalized events, sync state, aliases, tracked actors, and backfill state -- `src/srht_contrib/services/srht_client.py`: generic SourceHut GraphQL client with error handling and simple retries -- `src/srht_contrib/services/todo.py`: `todo.sr.ht` ingestion and normalization -- `src/srht_contrib/services/git.py`: `git.sr.ht` tracked-repository commit ingestion -- `src/srht_contrib/services/aggregator.py`: per-day aggregation and streak/stat calculations -- `src/srht_contrib/jobs/poller.py`: repeated-safe polling and idempotent persistence -- `src/srht_contrib/api/`: FastAPI routes, auth dependencies, and repository management -- `alembic/`: schema migration environment and versioned migrations - -## Supported sr.ht Services - -### Implemented - -- `todo.sr.ht` -- `git.sr.ht` - -Current normalized event types: - -- `ticket_created` -- `ticket_comment` -- `ticket_closed` -- `commit` - -`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories. - -## Canonical Event Model - -All ingestion services normalize external activity into this shape: - -- `service` -- `event_type` -- `actor` -- `repo_name` -- `resource_id` -- `external_uid` -- `occurred_at` -- `weight` -- `raw_payload_json` - -The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat. - -Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories. - -## Configuration - -Environment variables: - -- `API_KEY`: required header token for mutating/admin routes via `X-API-Key` -- `ENABLE_SCHEDULER`: defaults to `false`; enables in-process polling when set to `true` -- `SRHT_TOKEN`: bearer token for SourceHut GraphQL -- `TODO_SRHT_ENDPOINT`: defaults to `https://todo.sr.ht/query` -- `GIT_SRHT_ENDPOINT`: defaults to `https://git.sr.ht/query` -- `DATABASE_URL`: defaults to `sqlite:///./srht_contrib.db` -- `DEFAULT_ACTOR`: actor used by the scheduled poll job -- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds -- `DISCOVERY_BATCH_SIZE`: max number of due actors to process per scheduler pass -- `INDEXED_ACTOR_REPOLL_SECONDS`: how long to wait before re-polling an already indexed actor -- `DISCOVERY_ERROR_BACKOFF_SECONDS`: base retry delay after a failed scheduled poll -- `DISCOVERY_ERROR_BACKOFF_MAX_SECONDS`: maximum retry delay after repeated scheduled poll failures -- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping -- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling - -Example `.env`: - -```env -API_KEY=replace-me -ENABLE_SCHEDULER=false -SRHT_TOKEN=replace-me -TODO_SRHT_ENDPOINT=https://todo.sr.ht/query -GIT_SRHT_ENDPOINT=https://git.sr.ht/query -DATABASE_URL=sqlite:///./srht_contrib.db -DEFAULT_ACTOR=~your-user -POLL_INTERVAL_SECONDS=300 -DISCOVERY_BATCH_SIZE=20 -INDEXED_ACTOR_REPOLL_SECONDS=21600 -DISCOVERY_ERROR_BACKOFF_SECONDS=3600 -DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600 -ACTOR_ALIASES_JSON={"~your-user":["you@example.com","Your Name"]} -GIT_TRACKED_REPOSITORIES=["your-repo","~your-user/your-site"] -``` - -## Local Run Instructions - -### 1. Create a virtual environment and install dependencies - -Using `uv`: - -```bash -uv venv -source .venv/bin/activate -uv pip install -e ".[dev]" -``` - -Using `pip`: - -```bash -python3.12 -m venv .venv -source .venv/bin/activate -pip install -e ".[dev]" -``` - -### 2. Configure environment - -```bash -cp .env.example .env -``` - -Set at least: - -- `API_KEY` -- `SRHT_TOKEN` -- `DEFAULT_ACTOR` -- `GIT_TRACKED_REPOSITORIES` if you want to force-include extra repositories beyond the actor's owned repos - -### 3. Run database migrations - -```bash -alembic upgrade head -``` - -### 4. Run the API - -```bash -uvicorn srht_contrib.main:app --reload -``` - -## Manual Polling - -Manual polling is exposed as an API endpoint: - -```bash -curl -X POST "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \ - -H "X-API-Key: replace-me" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "inserted_events": 3, - "services": ["todo", "git"] -} -``` - -Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, runs one poll immediately at startup, and public contribution reads register additional actors for later background polling and one-year backfill. - -The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to `DISCOVERY_BATCH_SIZE` per run, performs a fast first indexing pass before bounded one-year backfill work, then reschedules indexed actors with `INDEXED_ACTOR_REPOLL_SECONDS` and failed actors with capped backoff based on `DISCOVERY_ERROR_BACKOFF_SECONDS`. - -Clients can explicitly signal that a public contribution read is for the signed-in user's own graph by sending `prioritize_self=true` on the read request. That temporarily boosts the actor to the front of the due queue for the next indexing pass, then clears the boost after the poll completes. - -## Bulk Enqueue Without Immediate Indexing - -To durably queue a large username list without polling it immediately: - -```bash -srht-enqueue-actors srht_usernames.txt --stagger-seconds 60 -``` - -This command stores usernames in `tracked_actors`, marks them queued, and spaces out their first eligible poll time. With `--stagger-seconds 60`, a file of 15,771 users will be spread across roughly 11 days before becoming due for first poll. - -For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either: - -- `"Hutch"` for a repository owned by `DEFAULT_ACTOR` -- `"~your-user/your-site"` for an explicit owner/repository pair - -## API Endpoints - -### Health - -```bash -curl "http://127.0.0.1:8000/health" -``` - -Response: - -```json -{"status":"ok"} -``` - -### Contribution Calendar by Year - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026" -``` - -### Contribution Calendar by Date Range - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-01-01&to=2026-03-30" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "from": "2026-01-01", - "to": "2026-03-30", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "days": [ - {"date": "2026-03-28", "count": 3, "score": 3.5}, - {"date": "2026-03-29", "count": 0, "score": 0.0}, - {"date": "2026-03-30", "count": 7, "score": 8.25} - ] -} -``` - -### Contribution Stats - -```bash -curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?year=2026" -``` - -Example response: - -```json -{ - "actor": "~your-user", - "from": "2026-01-01", - "to": "2026-12-31", - "is_indexed": true, - "last_polled_at": "2026-04-11T18:05:00Z", - "indexing_state": "indexed", - "is_recent_window_backfilled": true, - "recent_backfill_state": "completed", - "recent_backfill_completed_at": "2026-04-11T18:02:00Z", - "total_events": 42, - "total_score": 37.5, - "active_days": 18, - "longest_streak": 5, - "current_streak": 2 -} -``` - -### Tracked Repositories - -List tracked repositories: - -```bash -curl "http://127.0.0.1:8000/api/repositories?actor=~your-user" \ - -H "X-API-Key: replace-me" -``` - -Create a tracked repository: - -```bash -curl -X POST "http://127.0.0.1:8000/api/repositories" \ - -H "X-API-Key: replace-me" \ - -H "Content-Type: application/json" \ - -d '{"actor":"~your-user","repo_name":"your-repo"}' -``` - -Get, update, and delete a tracked repository: - -```bash -curl "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" - -curl -X PATCH "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" \ - -H "Content-Type: application/json" \ - -d '{"repo_name":"~your-user/your-site"}' - -curl -X DELETE "http://127.0.0.1:8000/api/repositories/1" \ - -H "X-API-Key: replace-me" -``` - -## Event Weighting - -Weights live in `src/srht_contrib/config.py` so they are easy to tune without touching aggregation code: - -- `commit`: `1.0` -- `ticket_created`: `1.0` -- `ticket_comment`: `0.5` -- `ticket_closed`: `0.75` -- `build_started`: `0.25` -- `build_passed`: `0.25` - -## Testing - -Run the test suite with: - -```bash -pytest -``` - -Covered areas: - -- health endpoint -- public read-only contribution endpoints -- API key enforcement for mutating/admin routes -- calendar aggregation -- zero-filled ranges -- stats calculations -- invalid date handling -- idempotent ingestion -- todo feed fallback traversal -- repository CRUD and normalization -- SourceHut error mapping -- git commit alias normalization -- Alembic upgrade path - -## SourceHut Schema Assumptions - -The SourceHut-specific assumptions are isolated to the service modules: - -- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery. -- `src/srht_contrib/services/git.py` discovers owned repositories for an actor, polls each repository `log(cursor)`, and attributes commits through the configured alias map. -- the service only retains the most recent 365 days of contribution history and periodically prunes older rows - -## Known Limitations - -- `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API -- scheduled polling runs in-process, so it is not a distributed scheduler -- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs -- the service is intentionally limited to a rolling one-year history window; older activity is not retained -- one-year backfill still runs in bounded batches, so a newly requested actor may take multiple scheduler passes before their visible graph is fully filled in -- alias management is config-driven; there is no alias CRUD API yet -- current deployment model is trusted-operator V1, not a public multi-tenant service - -## Deployment Notes - -- Add a `.dockerignore` when building container images so local secrets and SQLite files are never sent to the build context. -- For production, prefer exposing the service behind a reverse proxy instead of publishing the application port directly to the internet. -- Set `ENABLE_SCHEDULER=true` only for single-instance deployments where this service should own polling. - -## Recommended Next Steps - -1. Add alias-management APIs or seed files for stronger actor identity mapping. -2. Add more SourceHut services such as `builds.sr.ht` and `lists.sr.ht`. -3. Move scheduled polling into an external worker if the deployment grows past a single process. diff --git a/README.nfo b/README.nfo new file mode 100644 index 0000000..788d8ec --- /dev/null +++ b/README.nfo @@ -0,0 +1,86 @@ +┌──────────────────────────────────────────────────────────────┐ +│ H U T C H - S T A T S [ KRZ ] krz.sh │ +└──────────────────────────────────────────────────────────────┘ + +WHAT + srht-contrib. python service. polls sourcehut activity, normalizes + it into one event model, stores it in sqlite, and serves a + contribution-calendar json api an ios app renders directly. + + v1 scope: + - fastapi json api + - sqlite persistence + - polling ingestion + - full todo.sr.ht path + - git.sr.ht commit ingestion with repo auto-discovery + - public read-only endpoints, api-key auth for mutating routes + - alembic migrations + +DOES + collects activity from sr.ht graphql services, aggregates by day, + and returns zero-filled ranges so the client never patches missing + dates. polls incrementally, backfills the last 365 days for new + actors, prunes older activity. + +SERVICES + implemented: todo.sr.ht, git.sr.ht. + event types: ticket_created, ticket_comment, ticket_closed, commit. + +ENV + API_KEY=replace-me + ENABLE_SCHEDULER=false + SRHT_TOKEN=replace-me + TODO_SRHT_ENDPOINT=https://todo.sr.ht/query + GIT_SRHT_ENDPOINT=https://git.sr.ht/query + DATABASE_URL=sqlite:///./srht_contrib.db + DEFAULT_ACTOR=~your-user + POLL_INTERVAL_SECONDS=300 + + more knobs (discovery batch size, repoll and backoff intervals, + actor aliases, tracked repositories) live in config.py. + +RUN + uv venv + source .venv/bin/activate + uv pip install -e ".[dev]" + cp .env.example .env + alembic upgrade head + uvicorn srht_contrib.main:app --reload + +POLL + manual poll: + + curl -X POST \ + "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \ + -H "X-API-Key: replace-me" + + scheduled polling runs only when ENABLE_SCHEDULER=true. it drains + due actors in batches, indexes fast, then backfills a year in + bounded passes. + + bulk queue without immediate indexing: + + srht-enqueue-actors srht_usernames.txt --stagger-seconds 60 + +ENDPOINTS + calendar by year, calendar by date range, stats, and repository + crud. full request and response shapes are in API.txt. + +WEIGHTS + commit 1.0, ticket_created 1.0, ticket_comment 0.5, + ticket_closed 0.75, build_started 0.25, build_passed 0.25. + +TESTS + pytest + +LIMITS + - git polling assumes repos are discoverable via graphql + - the scheduler runs in-process, not distributed + - new actors index asynchronously; the first read may be empty + - rolling one-year window; older activity is not kept + - alias management is config-driven, no crud api yet + - trusted-operator v1, not a public multi-tenant service + +┌──────────────────────────────────────────────────────────────┐ +│ krz.sh │ +└──────────────────────────────────────────────────────────────┘ -- cgit v1.2.3