aboutsummaryrefslogtreecommitdiff

srht-contrib

srht-contrib is a small Python service that polls SourceHut activity, normalizes it into one internal event model, stores it in SQLite, and exposes a contribution-calendar JSON API that an iOS app can render directly.

The current V1 is intentionally narrow and production-oriented:

  • FastAPI JSON API only
  • SQLite-backed persistence
  • polling-based ingestion
  • complete todo.sr.ht ingestion path
  • practical git.sr.ht commit ingestion with automatic repository discovery
  • public read-only contribution endpoints plus API-key protection for mutating/admin routes
  • Alembic-managed schema migrations

What It Does

The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs recent incremental polling, backfills the most recent 365 days for newly requested actors, and prunes older activity from storage.

Example use cases:

  • render a GitHub-style contribution grid in an iOS app
  • show total score and streak stats for a SourceHut user
  • poll recent activity on a schedule or trigger polling manually

Architecture Overview

The code is split into small, testable layers:

  • src/srht_contrib/config.py: environment-driven settings and event weights
  • src/srht_contrib/db.py: SQLAlchemy engine/session setup and app-scoped DB access
  • src/srht_contrib/models.py: ORM models for normalized events, sync state, aliases, tracked actors, and backfill state
  • src/srht_contrib/services/srht_client.py: generic SourceHut GraphQL client with error handling and simple retries
  • src/srht_contrib/services/todo.py: todo.sr.ht ingestion and normalization
  • src/srht_contrib/services/git.py: git.sr.ht tracked-repository commit ingestion
  • src/srht_contrib/services/aggregator.py: per-day aggregation and streak/stat calculations
  • src/srht_contrib/jobs/poller.py: repeated-safe polling and idempotent persistence
  • src/srht_contrib/api/: FastAPI routes, auth dependencies, and repository management
  • alembic/: schema migration environment and versioned migrations

Supported sr.ht Services

Implemented

  • todo.sr.ht
  • git.sr.ht

Current normalized event types:

  • ticket_created
  • ticket_comment
  • ticket_closed
  • commit

todo.sr.ht uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. git.sr.ht polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories.

Canonical Event Model

All ingestion services normalize external activity into this shape:

  • service
  • event_type
  • actor
  • repo_name
  • resource_id
  • external_uid
  • occurred_at
  • weight
  • raw_payload_json

The database enforces uniqueness on (service, external_uid) so polling is safe to repeat.

Tracked git repositories are persisted in the tracked_repositories table and stored in canonical ~owner/repo form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories.

Configuration

Environment variables:

  • API_KEY: required header token for mutating/admin routes via X-API-Key
  • ENABLE_SCHEDULER: defaults to false; enables in-process polling when set to true
  • SRHT_TOKEN: bearer token for SourceHut GraphQL
  • TODO_SRHT_ENDPOINT: defaults to https://todo.sr.ht/query
  • GIT_SRHT_ENDPOINT: defaults to https://git.sr.ht/query
  • DATABASE_URL: defaults to sqlite:///./srht_contrib.db
  • DEFAULT_ACTOR: actor used by the scheduled poll job
  • POLL_INTERVAL_SECONDS: scheduler interval in seconds
  • DISCOVERY_BATCH_SIZE: max number of due actors to process per scheduler pass
  • INDEXED_ACTOR_REPOLL_SECONDS: how long to wait before re-polling an already indexed actor
  • DISCOVERY_ERROR_BACKOFF_SECONDS: base retry delay after a failed scheduled poll
  • DISCOVERY_ERROR_BACKOFF_MAX_SECONDS: maximum retry delay after repeated scheduled poll failures
  • ACTOR_ALIASES_JSON: optional JSON object for actor/email/display-name alias mapping
  • GIT_TRACKED_REPOSITORIES: optional JSON array of repository names or owner/repo strings to union into git polling

Example .env:

API_KEY=replace-me
ENABLE_SCHEDULER=false
SRHT_TOKEN=replace-me
TODO_SRHT_ENDPOINT=https://todo.sr.ht/query
GIT_SRHT_ENDPOINT=https://git.sr.ht/query
DATABASE_URL=sqlite:///./srht_contrib.db
DEFAULT_ACTOR=~your-user
POLL_INTERVAL_SECONDS=300
DISCOVERY_BATCH_SIZE=20
INDEXED_ACTOR_REPOLL_SECONDS=21600
DISCOVERY_ERROR_BACKOFF_SECONDS=3600
DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600
ACTOR_ALIASES_JSON={"~your-user":["[email protected]","Your Name"]}
GIT_TRACKED_REPOSITORIES=["your-repo","~your-user/your-site"]

Local Run Instructions

1. Create a virtual environment and install dependencies

Using uv:

uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"

Using pip:

python3.12 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

2. Configure environment

cp .env.example .env

Set at least:

  • API_KEY
  • SRHT_TOKEN
  • DEFAULT_ACTOR
  • GIT_TRACKED_REPOSITORIES if you want to force-include extra repositories beyond the actor's owned repos

3. Run database migrations

alembic upgrade head

4. Run the API

uvicorn srht_contrib.main:app --reload

Manual Polling

Manual polling is exposed as an API endpoint:

curl -X POST "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \
  -H "X-API-Key: replace-me"

Example response:

{
  "actor": "~your-user",
  "inserted_events": 3,
  "services": ["todo", "git"]
}

Scheduled polling only runs when ENABLE_SCHEDULER=true. The scheduler seeds DEFAULT_ACTOR as an initial known actor, runs one poll immediately at startup, and public contribution reads register additional actors for later background polling and one-year backfill.

The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to DISCOVERY_BATCH_SIZE per run, performs a fast first indexing pass before bounded one-year backfill work, then reschedules indexed actors with INDEXED_ACTOR_REPOLL_SECONDS and failed actors with capped backoff based on DISCOVERY_ERROR_BACKOFF_SECONDS.

Clients can explicitly signal that a public contribution read is for the signed-in user's own graph by sending prioritize_self=true on the read request. That temporarily boosts the actor to the front of the due queue for the next indexing pass, then clears the boost after the poll completes.

Bulk Enqueue Without Immediate Indexing

To durably queue a large username list without polling it immediately:

srht-enqueue-actors srht_usernames.txt --stagger-seconds 60

This command stores usernames in tracked_actors, marks them queued, and spaces out their first eligible poll time. With --stagger-seconds 60, a file of 15,771 users will be spread across roughly 11 days before becoming due for first poll.

For git.sr.ht, owned repositories are auto-discovered for the actor. GIT_TRACKED_REPOSITORIES can still be used to union in extra repositories. Entries may be either:

  • "Hutch" for a repository owned by DEFAULT_ACTOR
  • "~your-user/your-site" for an explicit owner/repository pair

API Endpoints

Health

curl "http://127.0.0.1:8000/health"

Response:

{"status":"ok"}

Contribution Calendar by Year

curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026"

Contribution Calendar by Date Range

curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-01-01&to=2026-03-30"

Example response:

{
  "actor": "~your-user",
  "from": "2026-01-01",
  "to": "2026-03-30",
  "is_indexed": true,
  "last_polled_at": "2026-04-11T18:05:00Z",
  "indexing_state": "indexed",
  "is_recent_window_backfilled": true,
  "recent_backfill_state": "completed",
  "recent_backfill_completed_at": "2026-04-11T18:02:00Z",
  "days": [
    {"date": "2026-03-28", "count": 3, "score": 3.5},
    {"date": "2026-03-29", "count": 0, "score": 0.0},
    {"date": "2026-03-30", "count": 7, "score": 8.25}
  ]
}

Contribution Stats

curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?year=2026"

Example response:

{
  "actor": "~your-user",
  "from": "2026-01-01",
  "to": "2026-12-31",
  "is_indexed": true,
  "last_polled_at": "2026-04-11T18:05:00Z",
  "indexing_state": "indexed",
  "is_recent_window_backfilled": true,
  "recent_backfill_state": "completed",
  "recent_backfill_completed_at": "2026-04-11T18:02:00Z",
  "total_events": 42,
  "total_score": 37.5,
  "active_days": 18,
  "longest_streak": 5,
  "current_streak": 2
}

Tracked Repositories

List tracked repositories:

curl "http://127.0.0.1:8000/api/repositories?actor=~your-user" \
  -H "X-API-Key: replace-me"

Create a tracked repository:

curl -X POST "http://127.0.0.1:8000/api/repositories" \
  -H "X-API-Key: replace-me" \
  -H "Content-Type: application/json" \
  -d '{"actor":"~your-user","repo_name":"your-repo"}'

Get, update, and delete a tracked repository:

curl "http://127.0.0.1:8000/api/repositories/1" \
  -H "X-API-Key: replace-me"

curl -X PATCH "http://127.0.0.1:8000/api/repositories/1" \
  -H "X-API-Key: replace-me" \
  -H "Content-Type: application/json" \
  -d '{"repo_name":"~your-user/your-site"}'

curl -X DELETE "http://127.0.0.1:8000/api/repositories/1" \
  -H "X-API-Key: replace-me"

Event Weighting

Weights live in src/srht_contrib/config.py so they are easy to tune without touching aggregation code:

  • commit: 1.0
  • ticket_created: 1.0
  • ticket_comment: 0.5
  • ticket_closed: 0.75
  • build_started: 0.25
  • build_passed: 0.25

Testing

Run the test suite with:

pytest

Covered areas:

  • health endpoint
  • public read-only contribution endpoints
  • API key enforcement for mutating/admin routes
  • calendar aggregation
  • zero-filled ranges
  • stats calculations
  • invalid date handling
  • idempotent ingestion
  • todo feed fallback traversal
  • repository CRUD and normalization
  • SourceHut error mapping
  • git commit alias normalization
  • Alembic upgrade path

SourceHut Schema Assumptions

The SourceHut-specific assumptions are isolated to the service modules:

  • src/srht_contrib/services/todo.py uses the authenticated events(cursor) feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery.
  • src/srht_contrib/services/git.py discovers owned repositories for an actor, polls each repository log(cursor), and attributes commits through the configured alias map.
  • the service only retains the most recent 365 days of contribution history and periodically prunes older rows

Known Limitations

  • git.sr.ht polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API
  • scheduled polling runs in-process, so it is not a distributed scheduler
  • newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs
  • the service is intentionally limited to a rolling one-year history window; older activity is not retained
  • one-year backfill still runs in bounded batches, so a newly requested actor may take multiple scheduler passes before their visible graph is fully filled in
  • alias management is config-driven; there is no alias CRUD API yet
  • current deployment model is trusted-operator V1, not a public multi-tenant service

Deployment Notes

  • Add a .dockerignore when building container images so local secrets and SQLite files are never sent to the build context.
  • For production, prefer exposing the service behind a reverse proxy instead of publishing the application port directly to the internet.
  • Set ENABLE_SCHEDULER=true only for single-instance deployments where this service should own polling.
  1. Add alias-management APIs or seed files for stronger actor identity mapping.
  2. Add more SourceHut services such as builds.sr.ht and lists.sr.ht.
  3. Move scheduled polling into an external worker if the deployment grows past a single process.
generated by cgit v1.2.3 (git 2.39.1) at 2026-08-04 10:09:55 +0000