Skip to content

feat(server): git polling sync as webhook alternative for non-admin repos - #46

Merged
dvcdsys merged 2 commits into
developfrom
feat/git-polling-sync
May 20, 2026
Merged

feat(server): git polling sync as webhook alternative for non-admin repos#46
dvcdsys merged 2 commits into
developfrom
feat/git-polling-sync

Conversation

@dvcdsys

@dvcdsys dvcdsys commented May 20, 2026

Copy link
Copy Markdown
Owner

Summary

Adds git polling as an alternative to webhooks for keeping server-cloned repos in sync. Webhooks require admin:repo_hook to install, so repos you can only clone (not administer) could never auto-sync. Polling periodically fetches and re-indexes those repos.

What changed

  • Shared scheduler (internal/pollscheduler): one background goroutine drives polling for all polling repos, enqueueing clone_repo jobs into the existing bounded queue (CIX_WORKER_CONCURRENCY). No new queue, no per-repo goroutines — a fleet coming due at once queues and drains instead of stampeding the indexer.
  • Reuses the clone→index pipeline verbatim: PAT-authenticated git fetch (stays within rate limits, no REST calls) + incremental tree.Diff reindex. Full reindex only in the same edge cases as webhooks.
  • Cadence measured from the end of the last index run: completion handlers set next_poll_at = now + interval; the scheduler writes a provisional floor at enqueue for crash-safety.
  • Webhook XOR polling: enabling polling requires webhook_mode='disabled' (422 otherwise). When webhook_mode=auto registration fails (non-admin), the server auto-falls back to disabled + polling.
  • Schema: git_repos gains polling_enabled / poll_interval_seconds / next_poll_at via migration 9 (git_repos_polling), idempotent; next_poll_at exposed on the GitRepo API.
  • Config: CIX_DEFAULT_POLL_INTERVAL (5m), CIX_MIN_POLL_INTERVAL (60s), CIX_POLL_SCHEDULER_TICK (30s).
  • API: polling fields on AddGitRepoRequest + PATCH /api/v1/projects/{hash}/git-repo.
  • Docs: new doc/POLLING.md, cross-linked from WEBHOOKS.md + CONFIG_REFERENCE.md.

Reviewer notes

  • Rebased onto the Managed Tunnels work already on develop; the migration is numbered 9 (8 = tunnel_config). No version reuse.
  • openapi.gen.go was regenerated from the merged spec (go generate ./internal/httpapi/openapi/), not hand-edited.

Test plan

  • go build ./... and go vet ./... clean
  • go test ./... green (migration idempotency, gitrepos polling methods, scheduler tick + dedupe, reschedule helper, HTTP gating/fallback/PATCH)
  • Manual e2e against a real non-admin repo: enable polling, confirm next_poll_at advances from cycle end, push a commit and confirm incremental reindex, verify jobs drain at CIX_WORKER_CONCURRENCY

🤖 Generated with Claude Code

dvcdsys and others added 2 commits May 20, 2026 15:01
…epos

Webhooks require admin:repo_hook to install, so repos the user can only
clone (not administer) could never auto-sync. Add a polling path that
periodically fetches and re-indexes those repos.

- A single shared scheduler (internal/pollscheduler) drives polling for
  all polling repos, enqueueing into the existing bounded jobs queue
  (CIX_WORKER_CONCURRENCY) — no new queue, no per-repo goroutines, so a
  fleet coming due at once can't stampede the indexer.
- Reuses the clone_repo -> index_repo pipeline verbatim: PAT-authenticated
  git fetch (keeps within rate limits, no REST calls) and incremental
  tree.Diff reindex. Full reindex only in the same edge cases as webhooks.
- Cadence is measured from the END of the last index run: completion
  handlers set next_poll_at = now + interval; the scheduler writes a
  provisional floor at enqueue for crash-safety.
- Webhook XOR polling: enabling polling requires webhook_mode='disabled'
  (422 otherwise). When auto webhook registration fails (non-admin), the
  server auto-falls back to disabled + polling.
- git_repos gains polling_enabled / poll_interval_seconds / next_poll_at
  (migration 8, idempotent); next_poll_at is exposed on the GitRepo API.
- New config: CIX_DEFAULT_POLL_INTERVAL (5m), CIX_MIN_POLL_INTERVAL (60s),
  CIX_POLL_SCHEDULER_TICK (30s).
- API: polling fields on AddGitRepoRequest + PATCH /projects/{hash}/git-repo.

Tests cover migration idempotency, gitrepos polling methods, the scheduler
tick, the reschedule helper, and the HTTP gating/fallback/PATCH paths.
Docs: new doc/POLLING.md, cross-linked from WEBHOOKS.md + CONFIG_REFERENCE.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- gofmt cmd/cix-server/main.go (pollscheduler/repojobs imports were out of
  alphabetical order) and the two new test files' struct alignment.
- Correct stale "migration 8" references to migration 9 in schema.go and
  db.go comments (8 = tunnel_config, 9 = git_repos_polling after the rebase).
- Rename TestOpenMigratesPreM8DB -> TestOpenMigratesPreM9DB to match.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@dvcdsys
dvcdsys merged commit f07d632 into develop May 20, 2026
1 check passed
@dvcdsys
dvcdsys deleted the feat/git-polling-sync branch May 20, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant