feat(server): git polling sync as webhook alternative for non-admin repos - #46
Merged
Conversation
…epos
Webhooks require admin:repo_hook to install, so repos the user can only
clone (not administer) could never auto-sync. Add a polling path that
periodically fetches and re-indexes those repos.
- A single shared scheduler (internal/pollscheduler) drives polling for
all polling repos, enqueueing into the existing bounded jobs queue
(CIX_WORKER_CONCURRENCY) — no new queue, no per-repo goroutines, so a
fleet coming due at once can't stampede the indexer.
- Reuses the clone_repo -> index_repo pipeline verbatim: PAT-authenticated
git fetch (keeps within rate limits, no REST calls) and incremental
tree.Diff reindex. Full reindex only in the same edge cases as webhooks.
- Cadence is measured from the END of the last index run: completion
handlers set next_poll_at = now + interval; the scheduler writes a
provisional floor at enqueue for crash-safety.
- Webhook XOR polling: enabling polling requires webhook_mode='disabled'
(422 otherwise). When auto webhook registration fails (non-admin), the
server auto-falls back to disabled + polling.
- git_repos gains polling_enabled / poll_interval_seconds / next_poll_at
(migration 8, idempotent); next_poll_at is exposed on the GitRepo API.
- New config: CIX_DEFAULT_POLL_INTERVAL (5m), CIX_MIN_POLL_INTERVAL (60s),
CIX_POLL_SCHEDULER_TICK (30s).
- API: polling fields on AddGitRepoRequest + PATCH /projects/{hash}/git-repo.
Tests cover migration idempotency, gitrepos polling methods, the scheduler
tick, the reschedule helper, and the HTTP gating/fallback/PATCH paths.
Docs: new doc/POLLING.md, cross-linked from WEBHOOKS.md + CONFIG_REFERENCE.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- gofmt cmd/cix-server/main.go (pollscheduler/repojobs imports were out of alphabetical order) and the two new test files' struct alignment. - Correct stale "migration 8" references to migration 9 in schema.go and db.go comments (8 = tunnel_config, 9 = git_repos_polling after the rebase). - Rename TestOpenMigratesPreM8DB -> TestOpenMigratesPreM9DB to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds git polling as an alternative to webhooks for keeping server-cloned repos in sync. Webhooks require
admin:repo_hookto install, so repos you can only clone (not administer) could never auto-sync. Polling periodically fetches and re-indexes those repos.What changed
internal/pollscheduler): one background goroutine drives polling for all polling repos, enqueueingclone_repojobs into the existing bounded queue (CIX_WORKER_CONCURRENCY). No new queue, no per-repo goroutines — a fleet coming due at once queues and drains instead of stampeding the indexer.git fetch(stays within rate limits, no REST calls) + incrementaltree.Diffreindex. Full reindex only in the same edge cases as webhooks.next_poll_at = now + interval; the scheduler writes a provisional floor at enqueue for crash-safety.webhook_mode='disabled'(422 otherwise). Whenwebhook_mode=autoregistration fails (non-admin), the server auto-falls back to disabled + polling.git_reposgainspolling_enabled/poll_interval_seconds/next_poll_atvia migration 9 (git_repos_polling), idempotent;next_poll_atexposed on theGitRepoAPI.CIX_DEFAULT_POLL_INTERVAL(5m),CIX_MIN_POLL_INTERVAL(60s),CIX_POLL_SCHEDULER_TICK(30s).AddGitRepoRequest+PATCH /api/v1/projects/{hash}/git-repo.doc/POLLING.md, cross-linked fromWEBHOOKS.md+CONFIG_REFERENCE.md.Reviewer notes
develop; the migration is numbered 9 (8 =tunnel_config). No version reuse.openapi.gen.gowas regenerated from the merged spec (go generate ./internal/httpapi/openapi/), not hand-edited.Test plan
go build ./...andgo vet ./...cleango test ./...green (migration idempotency, gitrepos polling methods, scheduler tick + dedupe, reschedule helper, HTTP gating/fallback/PATCH)next_poll_atadvances from cycle end, push a commit and confirm incremental reindex, verify jobs drain atCIX_WORKER_CONCURRENCY🤖 Generated with Claude Code