feat(relay): inventory unreachable Git objects (relay-only reland) - #2339
Open
thomaspblock wants to merge 1 commit into
Open
feat(relay): inventory unreachable Git objects (relay-only reland)#2339thomaspblock wants to merge 1 commit into
thomaspblock wants to merge 1 commit into
Conversation
…2264) Reland of #2264 (reverted in #2275), trimmed to buzz-relay only per review: - Dropped all Helm chart changes (deployment.yaml, values.yaml, values.schema.json). The worker is configured entirely via BUZZ_GIT_GC_* env vars with safe defaults (disabled); operators can opt in through the chart's existing relay.extraEnv. Dedicated chart knobs can land later as a deploy-only change. - Dropped the Hermit helm 3.16.4 pin (bin/helm) — it existed only to lint the removed chart changes. - Dropped the buzz-db rename (UsageMetricsLeader -> AdvisoryLockLeader). The GC worker reuses the existing try_lock_usage_metrics(lock_key) API with its own advisory lock key; the rename is deferred. The opt-in worker elects one relay replica via a Postgres advisory lock, marks objects reachable from every current repository pointer, and inventories the shared manifests/, packs/, and idx/ namespaces, exporting candidate counts, bytes, scan freshness, and completeness metrics. Bounded pointer count, listed objects, manifest transfer, and wall-clock duration. Dry-run only: nothing is ever deleted, and the worker does not start unless BUZZ_GIT_GC_ENABLED=true. Co-authored-by: Thomas Petersen <thomasp@squareup.com> Signed-off-by: Thomas Petersen <thomasp@squareup.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Relay-only reland of #2264 (reverted in #2275). Per review feedback from Tyler and Wes, this version removes everything outside
crates/buzz-relay:BUZZ_GIT_GC_*env vars with safe defaults (disabled). Operators can enable bounded dry-run scans in staging through the chart's existingrelay.extraEnv— nodeployment.yaml/values.yaml/values.schema.jsonedits. Dedicated chart knobs can land later as a separate deploy-only PR.bin/helmexisted only to lint the removed chart changes.buzz-dbchanges. The previous PR renamedUsageMetricsLeader→AdvisoryLockLeadercosmetically; this version reuses the existingtry_lock_usage_metrics(lock_key)API unchanged with a distinct GC advisory lock key (a code comment marks the deferred rename).Functionality is unchanged from #2264: an opt-in worker elects one relay replica, marks objects reachable from every current repository pointer, and inventories the shared
manifests/,packs/, andidx/namespaces. It exports observed candidate counts, bytes, scan freshness, and completeness while bounding pointer count, listed objects, manifest transfer, and wall-clock duration.This PR deliberately does not delete anything. Candidate status is not yet deletion proof: a later phase must persist continuous unreachability for a grace period and coordinate with concurrent publishers before any sweep can be safe. Disabled by default (
BUZZ_GIT_GC_ENABLED=false) — dead code at runtime until an operator opts in.Test plan
cargo test -p buzz-relay— 723 passed, 27 ignored (the one failure,api::mesh_demo::demo_join_forwarded_arm_round_trips_echo, fails identically onorigin/mainin this environment and is unrelated)cargo clippy -p buzz-relay --all-targets -- -D warnings— cleancargo fmt— cleancrates/buzz-relay,.env.example, anddocs/— no db, helm, or hermit changes