fix: tighten state-sync defaults (keep=2, blockInterval=100000) - #255
Merged
Conversation
Each state-sync snapshot is currently ~30-45GB. With Keep=6 a snapshot-serving node accumulates ~180GB+ of snapshots on top of chain data and Postgres, which exhausted disk on creatornode2.audius.co (1TB PVC, 100% full) and put Postgres into a checkpoint PANIC / recovery loop. Two snapshots is enough to serve incoming state-syncers (a fresh one plus a fallback) while leaving headroom for chain growth. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
A 100-block interval would create a new snapshot roughly every minute on mainnet, which is far too aggressive — both for disk and CPU. Production already overrides this to 100000 (the values seen in /data/bolt/snapshots_*/height_0024200000 etc.); align the default with the value validators are actually using. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
raymondjacobson
changed the title
5 tasks
raymondjacobson
added a commit
that referenced
this pull request
May 19, 2026
* Reland OPENAUDIO_ prefixed env vars (reverts #197) This re-applies #195 (reverted in #197). The handshake panic that prompted the original revert (`state.AppHash does not match AppHash after replay. Got E3B0..., expected .`) was caused by an unrelated state-sync bug: CometBFT's state.db could end up at height 0 while the app database had block records, producing the mismatch. That class of failure has since been fixed by #253 and related state-sync reliability work, so the env-var change is safe to bring back on top of current main. Changes since #195: - Keep state-sync defaults from #255 (Keep=2, BlockInterval=100000) - Carry forward #224's separate archive bucket DSN (now goes through env.String with AUDIUS_ARCHIVE_STORAGE_DRIVER_URL fallback) - Update pkg/uptime/uptime.go which post-revert added a `mustGetenv("nodeEndpoint")` call — read OPENAUDIO_NODE_ENDPOINT first with legacy fallback, drop the now-unused mustGetenv/getenvWithDefault - CLAUDE.md: keep the validator-node terminology from #264 and add the pkg/env helper list Legacy env vars (nodeEndpoint, delegatePrivateKey, dbUrl, audius_core_root_dir, spOwnerWallet, etc.) continue to work via fallback, so legacy creator / discovery node registrations are unchanged. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * fix(entrypoint): promote legacy env vars before sourcing prod.env On a legacy creator/discovery node, the bundled /env/${NETWORK}.env declares OPENAUDIO_ defaults (e.g. OPENAUDIO_CORE_ROOT_DIR=/data/core). If the operator only set the legacy name with a *different* value (e.g. audius_core_root_dir=/data/bolt), the Go env helper would pick the OPENAUDIO_ default and look at a path that has no chain data — producing the CometBFT `state.AppHash does not match AppHash after replay` panic that originally got #195 reverted. Promote externally-set legacy keys to their OPENAUDIO_ counterparts before source_env_file runs. source_env_file is a no-op for already-set keys, so the legacy value wins; if the operator instead set the OPENAUDIO_ name directly, that already-set value is left alone. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two related default changes in
pkg/core/config/config.goto bring the state-sync defaults in line with what production validators actually use:stateSyncKeep:6→2— each snapshot is currently ~30–45 GB. WithKeep=6a snapshot-serving node accumulates ~180 GB+ of snapshots on top of chain data and Postgres. This exhausted the 1 TB disk oncreatornode2.audius.cotoday (the only prod node withstateSyncServeSnapshots=true), which put Postgres into a checkpoint PANIC / recovery loop until snapshots were manually deleted. Two snapshots is enough to serve an incoming state-syncer (newest) plus one fallback (in case the newest is mid-creation).stateSyncBlockInterval:100→100000— a 100-block interval would create a new snapshot roughly every minute on mainnet, which is far too aggressive both on disk and CPU. Prod already overrides this to100000(the height boundaries seen in/data/bolt/snapshots_*/height_002420****etc.); align the default with what validators actually use.Operators who want different values can still override via the
stateSyncKeep/stateSyncBlockIntervalenv vars.Test plan
creatornode2.audius.co, confirm only the most recent 2 snapshot directories exist under/data/bolt/snapshots_<chainID>/after the next snapshot creation cycle._00000).🤖 Generated with Claude Code