chore(ci): try Swatinem/rust-cache for target/ caching

[pront]

Status: paused

Exploration / prototype. Do not merge as-is — see Pre-merge cleanup below. Findings recorded here for whoever picks this up next; primary follow-up is to evaluate sccache + a Datadog-internal S3 cache instead of GHA cache (see Next steps).

Summary

Replaces the hand-rolled actions/cache step in .github/actions/setup/action.yml with Swatinem/rust-cache@v2.9.1, which caches ~/.cargo AND target/ build artifacts. The current cache step only caches ~/.cargo/registry, so every Vector CI job does a cold cargo build today.

Configuration:

• prefix-key: "v1-vector" — explicit cache version for easy invalidation
• Per-job cache partitioning (Swatinem's default — no shared-key); attempting a single shared key races between jobs and only the lightest job wins
• save-if: true (TEMP) — needs revert to github.ref == 'refs/heads/master' before merging

Measurements

Empirical per-job cache entry sizes from this PR's CI runs:

Job	Size	Notes
check-fmt	486 MB	registry only, no real compile
check-events	486 MB	registry only
build-vrl-playground	709 MB
test-vrl	1.16 GB
check-clippy	1.38 GB
check-rust-docs	1.47 GB
tests (workspace)	2.35 GB
check-generated-docs	2.55 GB	heaviest single job
Sum	~10.1 GB

Cache HIT was confirmed. On a second push to this branch (no Cargo.lock change), check-clippy reported:
Cache hit for: v1-vector-check-clippy-Linux-x64-f996a9aa-c3e9f435
Restored from cache key ... full match: true.

Cargo's first activity after restore was workspace-local code (fakedata, vector-vrl/web-playground) instead of cold-start third-party crates (portable-atomic, critical-section, itoa). Compile time for check-clippy dropped from ~6 min cold → ~2 min warm.

Limitations (the reason this is paused)

The 10 GB-per-repo GHA cache cap is the binding constraint.

Sum of Vector's per-job entries (~10 GB) saturates the entire repo budget on one CI cycle. Between the two pushes on this branch (~40 min apart), LRU eviction killed 2 of 7 entries, including the largest one (check-generated-docs at 2.55 GB). Cache works at the macro level but individual entries are at constant eviction risk.

Pre-PR repo state was already at 91% saturation: 11.56 GB across 62 caches, with 26 redundant Linux-cargo-* entries (~9.8 GB) because the current cache key includes hashFiles('**/Cargo.lock') and every Cargo.lock variant created a new ~400 MB save.

After this PR's runs, repo cache settled to 9.88 GB across 5 caches — Swatinem's pruning and unified per-job entries replaced the noise.

What works in steady state with save-if: refs/heads/master

• Master pushes save the cache (saturated, ~10 GB total).
• PR jobs read master's cache without writing — no per-PR explosion.
• PRs that don't touch Cargo.lock get full hits on the surviving entries; only the touched workspace members recompile.
• 1-2 entries per PR likely miss and rebuild cold (whichever was at the eviction frontier).

What doesn't work

• All caches co-resident: impossible. We always lose 1-2 entries between master pushes.
• Cargo.lock churn on master: every change creates a new ~2-3 GB save per job per OS, evicting older entries. Vector's typical 5-15 lockfile changes/week saturate quickly.
• Cross-OS caching (macOS): not yet covered; would double the budget pressure.
• cross.yml: still cold (Docker-based cross-compile, host cache doesn't reach the container). Confirmed out of scope per discussion — release builds always cold.

Comparison vs status quo

	Status quo (today's master)	This PR
Cache action	actions/cache@v5	Swatinem/rust-cache@v2.9.1
Cached paths	~/.cargo/registry/{index,cache}, ~/.cargo/git/db	All of those + ~/.cargo/bin + target/
Key strategy	${runner.os}-cargo-${hash(Cargo.lock)} — creates one ~400 MB entry per Cargo.lock variant	Per-job, lockfile-hashed, prefix-keyed; Swatinem prunes before save
Active entries observed	62 (26 of which are stale Linux-cargo-* near-duplicates totalling ~9.8 GB)	5 useful per-job entries totalling ~9.88 GB
target/ cached	No — every CI job does a cold cargo build	Yes
Cold PR build time	~15-20 min	unchanged
Warm PR build time	n/a (always cold)	~3-5 min for src-only changes
Goes over 10 GB cap	Yes (currently 91% from stale entries)	Yes (saturated by useful entries, LRU evicts)

Pre-merge cleanup (when resuming)

1. Revert save-if: true to save-if: ${{ github.ref == 'refs/heads/master' }} in .github/actions/setup/action.yml.
2. Revert the src/main.rs source-touch added to trigger the changes filter. (The filter at .github/workflows/changes.yml doesn't include .github/actions/setup/**; possibly worth adding as a follow-up so this hack isn't needed next time.)
3. Manually delete stale Linux-cargo-* entries before merging so the new Swatinem cache has room to seed cleanly:

   gh cache list --repo vectordotdev/vector --key "Linux-cargo-" --limit 100 --json id -q '.[].id' \
     | xargs -I{} gh cache delete {} --repo vectordotdev/vector

4. Restore the original commit history if desired (current branch carries iterative debug commits — squash before merging).

Next steps (not in scope for this PR)

The right long-term answer for Vector's cache budget is to move sccache to a Datadog-internal S3 backend, removing the 10 GB GHA cap entirely. The bucket dd-sccache-storage-us1-ddbuild-io already exists (used by observability-pipelines-worker on Datadog's GitLab CI per their .gitlab-ci.yml), and is documented in the Confluence page 2025-07-02 - S3 Lifecycle Policy Exploration for sccache Storage. The bucket uses a 30-day TTL.

Blocker for Vector to use it from GHA-hosted runners: needs an IAM role + OIDC trust policy in the build-stable AWS account that allows repo:vectordotdev/vector:*. That requires coordination with the build-stable team (Bruce Guenter's group per the Confluence page).

Once that's in place, follow-up PR replaces Swatinem entirely with:

- uses: aws-actions/configure-aws-credentials@<sha>
  with:
    role-to-assume: arn:aws:iam::<acct>:role/vector-sccache-readwrite
    aws-region: us-east-1
- uses: mozilla-actions/sccache-action@<sha>
- run: |
    {
      echo "SCCACHE_BUCKET=dd-sccache-storage-us1-ddbuild-io"
      echo "SCCACHE_REGION=us-east-1"
      echo "SCCACHE_S3_KEY_PREFIX=vector"
      echo "SCCACHE_S3_USE_SSL=true"
      echo "RUSTC_WRAPPER=sccache"
      echo "CC=sccache cc"
      echo "CXX=sccache c++"
    } >> "\$GITHUB_ENV"

One conflict to resolve: RUSTC_WRAPPER=sccache collides with the mold wrapper at .github/actions/setup/action.yml:153-198. Options: drop the mold wrapper in favor of RUSTFLAGS="-C link-arg=-fuse-ld=mold", or use update-alternatives for a system-level linker swap. Either is cleaner than the current wrapper-script approach and removes mold from Cargo's fingerprint inputs.

Change Type

• Non-functional (chore, refactoring, docs)

Is this a breaking change?

• No

Does this PR include user facing changes?

• No. A maintainer will apply the no-changelog label to this PR.

🤖 Generated with Claude Code