test(record-caching): run the multi-worker suite on real workers + add cross-worker coherence anchors

[kriszyp]

Summary

The #410 record-caching integration test's record-caching [rocksdb] 4-worker suite was actually running single-worker, so the feature's central promise — cross-worker cache invalidation via the shared VerificationTable — was never exercised by CI.

Root cause: record-caching.test.ts passed env: { HARPER_WORKER_COUNT: '4' }. Harper does not read a HARPER_WORKER_COUNT env var (it's a test-file convention that other suites read into config.threads.count, e.g. read-snapshot-consistency.test.ts), and the integration harness forces --THREADS_COUNT=1 on the CLI — so the suite ran on one worker and passed for the wrong reason.

This PR:

• Fixes the existing suite to set worker count via config.threads.count, with a runtime assertMultiWorker guard (via system_information) so it can't silently regress to a no-op again.
• Adds a small shared helper recordCachingWorkers.ts (env→config translation + the guard) documenting the trap in one place.
• Adds two promoted exploratory-QA anchors, both genuine multi-worker:
  • record-caching-cross-worker.test.ts — update / delete / hot-key coherence: once a write is acked, no worker may serve a stale or ghost point-read (200-key churn, delete→recreate, 60-round hot-key monotonic-read).
  • record-caching-scan-coherence.test.ts — the cached point-read path (getEntry) must agree with the uncached scan/query path (getRange, which bypasses the cache) for the same committed record, across update, delete (both orderings), and rapid churn.
• Shared fixture record-caching-coherence/ (indexed name so the scan path can be queried).

Test-only; no product source touched. All three suites pass on real multi-worker; lint + typecheck clean.

Where to look

• record-caching.test.ts diff is the load-bearing correctness change (2 lines) — everything else is added coverage.
• The two new anchors assert post-ack state only (a read is compared against a write we've already seen a 200 for), so a failure is a genuine coherence bug, not a harness race. They don't probe the pre-ack in-flight window.

No linked issue (surfaced by the exploratory-QA loop, tracked internally as F-132).

🤖 Generated by an LLM (Claude Opus 4.8). Test-only change; no cross-model review run (nothing for a correctness/security lens beyond the diff itself).

Update — additional record-cache anchors (2026-07-12)

Extended with three more genuine-multi-worker record-cache anchors (same shared recordCachingWorkers helper, all green on 4-worker RocksDB):

• record-caching-ttl-eviction.test.ts — TTL eviction (async store.remove() sweep, distinct from removeSync) invalidates every worker's cache; no cross-worker ghost after expiry.
• record-caching-blob.test.ts — file-backed Blob attributes reflect replace/delete cross-worker (SHA-256-verified; no stale/dangling/ENOENT read, no post-delete ghost, no orphaned blob file).
• record-caching-invalidate.test.ts — invalidate() / audited-delete write a versioned-null root; every worker's cache reflects it.

Together with the cross-worker + scan-coherence anchors above, this covers #410's point-read cache across update/delete, scan-divergence, TTL-eviction, blob, and invalidate/null — all single-node multi-worker. (A cluster-tier replication anchor exists too but lives in harper-pro, so it's not part of this core PR.)

[gemini-code-assist]

Code Review

This pull request introduces comprehensive integration tests to verify cross-worker record-cache coherence and point-read vs. scan coherence under update/delete churn. It also adds a shared helper to enforce multi-worker execution, ensuring the tests are not run in a vacuous single-worker environment. The review feedback suggests improving the robustness of the environment variable parsing for worker counts and recommends limiting concurrent HTTP requests across several test suites to prevent socket exhaustion and transient network failures in CI environments.

[claude]

Reviewed; no blockers found.

[cb1kenobi]

Nit: failures is shared across S1/S2/S3 and never reset between tests

The suite-level failures array accumulates for the whole suite. Once any test pushes an entry (e.g. S1 fails), every later test's strictEqual(failures.length, 0, ...) also fails and reprints the earlier test's messages — so S2/S3 report a failure they didn't cause, obscuring which test actually found the defect. S2 also breaks on the first round because failures.length > 0 is already true from S1.

Giving each test its own local failures array (or resetting in a beforeEach) keeps a real failure attributed to the test that found it.

—
Generated by Barber AI

[cb1kenobi]

Low: transient cross-worker ghost signals are collected but never asserted

ghostObservations (including this same-instant any404 && any200 case, commented as "the strongest signal") and scanDivergences are only console.error'd — the test can only fail on the post-quiescent stickyGhosts / finalScanLive / confirmedCount checks. A genuine transient cross-worker incoherence during the eviction window would be logged but still pass CI, which narrows this anchor's protective value versus its stated purpose.

If that's intentional (the async sweep propagates per-worker with unavoidable lag, so a brief disagreement window would be flaky to enforce), a one-line comment saying so would stop a future reader assuming it's enforced. If not, assert ghostObservations/scanDivergences are empty (perhaps gated on msSinceConfirm past a margin).

—
Generated by Barber AI