fix(indexing): treat shutdown-closed store during backfill as benign; de-flake reindex test

[kriszyp]

Summary

Fixes the ~50% CI flake on integrationTests/server/vector-index-integrity.test.ts ("reindex backfill") and removes misleading error noise during worker shutdown.

Root cause (investigated with instrumentation + local repro)

When restart_service http_workers tears a worker down mid-backfill, runIndexing's range scan/puts throw against the closing store (Database not open). The new worker generation re-runs the backfill via the existing crash-recovery trigger (indexingPID !== process.pid / restartNumber < currentRestartGeneration). During that recovery-reindex window the index legitimately reports isIndexing=true, so searches transiently return 503 INDEX_REBUILDING. The test's step-4 gate waited for a single successful search, then step 5 fired searches with no retry — a transient 503 in that window failed the test.

Instrumented/amplified local runs confirmed the index always recovers (no permanent stuck state); the failure is the transient 503 plus benign-but-misleading teardown logging (Error in indexing + Failed to persist indexing failure state).

Changes

• resources/databases.ts — in runIndexing's outer catch and per-attribute error catch, treat a store closed by shutdown (primaryStore.rootStore?.status === 'closed') as a benign interruption: early-return without the doomed indexingFailed persist, log at debug. Recovery behavior is unchanged (handled by the pid/restartNumber trigger).
• integrationTests/server/vector-index-integrity.test.ts — vectorSearchStable() retries a transient INDEX_REBUILDING (503) up to 30s; used in steps 5/6. A permanently-stuck index (never clears) still surfaces the 503 and fails.
• unitTests/resources/indexRestartNumber.test.js — new test for the shutdown-interruption path (runIndexing resolves; indexingFailed not set).

Where to look / what to verify

• The status === 'closed' early-return is gated so it only skips a persist that could not have succeeded anyway (closed store). On LMDB, dbisDB shares the env with primaryStore, so the prior dbisDB.put already failed there too — no recovery regression. On RocksDB (CI), resetDatabases() never closes the store, so closed only occurs on real worker teardown → recovery via a new generation.

Cross-model review (DLC step 10 — thorough)

Codex + Gemini + Harper domain pass. No open blockers. Two findings surfaced, both confirmed pre-existing / out of scope, not introduced here:

• LMDB same-process resetDatabases() mid-backfill can leave an index stuck — this predates the change (the old persist already failed against the same closed env; recovery there doesn't depend on indexingFailed). LMDB is not the indexing/replication target path. Flagged for awareness, not fixed here.
• Non-last-put unhandled rejection for multi-value attributes — already documented in-code as pre-existing and out of scope.

Generated by an LLM (Claude Opus 4.8).

[gemini-code-assist]

Code Review

This pull request improves the resilience of vector indexing during worker shutdowns by treating closed stores as benign interruptions rather than errors, and introduces a robust vectorSearchStable helper to handle transient indexing states in integration tests. The review feedback suggests minor improvements: simplifying the sleep utility in tests by using an existing import, and preserving error context in debug logs when logging interrupted indexing operations.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[claude]

Reviewed; no blockers found.