fix(components): preserve entry events across deploy scans

[kylebernhardy]

Summary

Closes #1934 and restores the scope.handleEntry() event contract that regressed in #1806 while preserving the new-component fix for #674.

Component deploys pause file watchers while replacing the component tree. When the watcher resumed, chokidar performed a fresh scan and reported every surviving file as add, reported no deleted files, and made unchanged and modified files indistinguishable. Fixing that separately in each consumer would preserve the incompatible watcher behavior and duplicate incomplete state tracking.

Fix

EntryHandler now retains a compact snapshot and compares every replacement watcher generation with the last consumer-observed state:

• unchanged files and directories remain silent
• modified files emit change with their new contents
• new paths emit add / addDir
• removed paths emit unlink / unlinkDir, with descendants emitted before their directory
• file/directory type changes emit the old removal followed by the new addition

Snapshots retain path metadata and SHA-256 file digests, not file contents. Watcher generations and per-path sequences prevent callbacks or slow reads from an obsolete generation from emitting events or polluting the new snapshot. Entries emitted by an interrupted initial scan are carried into the next generation, and ready is generation-scoped and fires only after the scan and all of that generation's reads complete.

Restart-free redeploys now require layered runtime-equivalence evidence:

• the JS loader records every application-local module actually loaded plus its resolution edges, including transitive relative imports, package imports/self-references, pure-ESM export targets, and extensionless resolution candidates
• native addons delegate to Node's .node loader in CJS, ESM, and compartment modes and conservatively make the runtime restart-required
• package metadata is compared after installation; custom/opaque installs, install scripts, bundled node_modules, and unlocked dependency installs fail closed
• extraction and installation are one transaction, retaining the previous component tree until preparation succeeds and restoring it after extraction or install failure

Deploy lifecycle events carry a deployment UUID and owner thread. If the owner worker exits, peers ignore late messages, wait until the worker and its installer process groups are confirmed gone, then reclaim the exact orphaned deployment and resume paused scopes. Overlapping live deployments remain gated.

The consumer-specific redeploy workaround is removed from jsResource; loadEnv, Fastify routes, JS resources, static files, roles, and plugin handleEntry users receive the corrected central events without a new API. Static path ownership is updated incrementally so restart-free static-only deploys serve the replacement content immediately.

Validation

• npm run build
• npm run lint:required
• npm run test:types
• focused unit suites: green (latest lifecycle/Scope/thread tests: 47 passing; broader redeploy-focused run: 134 passing)
• live Harper deploy proof: 22/22 passing across four suites
  • identical deploy remains restart-free
  • changed static asset is served immediately without restart
  • transitive relative/package imports, pure-ESM exports retargeting, and extensionless resolution changes require restart
  • package formatting/key order remains restart-free; changed installed dependencies require restart
  • failed install restores the prior on-disk tree and live resource
  • code/resource deletion cases set restartRequired
• full integration suite previously completed with 1,643 passing, 0 failures, 6 cancelled, and 20 skipped; the nonzero exit was the existing optional Ollama setup incompatibility under local Node 26 (ERR_IMPORT_ATTRIBUTE_MISSING for json/systemSchema.json), with those files unchanged

Review coverage

Author: Codex. Standard cross-model-review/bin/prepush-review.mjs route run repeatedly in exact-commit delta mode with independent Claude, Gemini, Grok, and Harper-domain adjudication. The final exact-head review of f7be8aff1 reports LGTM with no new blocker or major regression.

Review findings fixed and regression-tested include stale watcher generations, suppressed restart requests, post-install package evidence, transitive runtime modules/resolution cache invalidation, pure-ESM and native-addon loading, transactional rollback under a live writer, deploy-aware load timing, static URL ownership, and dead deploy-owner/process-group reclamation.

PR implementation generated by Codex.

[claude]

Reviewed; no blockers found.

[kriszyp]

Requesting changes because I think this is fixing the symptoms at the consumer layer instead of restoring the scope.handleEntry() event contract.

handleEntryWithRedeployRestart is now a second file/deploy state machine for loadEnv and Fastify, while jsResource retains another copy and every other built-in or third-party scope.handleEntry() consumer still receives the lossy stream: every surviving file is replayed as add, deleted files produce no unlink, and unchanged files are indistinguishable from changed ones. That is the backwards-compatibility break we should fix centrally.

I think EntryHandler.pause()/resume() should preserve the previous watcher generation, discard or generation-tag stale pending reads, buffer the resumed initial scan, and diff it at ready:

• new entry → add / addDir
• changed file → change
• removed entry → unlink / unlinkDir
• unchanged entry → no event

That restores the existing API for every consumer, avoids marking a byte-identical redeploy as restart-required, and lets us remove both this helper and the compensating jsResource bookkeeping.

The inline stale-read race is also a concrete correctness blocker in the current approach. The focused helper tests cannot expose it because they do not exercise the real Scope.#deployInFlight restart suppression or EntryHandler.pause()/resume() lifecycle.

The history is split: #822 introduced pause/resume with fresh-add replay, #1540 introduced the atomic directory swap, and #1806 explicitly relies on existing-component watchers to detect what changed. Regardless of attribution, the released behavior is incompatible and #1936 is the point where we should correct course rather than add another compensating API.

We should keep #1806's explicit restart signal for genuinely new components, since they have no live Scope to emit events through; that preserves the #674 actionable 404 behavior.

This is far enough into the wrong abstraction that I would prefer replacing or substantially reworking this PR rather than expanding the helper. I am happy to take over this branch or open a replacement PR, whichever you prefer.

[kylebernhardy]

Kyle's call: please take this one — it's yours.

Your read is right, and I want to be specific about what I confirmed rather than just conceding:

• The stale-read race is real and this helper cannot fix it. EntryHandler.pause() closes the watcher but never cancels #pendingFileReads, so a readFile() started pre-pause still resolves and emits add. That lands after my deploy:start reset, gets recorded as part of the fresh scan, and Scope.requestRestart() swallows it under #deployInFlight. If the deploy deleted that file, the ready diff sees it as present and never requests a restart — a deleted env value or route stays live in the workers. Only EntryHandler knows about the pending-read generation, so no amount of bookkeeping in a consumer closes it.
• The altitude critique lands. Three copies of the same compensation — loadEnv, fastifyRoutes, and jsResource's inline one — is the tell that the contract, not the consumers, is what's broken. Restoring pause()/resume() to diff the resumed scan into real add/change/unlink fixes it once for every handleEntry consumer, and stops flagging a byte-identical redeploy as restart-required, which this PR never even attempted.
• The Sinon objection is simply correct and I should have caught it — the repo convention prohibits new Sinon and I used spies anyway.

The branch is fix/redeploy-env-restart-1934; take it over or open a replacement, whichever suits you. I've stopped work on it, and I've left your two threads open rather than resolving them. If the integration fixture or the loadEnv/fastifyRoutes wiring is useful as a starting point, it's there; if it's cleaner to start fresh, discard it without ceremony.

Two things worth carrying forward, since they're the actual failure modes #1934 hit: a file deleted during a redeploy produces no unlink today, and a file first seen during a post-redeploy scan reaches only the scanning thread. Both should fall out of the diff-at-ready design naturally, but they're the cases I'd want covered.

Separately — your #1918 review is the same shape and is a separate decision still with Kyle, so nothing there is handed over yet.

Comment generated by kAIle (Claude Opus 4.8).

Superseded by the central EntryHandler generation-diff implementation in 857aa7a; the requested architecture and stale-read regression are now addressed.

[kriszyp]

I took over the handed-off branch and replaced the consumer-level workaround with the central course correction discussed above. EntryHandler now diffs the retained pre-deploy snapshot against the resumed watcher generation and emits logical add/change/unlink/addDir/unlinkDir events, while generation guards discard stale reads. The old helper and jsResource-specific deploy bookkeeping are gone, and #674 remains covered by the explicit new-component restart path. The branch has been rebased and force-updated at 857aa7a; the PR body now includes the full validation results and degraded outside-model review status.

[kriszyp]

Cross-model review rerun completed with real outside-model coverage: Claude ✓ / Gemini agy ✗ → direct API ✓. The review found and we fixed suppressed non-watcher restart requests, package metadata changes outside plugin globs, cold-recreation/update identity loss, interrupted-first-scan replay, readiness under listener errors, and static URL-key collisions. Final targeted Claude verification reports no remaining blocker or significant concern. Updated head: 7d8e370.

[kriszyp]

Thanks — this was the right review, and the current head addresses the concrete failures you identified.

• Post-install package evidence: the old installed metadata is captured before extraction and the replacement metadata is captured in prepareApplication() only after installApplication(). package.json is normalized as parsed JSON; installed lockfiles remain exact evidence. Custom commands, install scripts, bundled node_modules, and dependency installs without a lock are treated as opaque and require restart.
• Non-globbed loaded code: rather than hashing the whole extracted tree (which would restart for unused source and generated caches), ApplicationScope/jsLoader now record every transitively loaded application-local module plus relative-resolution candidate state. A changed/missing imported helper or a new shadowing candidate such as foo.js ahead of foo.json requests restart; unused files stay restart-free. Native npm dependencies delegated by the default loader are covered by the post-install package/lock evidence.
• Read failures: redeploy scans retain failed paths from the previous snapshot, exclude them from synthesized removals, resolve ready, and then report the read error. There is deterministic EACCES coverage.
• Listener errors: file-read rejection and consumer exceptions are separate promise branches. An exhaustion-shaped consumer throw is reported as a consumer error and cannot flip the watcher to polling; deferred scan errors are surfaced after readiness commits.
• update() race/readiness: update() now invalidates the outgoing generation synchronously before replacing #component or re-arming readiness. A deferred old-generation read racing a URL-path update is covered and produces the required old-URL unlink plus new-URL add.

The smaller items are also covered: watcher errors are generation-scoped; static URL collisions retain ownership and restore a surviving path; roles ignores directory/unlink events; close rejects a pending readiness latch; type/URL transitions still emit their paired addition if a removal listener throws; and unreadable package evidence fails closed to “changed.” I corrected the stale deletion-test rationale in b5b48e1.

One point is intentionally not converted into retry behavior: EntryHandler snapshots filesystem observation, not successful completion of arbitrary async consumer work. Scope reports async handler rejection and awaits it during initial load, but a later byte-identical deploy does not replay entries as an implicit retry mechanism. If we want durable consumer-operation retries, that needs its own explicit contract rather than depending on cold watcher replay.

The #1849 merge-order note remains valid; that PR is still open and conflicted, so whichever lands second will need to reconcile its restart-gate refactor.

Current head is b5b48e1. Focused regressions cover stale reads, deletions, read failure, listener failure, update races, close readiness, static collisions, roles events, transitive module provenance, resolution shadowing, and post-install metadata equivalence.