This repository was archived by the owner on Aug 10, 2026. It is now read-only.
fix(agent): reconciler change-detection + self-trigger break + failure backoff — the P0 engine (agent#187, 0.2.5-rc.2) - #188
Merged
Merged
Conversation
…e backoff — the P0 engine (agent#187, 0.2.5-rc.2) The reconciler was the ENGINE of the 2026-07-02 host-wide port-exhaustion outage (team vault Log/2026-07-02-port-exhaustion-incident): when auth broke underneath the daemon, three compounding defects turned the 60s poll into a continuous request storm (15K 'instantiated "uni"' + ~40K tag-schema PUTs; two loops 401'd 859× and 444× with zero widening). This fixes all three. 1. Change-detection memo (no-diff re-instantiation). instantiate()/instantiateThread() now fingerprint the def/thread's instantiation-relevant fields (spawn spec + wants + a thread's agent_status) — EXCLUDING every derived/bookkeeping field (status, pending, usage, session, turn_count, …). An unchanged, already-live agent takes a FAST PATH that skips the channel rebuild + spawn re-register + grant reconcile + "instantiated" log; it STILL re-resolves status so a hub grant-approval flip (pending→enabled — a change with no note edit) propagates via the poll. A real edit (prompt/backend/mode/model/wants) changes the fingerprint and re-instantiates. Schemas are declared ONCE per (origin,vault) per daemon lifetime instead of on every channel rebuild. 2. Self-trigger loop break (diff-before-write). Each instantiate's patchStatus wrote the def note → fired the def-watch EDIT trigger → webhook → reload → instantiate → patchStatus → … maybePatchStatus() now writes ONLY when the resolved status/pending actually differ from what's already on the note, so a status-only echo writes nothing → no updated event → the trigger doesn't fire → the loop is dead. 3. Backoff + circuit breaker (failure loops). A shared Backoff primitive (src/backoff.ts, exponential base→cap + jitter, injectable clock) gates: the runner's loadJobs (runner.ts), the def-vault listing inside loadAll (per-vault breaker — the 401-loop the incident cited), and the agent-def poll (daemon.ts, outer belt). Each widens on consecutive failures, resets on success, and logs when the breaker opens/recovers. Config via PARACHUTE_AGENT_BACKOFF_BASE_MS / _CAP_MS with sane defaults (60s → 30m); no new required config. Tests: change-detection both directions (unchanged→skip, edited/wants-changed→re-instantiate, status-only→skip), status diff-before-write (equal→no PATCH, differ→one PATCH), backoff progression on a fake clock + env parsing, runner breaker skips ticks during a cooldown then recovers, once-per-vault schema guard (dedup + retry-on-failure). Gates: bun run typecheck clean; bun test ./src 1475 pass / 0 fail; biome 0 errors (2 pre-existing warnings). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
…rant approval (agent#187 review fold) Folds the reviewer's nit on PR #188: the safety-critical claim behind the change-detection fast path — that an UNCHANGED def (fast path active) still flips its note status pending→enabled when a grant is approved HUB-SIDE between two loadAll() passes (no note edit) — was traced-as-correct but rested on reasoning, not an assertion. This asserts it (the #184 Approve-button flow): - pass 1: grant pending → status stamped pending, connection listed; - operator approves the grant on the hub (mutate the fake grants client's status, NO def-note edit); - pass 2: def fingerprint unchanged → FAST PATH (asserted: no 2nd ensureChannel / setupAndRegister) → yet refreshDefStatus re-resolves grants → the note is corrected pending→enabled (one new PATCH) and the live agent is now enabled + the connection resolves approved. Follow-up (NOT fixed here): the runner's loadJobs backoff isn't per-vault isolated (VaultJobStore.listAll has no per-transport try/catch) → filed as agent#189. Gates: bun run typecheck clean; bun test ./src 1476 pass / 0 fail; biome 0 errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes agent#187 — the reconciler was the ENGINE of the 2026-07-02 host-wide port-exhaustion outage (team vault
Log/2026-07-02-port-exhaustion-incident). When auth broke underneath the daemon (a rogue loopback listener served an empty JWKS — hub#737), three compounding reconciler defects turned the 60s poll into a continuous request storm: ~15Kinstantiated "uni"lines + ~40K tag-schema PUTs, and two loops that 401'd 859× and 444× with zero widening.This lands all three fixes.
1. Change-detection memo — kills the no-diff re-instantiation churn
instantiate()/instantiateThread()now compute an instantiation fingerprint of only the fields that affect instantiation (the resolved spawn spec + declaredwants:+ a thread'sagent_status), deliberately excluding every derived/bookkeeping field the module or worker stamps back (status,pending,usage,session,turn_count,last_turn_at, …). An unchanged, already-live agent takes a fast path that skips the channel teardown+rebuild, spawn re-register, grant reconcile, and theinstantiatedlog. It still re-resolves status viaresolveStatusWithGrantsso a hub grant-approval flip (pending→enabled— a change that never edits the note) keeps propagating on the poll. A real edit (system prompt / backend / mode / model / wants) changes the fingerprint and re-instantiates. Tag schemas are declared once per (origin, vault) per daemon lifetime instead of on every channel rebuild (VaultTransportguard).2. Self-trigger loop break — diff-before-write
Each instantiate's
patchStatuswrote the def note → fired the def-watch EDIT trigger onagent/definition→ webhook → reload → instantiate →patchStatus→ … self-sustaining.maybePatchStatus()now writes only when the resolvedstatus/pendingactually differ from what's already on the note. A status-only echo writes nothing → noupdatedevent → the trigger never fires → the loop is dead. Verified: a status-only patch schedules no further reload (test).3. Backoff + circuit breaker — failure loops widen instead of hammering
New shared
src/backoff.ts(exponential base→cap with jitter, injectable clock/RNG) gates the three loops the incident named:loadJobs(runner.ts:tick) — the scheduler no longer re-hits a failing vault every 30s.loadAll(per-vault breaker) — the 401-loop the incident cited (859×/444×).daemon.ts) — outer belt for aloadAllthat throws.Each widens on consecutive failures (
60s → 2m → 4m → … → 30m + jitter), resets on success, and logs a line when the breaker opens/recovers so operators see it. Config viaPARACHUTE_AGENT_BACKOFF_BASE_MS/PARACHUTE_AGENT_BACKOFF_CAP_MS, sane defaults, no new required config. (runNowis intentionally not gated — it's an explicit operator action, not the failing loop.)Correctness notes
deregisterByNote,evictOtherLiveByName) so a re-created note re-instantiates fully — audited: boththis.live.deletesites clear it.Tests
60s→2m→4m→…→cap, jitter bounds,ready()/succeed()semantics) + env parsing/clamping.Gates
bun run typecheck— clean (exit 0)bun test ./src— 1475 pass / 0 fail (was 1456; +19 new)biome check .— 0 errors (2 pre-existinganywarnings, unrelated)0.2.5-rc.1 → 0.2.5-rc.2Do not merge — reviewer pass pending (per governance). Daemon NOT restarted (code only; the live daemon is healthy).
🤖 Generated with Claude Code
https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB