Skip to content
This repository was archived by the owner on Aug 10, 2026. It is now read-only.

fix(agent): reconciler change-detection + self-trigger break + failure backoff — the P0 engine (agent#187, 0.2.5-rc.2) - #188

Merged
unforced merged 2 commits into
mainfrom
ag-unforced-dev
Jul 2, 2026
Merged

fix(agent): reconciler change-detection + self-trigger break + failure backoff — the P0 engine (agent#187, 0.2.5-rc.2)#188
unforced merged 2 commits into
mainfrom
ag-unforced-dev

Conversation

@unforced

@unforced unforced commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Fixes agent#187 — the reconciler was the ENGINE of the 2026-07-02 host-wide port-exhaustion outage (team vault Log/2026-07-02-port-exhaustion-incident). When auth broke underneath the daemon (a rogue loopback listener served an empty JWKS — hub#737), three compounding reconciler defects turned the 60s poll into a continuous request storm: ~15K instantiated "uni" lines + ~40K tag-schema PUTs, and two loops that 401'd 859× and 444× with zero widening.

This lands all three fixes.

1. Change-detection memo — kills the no-diff re-instantiation churn

instantiate() / instantiateThread() now compute an instantiation fingerprint of only the fields that affect instantiation (the resolved spawn spec + declared wants: + a thread's agent_status), deliberately excluding every derived/bookkeeping field the module or worker stamps back (status, pending, usage, session, turn_count, last_turn_at, …). An unchanged, already-live agent takes a fast path that skips the channel teardown+rebuild, spawn re-register, grant reconcile, and the instantiated log. It still re-resolves status via resolveStatusWithGrants so a hub grant-approval flip (pending→enabled — a change that never edits the note) keeps propagating on the poll. A real edit (system prompt / backend / mode / model / wants) changes the fingerprint and re-instantiates. Tag schemas are declared once per (origin, vault) per daemon lifetime instead of on every channel rebuild (VaultTransport guard).

2. Self-trigger loop break — diff-before-write

Each instantiate's patchStatus wrote the def note → fired the def-watch EDIT trigger on agent/definition → webhook → reload → instantiate → patchStatus → … self-sustaining. maybePatchStatus() now writes only when the resolved status/pending actually differ from what's already on the note. A status-only echo writes nothing → no updated event → the trigger never fires → the loop is dead. Verified: a status-only patch schedules no further reload (test).

3. Backoff + circuit breaker — failure loops widen instead of hammering

New shared src/backoff.ts (exponential base→cap with jitter, injectable clock/RNG) gates the three loops the incident named:

  • runner loadJobs (runner.ts:tick) — the scheduler no longer re-hits a failing vault every 30s.
  • def-vault listing inside loadAll (per-vault breaker) — the 401-loop the incident cited (859×/444×).
  • agent-def poll (daemon.ts) — outer belt for a loadAll that throws.

Each widens on consecutive failures (60s → 2m → 4m → … → 30m + jitter), resets on success, and logs a line when the breaker opens/recovers so operators see it. Config via PARACHUTE_AGENT_BACKOFF_BASE_MS / PARACHUTE_AGENT_BACKOFF_CAP_MS, sane defaults, no new required config. (runNow is intentionally not gated — it's an explicit operator action, not the failing loop.)

Correctness notes

  • The fast path preserves grant-approval propagation (status is re-resolved every pass; only the expensive rebuild is skipped).
  • The fingerprint memo is cleared on every teardown path (deregisterByNote, evictOtherLiveByName) so a re-created note re-instantiates fully — audited: both this.live.delete sites clear it.
  • The once-per-vault schema guard marks on success only, so a transient boot failure still retries; the tag-both write floor remains the correctness fallback.

Tests

  • Change-detection both directions: unchanged→skip, content-edited→re-instantiate, wants-changed→re-instantiate, status-only note change→skip.
  • Status diff-before-write: equal→no PATCH (no trigger), differ→exactly one PATCH, second pass stays quiet.
  • Backoff progression on a fake clock (60s→2m→4m→…→cap, jitter bounds, ready()/succeed() semantics) + env parsing/clamping.
  • Runner breaker skips ticks during the cooldown, then recovers.
  • Once-per-vault schema guard: dedup across rebuilds + retry-after-failure.

Gates

  • bun run typecheck — clean (exit 0)
  • bun test ./src1475 pass / 0 fail (was 1456; +19 new)
  • biome check . — 0 errors (2 pre-existing any warnings, unrelated)
  • Version bump 0.2.5-rc.1 → 0.2.5-rc.2

Do not merge — reviewer pass pending (per governance). Daemon NOT restarted (code only; the live daemon is healthy).

🤖 Generated with Claude Code

https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB

…e backoff — the P0 engine (agent#187, 0.2.5-rc.2)

The reconciler was the ENGINE of the 2026-07-02 host-wide port-exhaustion outage
(team vault Log/2026-07-02-port-exhaustion-incident): when auth broke underneath the
daemon, three compounding defects turned the 60s poll into a continuous request storm
(15K 'instantiated "uni"' + ~40K tag-schema PUTs; two loops 401'd 859× and 444× with
zero widening). This fixes all three.

1. Change-detection memo (no-diff re-instantiation). instantiate()/instantiateThread()
   now fingerprint the def/thread's instantiation-relevant fields (spawn spec + wants +
   a thread's agent_status) — EXCLUDING every derived/bookkeeping field (status, pending,
   usage, session, turn_count, …). An unchanged, already-live agent takes a FAST PATH that
   skips the channel rebuild + spawn re-register + grant reconcile + "instantiated" log; it
   STILL re-resolves status so a hub grant-approval flip (pending→enabled — a change with no
   note edit) propagates via the poll. A real edit (prompt/backend/mode/model/wants) changes
   the fingerprint and re-instantiates. Schemas are declared ONCE per (origin,vault) per
   daemon lifetime instead of on every channel rebuild.

2. Self-trigger loop break (diff-before-write). Each instantiate's patchStatus wrote the
   def note → fired the def-watch EDIT trigger → webhook → reload → instantiate → patchStatus
   → … maybePatchStatus() now writes ONLY when the resolved status/pending actually differ
   from what's already on the note, so a status-only echo writes nothing → no updated event →
   the trigger doesn't fire → the loop is dead.

3. Backoff + circuit breaker (failure loops). A shared Backoff primitive (src/backoff.ts,
   exponential base→cap + jitter, injectable clock) gates: the runner's loadJobs
   (runner.ts), the def-vault listing inside loadAll (per-vault breaker — the 401-loop the
   incident cited), and the agent-def poll (daemon.ts, outer belt). Each widens on
   consecutive failures, resets on success, and logs when the breaker opens/recovers.
   Config via PARACHUTE_AGENT_BACKOFF_BASE_MS / _CAP_MS with sane defaults (60s → 30m); no
   new required config.

Tests: change-detection both directions (unchanged→skip, edited/wants-changed→re-instantiate,
status-only→skip), status diff-before-write (equal→no PATCH, differ→one PATCH), backoff
progression on a fake clock + env parsing, runner breaker skips ticks during a cooldown then
recovers, once-per-vault schema guard (dedup + retry-on-failure). Gates: bun run typecheck
clean; bun test ./src 1475 pass / 0 fail; biome 0 errors (2 pre-existing warnings).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
…rant approval (agent#187 review fold)

Folds the reviewer's nit on PR #188: the safety-critical claim behind the
change-detection fast path — that an UNCHANGED def (fast path active) still flips
its note status pending→enabled when a grant is approved HUB-SIDE between two
loadAll() passes (no note edit) — was traced-as-correct but rested on reasoning,
not an assertion. This asserts it (the #184 Approve-button flow):

  - pass 1: grant pending → status stamped pending, connection listed;
  - operator approves the grant on the hub (mutate the fake grants client's status,
    NO def-note edit);
  - pass 2: def fingerprint unchanged → FAST PATH (asserted: no 2nd ensureChannel /
    setupAndRegister) → yet refreshDefStatus re-resolves grants → the note is
    corrected pending→enabled (one new PATCH) and the live agent is now enabled +
    the connection resolves approved.

Follow-up (NOT fixed here): the runner's loadJobs backoff isn't per-vault isolated
(VaultJobStore.listAll has no per-transport try/catch) → filed as agent#189.

Gates: bun run typecheck clean; bun test ./src 1476 pass / 0 fail; biome 0 errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XLZtmuSs1RirWGMGyCB1QB
@unforced
unforced merged commit 4b265c3 into main Jul 2, 2026
2 checks passed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant