test(role): the publish observer cannot hang, and its sizing is measured (#296)

[localai-bot]

Merge pull request #304 from mudler/row/ENG-OPERATOR-RECORD-COVERAGE-FIX

test(role): the publish observer cannot hang, and its sizing is measured (#296)

Closes #296, the two coverage limitations #285 left recorded rather than fixed.

LIMITATION 1 — a comment the code falsified. RECORD_TTL_SECONDS still said a
stale record is "unlinked by the next claim", false for this worktree's own
record after keep_canonical (it is replaced). The spec's copy had been
corrected; the source copy one screen above the function had not. A tree-wide
grep confirms there is no third copy.

LIMITATION 2 — the publish-NAME pin was escapable. It monkeypatch-watched four
I/O primitives, so an os.rename + open() publish walked past it while a real
concurrent show returned rc 3. The fix ships BOTH halves, because they fail
differently: a widened watcher (deterministic, catches the primitives it names)
and an outside observer, 2000 publishes in a subprocess under an exists()
poller (one-sided -- a hit proves absence, a miss proves nothing -- so it can
never false-RED, and it is the only half that can catch a primitive nobody
enumerated).

That is not a theory. Across this review three distinct publish shapes were
found that the deterministic half cannot see, each by a different fresh
reviewer:

• bytes first, then basename unlink, then os.rename(temp, name)
• and, this round, a NEW CLASS: absence caused by an ANCESTOR rather than the
  leaf -- rename the records DIRECTORY aside and back, so no watched call ever
  receives the record's own path
  Both escape the watcher, the hardlink witness and the residue check, and both
  are caught by the observer alone, 10/10 in multi-core and single-core regimes.

TWO REVIEW ROUNDS, and the first round's findings were both real:

1. The observer could HANG THE SUITE FOREVER. An unread stderr=PIPE plus an
   unbounded spin deadlocked once the child filled the 65536-byte pipe --
   measured on the CORRECT publish at 391 B/publish: rc=124, killed at 90 s,
   ~100% CPU. Reachable by anyone's change, since the child forks git rev-parse on every publish. Now: child output goes to FILES, the ready
   handshake is a bounded file poll, and both loops are bounded by a 60 s
   deadline. Both arms verified firing by an independent reviewer -- wedged
   mid-publish FAILED at 60.077 s, wedged before ready FAILED at 5.077 s with a
   shortened deadline -- with 0 orphan processes and 0 leaked files after ~60
   runs including three hard-failure paths.
2. The sizing rationale was falsified by measurement. The comment claimed
   caching worktree_id removed the subprocess dilution; record_path() ->
   common_dir() still forked git rev-parse every publish, which was 89% of
   the observed window. Caching it lifted single-core detection 0/30 -> 7/30.

The residual was then CLOSED rather than restated: with the cheaper loop, raising
the count 200 -> 2000 makes detection 20/20 in BOTH regimes, which falsifies the
spec's "costs seconds per invocation" declination. Independently measured cost:
+0.36 s on a 5.58 s suite, slightly cheaper than the implementer's own ~0.5 s
estimate.

VERIFIED BY THE OPERATOR rather than taken on report: the commit touches exactly
two files and scripts/agent-role.py is byte-unchanged, so no product behaviour
moves; test_agent_role 60 tests in 5.312 s at load 142; 5 consecutive runs
green. The reviewer independently confirmed 119 tests across the three suites,
0/40 unmutated flake in both regimes, and agent-preflight.sh --quiet --no-require-role all gates green.

ONE LOW FINDING RECORDED, NOT REPAIRED. The deadline comment claims "over two
orders of magnitude of headroom" and the failure text prints "~182x the measured
window". Measured under 42x single-core oversubscription (40 CPU hogs, both
processes on one core), correct runs took 18.0-19.3 s -- a 3.1x margin, not 182x;
crossing 60 s would need roughly 130x oversubscription. No false failure was
produced in 45 correct runs across three regimes, and what it replaces was an
unbounded hang, so the change strictly dominates. Tracked as a follow-up rather
than blocking a fifth round on a diagnostic string.

A note on process: the first attempt at this review was terminated by an API
session limit at the full-gate step and returned nothing, so it was re-run from
scratch rather than resumed on assumptions.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude-Code:claude-opus-5 [Claude Code]