test_cpu_x86_llamacpp_floor fails on a BUSY host, so preflight can fail itself: the contended-leg test asserts exit 2 but the unpinned real-loadavg quiet window exits 4 first

[localai-bot]

scripts/agent-preflight.sh goes RED on a busy host, on a change that cannot
possibly affect it. Found while landing a .agents/-and-docs/-only records
reconciliation (#527): zero src/, include/, tests/ or scripts/ files in
the diff, and test_cpu_x86_llamacpp_floor still failed inside preflight.

FAIL: test_a_contended_leg_is_discarded_and_never_summarised
AssertionError: 4 != 2 : ours rep=1 START load=73.49 91.67 65.95 builders=0

Mechanism

test_a_contended_leg_is_discarded_and_never_summarised
(tests/scripts/test_cpu_x86_llamacpp_floor.py:160-166) runs the real harness
with FOREIGN_MAX=-1 to force every leg contended, and asserts the harness
gives up with exit 2:

got = self.run_harness(out, FOREIGN_MAX="-1")
self.assertEqual(got.returncode, 2, got.stdout + got.stderr)

But scripts/cpu-x86-llamacpp-floor.sh gates every leg behind a quiet window
first:

QUIET_BUSY=${QUIET_BUSY:-10}                                   # :39
if [ "$p" -le "$QUIET_BUSY" ] && [ "$b" -eq 0 ]; then return 0; fi   # :122
echo "NO_QUIET_WINDOW after ${waited}s ..."                     # :126  -> exit 4
...
if [ "$attempt" -gt 24 ]; then echo "GIVING_UP too many discards"; exit 2; fi  # :336

The test does not pin QUIET_BUSY, so the harness reads the real host
loadavg. On a busy box wait_quiet times out and the harness exits 4
(NO_QUIET_WINDOW) before the discard loop can ever reach the 24 attempts that
produce exit 2. The sibling test
test_no_quiet_window_stops_instead_of_averaging_through_it asserts exit 4 and
pins QUIET_BUSY="-1" explicitly — so the pattern for making this hermetic is
already in the file, two tests down.

Why it matters more than a flake

Preflight runs its gates concurrently, so preflight raises the load that makes
this test fail — it can fail itself. Observed today across three preflight
runs at loadavg 73, 38 and 30, while the same test passed 10/10 standalone
twice on the same tree minutes apart at lower load. A gate whose result depends
on how busy the box is teaches readers to ignore it, and this repo has already
been burned by environment-manufactured failures (a full disk producing a fake
"137 failed / 10 errors" pytest result).

Suggested fix

Pin the environment the test claims to be testing, the way its sibling already
does: pass QUIET_BUSY (and WAIT_TIMEOUT) explicitly in
test_a_contended_leg_is_discarded_and_never_summarised so the quiet-window
check cannot pre-empt the discard path being asserted. Audit the other
run_harness callers in the file for the same unpinned dependency —
test_the_published_figures_are_computed_not_transcribed and
test_g5_load_is_recorded_before_and_after_every_leg both assert exit 0 and
look exposed to the same timeout.

NOT fixed in #527's PR: changing a checker/gate's semantics needs its own spec
and a RED-before/GREEN-after demonstration, per AGENTS.md.

[localai-bot]

Root cause CONFIRMED by direct manipulation, not inference.

At loadavg 36.13 on this host, the full file fails standalone (so it is not merely preflight's own parallelism -- preflight just makes it likelier):

Ran 10 tests in 46.362s
FAILED (failures=1)

Pinning only the quiet-window threshold, changing nothing else, makes the same test pass on the same busy host:

$ QUIET_BUSY=100 python3 tests/scripts/test_cpu_x86_llamacpp_floor.py \
    CpuX86FloorHarnessTests.test_a_contended_leg_is_discarded_and_never_summarised
Ran 1 test in 17.825s
OK

So the assertion under test (contended legs are discarded and the harness gives up with exit 2) is correct and reachable; it is only the unpinned QUIET_BUSY default of 10 that pre-empts it with exit 4 on a non-idle box. The fix is to pin QUIET_BUSY in the test, as test_no_quiet_window_stops_instead_of_averaging_through_it already does for the case it owns.

Observed across the day at loadavg 73.49, 53.10, 38.25, 36.13 (fail) and at lower load (pass 10/10, twice).

[localai-bot]

Reproduced on a clean checkout of main with no changes at all — a89b3c456, working tree clean — at loadavg 30.99:

Ran 10 tests in 107.945s
FAILED (failures=1)

So this is not specific to any branch or diff; it is the host's load. Combined with the QUIET_BUSY=100 result above (same test, same busy host, passes), the attribution is closed: the test's dependency on the real system loadavg is the whole defect.