27B decode per-call attribution is not reproducible across box states: the same binary/arm moved -6.5% on the dominant fp8 shape, twice the deficit it is used to rank

[localai-bot]

The 27B NVFP4 decode per-call attribution (~/work/dattr-evidence/RESULTS.txt, dgx)
ranks our fp8 tower against the pin at the ~3% level per call:

GDN in_proj_qkv  48 vs 48   18.3535 vs 17.825  +0.529 ms/step  (382.4 vs 371.4 us/call)
out_proj/o_proj  64 vs 64   10.2578 vs  9.6514 +0.602 ms/step  (160.28 vs 150.80 us/call)

Those two terms are 1.13 ms of the measured +1.81 ms/step decode gap, and every
lever this campaign has ranked was ranked on them.

They were taken across box states. The same binary, the same arm, the same
command, re-run after a box reboot, moved the dominant shape's per-call time
382.60 -> 357.59 us (-6.5%) and the o_proj site 158.0 -> 154.5 us (-2.2%).
dgx rebooted four times on 2026-08-12 (01:43, 13:45, 18:26, 23:18), and the
ours and pin legs of the accepted take are not pinned to one of those
sessions in the record.

A -6.5% same-arm, same-binary shift is twice the size of the +3% deficit being
attributed to our kernel. Until a both-arms table is taken inside ONE box state
with a same-arm repeat inside each take, we cannot say whether the fp8-tower
deficit is our code or the box, and neither can any lever ranked against it.

What this blocks / calls into question:

• the +0.529 / +0.602 per-site deficits themselves;
• any per-shape lever sized against them (cuBLASLt algo search, per-shape cuDNN
  routing, the bf16-D narrowing's expected win);
• the standalone-vs-in-situ transfer arguments, which subtract two numbers taken
  in different box states.

Owed: a both-arms (ours, pin) nsys decode capture, order-alternated
ours -> pin -> ours -> pin inside one uptime session, each arm repeated, with
the same-arm spread reported beside the cross-arm delta so the deficit can be
sized against its own noise.

Two environment defects that silently void an arm and must be fixed in the
harness first:

• /tmp/nvidia/nsight_systems is root-owned on dgx, so nsys launch as mudler
  fails with Permission denied and the arm surfaces only as ready=0. Set
  TMPDIR=$HOME/tmp.
• an aborted nsys run leaves its session name claimed, so a re-run fails with
  "Session name already in use" the same silent way. Session names must be unique
  per invocation.

[localai-bot]

The cause is named and measured: the GB10's SM clock is not stable across boots, and it moves 12.8%.

The chain added an SM-clock sampler across every captured window. Two runs of the
same binary (4064558d0), same arm, same command, same recipe, on two
consecutive boots:

boot	SM clock across the captured window	our uninstrumented ms/step
f6bbbfc6	n=61 min 2398 / med 2470 / max 2489 MHz	82.1664
2fca2b02	n=50 min 2190 / med 2190 / max 2190 MHz	88.1000

The second boot is not ramping — it is pinned flat at 2190 MHz for all 50
samples (clocks.max.sm reports 3003). That is a 12.8% clock difference
between boots, producing a +7.2% step-time difference on identical code. A
partly memory-bound decode step moving less than the clock ratio is exactly what
this should look like.

This accounts for every drift observation that opened this issue, without any of
them needing a code explanation:

• the reported same-arm 382.60 -> 357.59 us (-6.5%) across a reboot;
• two probe runs 8 minutes apart inside one boot differing ~+6% uniformly on all
  shapes (~/work/probe4 entered at 2398 MHz, ~/work/probe5 at 1781 MHz);
• the control kernel: marlin::Marlin, 129 calls/step, byte-identical
  invocation on both arms, no source change between a170c81c and
  4064558d0 (git diff --name-only a170c81c 4064558d0 | grep -i marlin is
  empty), moved 45.2845 -> 49.6544 ms/step (+9.65%) between the audited take
  and a fresh capture.

The control's own drift (+9.65%) is larger than either cross-arm deficit it
was used to rank (in_proj +2.97%, out_proj/o_proj +6.28%).

Conclusion for this issue's question: a per-call ratio taken across boots on this
box is not a measurement of the arms; at the 3-10% level it is a measurement of
the SM clock. Per-call absolute times are comparable only within one boot, with
the clock envelope recorded beside them. That control is now in the instrument
(~/work/boxstate/bs-cap.sh).

What is still owed and why it is not delivered: the both-arms one-box-state table
itself. Two attempts to capture it were destroyed by host reboots 17 minutes
apart, filed as #545 — the box's MTBF is now shorter than one four-leg chain. The
deficit therefore remains NOT ESTABLISHED, in the specific sense that it has
never been measured against a clock control, and the one control we do have is
bigger than it.