dgx.casa cannot currently complete a 27B decode nsys capture. Three attempts on
2026-08-13 between 00:21 and 00:47, each under the GPU lock with the box idle at
the gate and >=90G host RAM free:
| attempt |
boot |
died at |
outcome |
chain 1, leg oursA |
f6bbbfc6 |
00:27, after nsys stop had written the .nsys-rep, during the post-capture request pair |
box rebooted; capture salvaged by exporting the orphaned .nsys-rep |
chain 2, leg oursA |
2fca2b02 |
00:44, during nsys stop |
box rebooted; no .nsys-rep, leg lost |
last reboot for 2026-08-12/13: 01:43, 13:45, 18:26, 23:18, 00:27, 00:44.
The last two are 17 minutes apart and both landed in the same phase of the
same workload. A pin leg alone needs ~8 minutes just to reach ready=1, and a
four-leg both-arms chain needs ~30, so the mean time between reboots is now
shorter than the measurement.
There is no OOM-killer record in dmesg — consistent with the recorded GB10
unified-memory behaviour where the box reboots rather than swaps. The preceding
boot's kernel log does show driver-level distress:
NVRM: Xid (PCI:000f:01:00): 13, Graphics Exception: ESR 0x505730=0x200000e ...
NVRM: Xid (PCI:000f:01:00): 43, pid=22887, name=test_cuda_deeps, channel 0x00000002
NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051)
returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
NVRM: refcntRequestReference_IMPL: Failed to enter state 1 (current state: 0, status: 0x00000056)
Common factor in both reboots: a ~67 GiB 27B NVFP4 server resident while nsys
generates or finalises its report. The same recipe completed at 20:24 on the
18:26 boot, so this is a degradation of the host, not of the recipe.
Two things this blocks right now:
#543's owed both-arms one-box-state per-call table (spec
meas-27b-percall-boxstate.md);- any decode nsys attribution on this box, for anyone.
Also worth noting for whoever debugs it: the box is shared and the GPU lock does
not serialise CPU load. A capture that passed the load < 2.0 idle gate at
1.65 was running at loadavg 8.9-12.3 sixty seconds later, from another agent's
job.
dgx.casa cannot currently complete a 27B decode nsys capture. Three attempts on
2026-08-13 between 00:21 and 00:47, each under the GPU lock with the box idle at
the gate and >=90G host RAM free:
oursAf6bbbfc6nsys stophad written the .nsys-rep, during the post-capture request pairoursA2fca2b02nsys stoplast rebootfor 2026-08-12/13: 01:43, 13:45, 18:26, 23:18, 00:27, 00:44.The last two are 17 minutes apart and both landed in the same phase of the
same workload. A pin leg alone needs ~8 minutes just to reach
ready=1, and afour-leg both-arms chain needs ~30, so the mean time between reboots is now
shorter than the measurement.
There is no OOM-killer record in
dmesg— consistent with the recorded GB10unified-memory behaviour where the box reboots rather than swaps. The preceding
boot's kernel log does show driver-level distress:
Common factor in both reboots: a ~67 GiB 27B NVFP4 server resident while nsys
generates or finalises its report. The same recipe completed at 20:24 on the
18:26 boot, so this is a degradation of the host, not of the recipe.
Two things this blocks right now:
#543's owed both-arms one-box-state per-call table (specmeas-27b-percall-boxstate.md);Also worth noting for whoever debugs it: the box is shared and the GPU lock does
not serialise CPU load. A capture that passed the
load < 2.0idle gate at1.65 was running at loadavg 8.9-12.3 sixty seconds later, from another agent's
job.