Skip to content

fix(ci): DSR-ALLOW the server TU's profiler guards, and unbreak the evidence map - #194

Merged
localai-bot merged 1 commit into
mainfrom
row/ENG-CI-GREEN
Aug 9, 2026
Merged

fix(ci): DSR-ALLOW the server TU's profiler guards, and unbreak the evidence map#194
localai-bot merged 1 commit into
mainfrom
row/ENG-CI-GREEN

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

The bug

check-pr-size requires a checker change to ship mutation evidence in a companion test, and derives that path as tests/scripts/test_check_<name>.py. check-device-leakage.py's suite predates that convention — it lives at tests/scripts/test_device_leakage.py, and ci.yml runs it under that name.

So the derived path pointed at a file that does not exist, and every change to that checker failed with "requires semantic mutation evidence in tests/scripts/test_check_device_leakage.py". A checker nothing could modify.

Not hypothetical: check-device-leakage is red on main right nowvt_ifdef 37 > baseline 32, because #189 moved the server body into the shared layer carrying its 5 #ifdef VT_BENCH_PROFILE_CONTROL sites — and the ALLOWLIST repair its own error message recommends is exactly what this blocked.

Fixed via CHECKER_EVIDENCE_OVERRIDES, which exists for this and already carries four entries.

The audit

All 30 checkers, one more gap: check-dsv4-gguf-namemap.py has no test suite at all and is not in ci.yml. Inventing a suite for it here would be speculative, so it is named in a KNOWN_UNTESTED set — visible in a test rather than invisible in a naming rule, and explicitly not papered over.

The new test asserts every scripts/check-*.py maps to an evidence file that exists, so this class cannot recur silently.

Mutation: removing the override gives Lists differ: ['scripts/check-device-leakage.py -> tests/scripts/test_check_device_leakage.py'] != [].

What this does NOT do, and needs your ruling

It does not fix the DSR failure itself. Both available routes need a decision that isn't a CI repair:

route blocker
annotate the 5 sites with the checker's own // DSR-ALLOW hatch touches src/, so check-doc-checkpoint demands STATUS + BENCHMARKS + USAGE updates for a comment-only change — and docs/STATUS.md has 3 chars of ratchet headroom
use the per-file ALLOWLIST means editing check-device-leakage.py, which check-pr-size rejects with "has no affected POL rule mapping" — no POL rule in policy.csv names that checker, so binding it in CREATED_CHECKER_RULES asserts a governance mapping

I verified the DSR-ALLOW route works (DSR back to 32 == 32, baseline untouched, exemptions printed every run) and had it passing with a per-site test pinning that every exempted #ifdef VT_ in that TU is VT_BENCH_PROFILE_CONTROL — a per-file count of 5 would still read 5 if someone swapped one guard for a real device fork. I pulled it back out rather than manufacture three doc edits to get a comment past a gate.

Say which route you want and I'll land it. The genuine repair, separately, is to put the profiler seam behind a Platform capability query like every other device fork — a change to #189's TU, not to the gate.

Gates

check-agent-record, check-public-doc-tables, check-env-doc, check-state-order, check-now-current, check-role-discipline, check-readme-structure, check-policy, check-protocol-consistency, and test_check_pr_size 24/24, on a worktree pinned at dc2139b3. No product code touched. check-device-leakage stays red — this does not claim to fix it.

Please squash-merge.

…vidence map

Two fixes, both for gates that were red or unusable on main.

1) check-device-leakage was RED: DSR bucket vt_ifdef 37 > baseline 32.
Cause is mechanical, not a real leak. #189 moved the server body out of
examples/server/main.cpp, which the scanner never looked at, into
src/vllm/entrypoints/openai/server_main.cpp, which IS the device-agnostic
shared layer, carrying its 5 `#ifdef VT_BENCH_PROFILE_CONTROL` sites with
it. Not one line of that code changed; it crossed the scanner's boundary.

Exempted per-site with the checker's own `// DSR-ALLOW(<row-id>):` hatch --
comments only under src/, no behavior change. Per-site rather than a
per-file ALLOWLIST budget because a budget can be spent on something else:
swap one profiler guard for a real device fork and a count of 5 still reads
5. The baseline is NOT raised; DSR returns to 32 == 32 and the 5 exemptions
print in CI output every run, so the exception budget stays visible.

The genuine repair is to put the profiler seam behind a Platform capability
query like every other device fork. That is a change to #189's TU, not to
this gate, and is not attempted here.

2) check-pr-size could never accept a change to check-device-leakage.py.
recognized_evidence derives tests/scripts/test_check_<name>.py, but that
suite predates the convention and ci.yml runs it as
tests/scripts/test_device_leakage.py, so every change to that checker
failed demanding a file that does not exist. Added to
CHECKER_EVIDENCE_OVERRIDES, which exists for exactly this. An audit of all
30 checkers found one more: check-dsv4-gguf-namemap.py has no suite at all
and is not in ci.yml -- named in a KNOWN_UNTESTED set so the gap is visible
in a test rather than invisible in a naming rule.

The docs this change owes are real, not filler.

USAGE: --cuda-profile-graph-replays and --cuda-profile-graph-batch are
parsed by the server and were undocumented. They are the flags that DRIVE
the guarded code this change annotates, so they belong here.

BENCHMARKS: a "No number owed" row in the same shape as the 2026-08-08
async-readback DSR row directly above it.

STATUS said "All six record checkers now green on main", which #189 made
false. Corrected to the current state. It costs chars against a byte-tight
ratchet, so it is paid for by collapsing the env-doc hygiene sentence
beside it: that sentence spelled out which of two env vars went to
ENVIRONMENT.md and which to the allowlist, and both files state that
authoritatively. Net -111, and the ratchet is re-pinned byte-tight to
244015 to match the convention #190 set it with, rather than quietly
keeping the slack as headroom. The evidence test for that re-pin bounds
the slack in both directions, so a future red ratchet cannot be cleared by
inflating the number instead of shrinking the page.

Mutations, bytecode disabled:
  - downgrade one DSR-ALLOW to a plain comment -> per-site test fails
    "line 50 has no DSR-ALLOW on it or directly above it" AND the ratchet
    fails 33 > 32
  - remove the evidence override -> "Lists differ:
    ['scripts/check-device-leakage.py -> ...'] != []"
  - ratchet 248000 -> FAILED (3); 243000 -> FAILED (4); 244015 -> OK

Gates: check-device-leakage OK (32 == 32), check-agent-record,
check-public-doc-tables, check-env-doc, check-state-order,
check-now-current, check-role-discipline, check-readme-structure,
check-policy, check-protocol-consistency; test_device_leakage 26/26,
test_check_public_doc_tables 51/51, test_check_pr_size 24/24. Pinned at
dc2139b. Only comments change under src/.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: Claude-Code:claude-opus-5 [Claude Code]
@mudler
mudler force-pushed the row/ENG-CI-GREEN branch from e827056 to 3a9f059 Compare August 9, 2026 09:50
@localai-bot localai-bot changed the title fix(ci): check-device-leakage.py could never satisfy its own evidence rule fix(ci): DSR-ALLOW the server TU's profiler guards, and unbreak the evidence map Aug 9, 2026
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Took the DSR-ALLOW route as directed. The PR now fixes the red gate itself, not just the mapping that made it unfixable.

check-device-leakage was red on main: vt_ifdef 37 > baseline 32, because #189 moved the server body into the shared layer carrying 5 #ifdef VT_BENCH_PROFILE_CONTROL sites. Not one line of that code changed — it crossed the scanner's boundary. Exempted per-site with the checker's own hatch: comments only under src/, no behaviour change, baseline untouched, DSR back to 32 == 32, and the 5 exemptions print in CI output every run.

Per-site rather than a per-file ALLOWLIST budget because a budget can be spent on something else — swap one profiler guard for a real device fork and a count of 5 still reads 5. The new test pins that every exempted #ifdef VT_ in that TU is VT_BENCH_PROFILE_CONTROL.

The docs this triggered are real, not filler. Touching src/vllm/entrypoints/ demands STATUS + BENCHMARKS + USAGE, and I said earlier I wouldn't manufacture three edits to get a comment past a gate. Each one turned out to be genuinely owed:

The STATUS correction costs chars against #190's byte-tight ratchet, so it is paid for by collapsing the env-doc sentence beside it — that sentence spelled out which of two env vars went to ENVIRONMENT.md and which to the allowlist, and both files state that authoritatively. Net −111, and I re-pinned byte-tight to 244015 rather than quietly keeping the slack, matching the convention #190 set. The evidence test for that re-pin bounds slack in both directions, so a future red ratchet can't be cleared by inflating the number instead of shrinking the page.

Mutations (bytecode disabled):

mutation result
downgrade one DSR-ALLOW to a plain comment per-site test fails "line 50 has no DSR-ALLOW on it or directly above it" and ratchet fails 33 > 32
remove the evidence override Lists differ: ['scripts/check-device-leakage.py -> ...'] != []
ratchet 248000 / 243000 / 244015 FAILED(3) / FAILED(4) / OK

Still not attempted, and still the right long-term fix: put the profiler seam behind a Platform capability query like every other device fork — a change to #189's TU, not to this gate.

@localai-bot
localai-bot merged commit b859724 into main Aug 9, 2026
11 checks passed
@mudler
mudler deleted the row/ENG-CI-GREEN branch August 9, 2026 10:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants