Skip to content

test(audit-gate): assert per-directory scan coverage and re-derive the file-count floors (#4746 Item F1) - #4766

Merged
phasetr merged 1 commit into
mainfrom
refactor/audit-gate-f1-directory-coverage
Jul 28, 2026
Merged

test(audit-gate): assert per-directory scan coverage and re-derive the file-count floors (#4746 Item F1)#4766
phasetr merged 1 commit into
mainfrom
refactor/audit-gate-f1-directory-coverage

Conversation

@phasetr

@phasetr phasetr commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Part of #4746 (Item F1). Test-only: scripts/test_audit_gate.py. No change to
scripts/audit_gate.py, no Lean rebuild.

Why now

CheckedFilesTest's blanket floors were calibrated a few dozen files below the
measured count, so every batch of the safe-to-delete lane had to lower them:
2003 / 1996 (#4749) -> 1993 / 1986 (#4751) -> 1984 / 1977 (#4754). That is the
ratchet describing the deletions instead of constraining them, and the standing
decision recorded in the file said the first batch that would actually trip a
floor must land F1 -- per-directory assertions -- rather than recalibrate again.

Re-measured independently on f8911169 (this PR's base) with
python3 -c "import audit_gate as ag; print(len(ag.iter_checked_files()), len(ag.iter_lib_files()), len(ag.iter_v4_files()[0]))":

counter measured old floor deletions still allowed (assertGreater)
iter_checked_files() 1964 1957 6
iter_lib_files() 1957 1950 6
iter_v4_files() 1994 1977 16

The next ten-module batch trips two of the three floors, so this is that batch's
prerequisite.

What changed

F1-a — directory parity (no calibration). LIB_TOP_LEVEL_DIRS pins the 25
top-level IsingModel/ subdirectories, asserted twice: against a census of what
iter_checked_files() enumerates, and against an independent disk traversal
(LIB_DIR.iterdir() + rglob("*.lean")). The two are deliberately not the same
assertion -- deriving the expected set from the scan would be the
CAPSTONE_MIN_COUNT == len(read_capstones()) anti-pattern the suite already
refuses. Measured: 25 directories from the scan, 25 from disk, identical sets.
A filter such as if "Peierls" not in p.parts now fails by name at any
library size
; when the lane empties a directory, scan and disk shrink
together, so no recalibration follows. Only a whole-top-level-directory deletion
(last one: #4703, IsingModel/LatticeSystemBridge) costs an edit, and that edit
is the point.

F1-b — coarse per-directory floors. Parity only requires one surviving file
per directory, so LIB_DIR_FLOORS adds a count floor for the eight directories
at 40+ files plus the loose bucket, at roughly 60-70 % of the measurement:

directory measured floor directory measured floor
Concrete 843 600 Peierls 105 80
AmbientLattice 275 180 AmbientComplexAnalyticity 76 55
ClusterExpansion 205 150 Inequalities 71 50
TransferMatrix 109 80 Conditioning 48 35
<top level> (loose) 39 25

assertGreaterEqual, unlike the strict blanket floors. Smaller directories get
no count floor: parity is the whole guard for them and a floor of "3" would be
noise.

F1-c — one-time re-derivation of the blanket floors. With directory loss now
asserted directly, the floors come from a budget instead of from the current
measurement: 200 further deleted modules (about 100 for the derived remainder of
the Item A lane, plus the excluded #4563 SpecialCases cluster, plus room for
cascade growth), rounded down to a multiple of ten.

constant old new derivation
CHECKED_FILE_FLOOR 1957 1760 1964 - 200 -> 1760
LIB_FILE_FLOOR 1950 1750 1957 - 200 -> 1750
V4_FILE_FLOOR 1977 1790 1994 - 200 -> 1790

The floors are kept rather than deleted because they are the only assertions
that survive the two ways the exact git ls-files comparison can be made
vacuous: narrowing EXTRA_CHECKED_ROOTS and the suite's own ROOTS in one
commit, and a pathspec that matches nothing (set() - scanned is empty, so the
subset assertion passes trivially). Neither needs a tight floor. A new test
pins the floors' residual power as arithmetic: losing Concrete wholesale
(843 of 1964 files) must still take the count under CHECKED_FILE_FLOOR.

F1-d — mutation pairing. Two new mutations in MutationTest, per this
file's rule that every check is tested in the fixture and the mutation
direction, plus a LibraryCensusTest pinning the census helper on hand-built
paths (nested files, loose files, paths outside the library, missing bucket).

Verification

All commands run at d1f9ca7f.

  • python3 scripts/test_audit_gate.py: 179 tests OK (skipped=1), 18.3 s
    -- 166 before, 18.6 s, so +13 tests at unchanged runtime.
  • python3 scripts/audit_gate.py --self-test: OK (same 179).
  • python3 scripts/audit_gate.py --full: PASS -- V1 1964, V2 1964,
    V3 13 capstones (axiom union {propext, Classical.choice, Quot.sound}),
    V4 1994. Run against the identical Lean tree of the base commit in the warm
    build tree, since V3 needs oleans; the change is Python-test-only and V1/V2/V4
    were additionally re-run on this branch's own tree with the same counts.
  • rg -n "[Japanese ranges]" scripts/test_audit_gate.py: no match (V4 is green
    on the new file as part of the run above).

Injected-mutation check (that the new assertions really go red). Each
weakening was applied to scripts/audit_gate.py or to the pinned literal, the
new tests were run, and the file was restored (git diff clean afterwards):

injected weakening result
iter_checked_files filtered on "TransferMatrix" RED: test_the_scan_reaches_every_top_level_library_directory ("Items in the second set but not the first: 'TransferMatrix'") and test_every_large_directory_keeps_its_coarse_floor ("TransferMatrix is down to 0 scanned files (floor 80)"); disk-parity test stays green, as it must
Conditioning thinned to 3 files (45 removed) RED: only test_every_large_directory_keeps_its_coarse_floor among the count assertions; directory parity and both blanket floors stay green -- exactly the gap F1-b exists to close
"Peierls" removed from LIB_TOP_LEVEL_DIRS RED: both parity tests plus test_the_floor_table_names_only_buckets_that_exist
a non-existent name added to LIB_TOP_LEVEL_DIRS RED: both parity tests, naming GhostDirectory

The two committed mutation tests are pinned by load_mutated, which raises if
its target text stops matching, so they cannot go vacuous.

Effect on the lane

Headroom after this PR: blanket 204 / 207 / 204 file deletions, Concrete 243,
and no count floor at all on the small directories (parity only). The remaining
Item A lane is about 100 modules, so no further recalibration is expected;
the next constant edit would be a deliberate top-level-directory deletion or a
shrink beyond the 200-module budget, which is a governance event by
construction.

Not included: the .self-local/issues/4746.md mirror entry for Item F1, which
is dev-pr-clerk's file and is being edited concurrently on another branch.

🤖 Generated with Claude Code

…e file-count floors (#4746 Item F1)

`CheckedFilesTest`'s three blanket floors (`CHECKED_FILE_FLOOR`,
`LIB_FILE_FLOOR`, `V4_FILE_FLOOR`) were calibrated a few dozen files below the
measured count, so every deletion batch of the #4746 safe-to-delete lane had to
lower them: 2003/1996 (#4749) -> 1993/1986 (#4751) -> 1984/1977 (#4754). The
ratchet described the deletions instead of constraining them, and the file's
own standing decision was that the first batch which would actually trip a
floor must land F1 (per-directory assertions) rather than recalibrate again.
Measured on `f8911169` the slack is 6 further file deletions on two of the three
floors, so that batch is the next one.

F1-a: pin `LIB_TOP_LEVEL_DIRS` (25 directories, measured) and assert it both
against a census of what the scan enumerates and against an independent disk
traversal. A filter such as `if "Peierls" not in p.parts` now fails by name at
any library size, with no constant to move; a directory the deletion lane
empties disappears from both sides at once, so no recalibration follows.

F1-b: `LIB_DIR_FLOORS`, coarse per-directory floors for the eight directories
at 40+ files plus the loose bucket (Concrete 843 -> 600, AmbientLattice
275 -> 180, ClusterExpansion 205 -> 150, TransferMatrix 109 -> 80, Peierls
105 -> 80, AmbientComplexAnalyticity 76 -> 55, Inequalities 71 -> 50,
Conditioning 48 -> 35, loose 39 -> 25). Parity only requires one surviving file
per directory; these keep the big ones populated. `assertGreaterEqual`, not the
strict comparison the blanket floors use.

F1-c: with directory loss now asserted directly, the blanket floors are derived
from a budget rather than from today's measurement: 200 further deleted modules
below the 2026-07-29 measurement (1964/1957/1994), rounded down to a multiple of
ten, giving 1760/1750/1790. Their residual job -- the two ways the exact
`git ls-files` comparison can be made vacuous -- does not need a tight floor.

F1-d: two mutation tests. `iter_checked_files` filtered on "Peierls" loses the
directory from the census while staying above the blanket floor; the same
function thinned to three `Conditioning` files keeps parity and the blanket
floors green and trips only the per-directory floor.

No change to `scripts/audit_gate.py`; V1/V2/V4 read the same 1964/1964/1994
files as before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@phasetr
phasetr merged commit 536a7ce into main Jul 28, 2026
1 check passed
@phasetr
phasetr deleted the refactor/audit-gate-f1-directory-coverage branch July 28, 2026 18:07
phasetr added a commit that referenced this pull request Jul 28, 2026
…a7ce) (#4767)

docs(issue-mirror): record #4746 Item F1 progress (PR #4766, main 536a7ce)

Sync the local mirror with the merged per-directory ratchet-floor PR so the
mirror stays the single source of truth for the next session.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
phasetr added a commit that referenced this pull request Jul 28, 2026
…re-derive its declaration floor

The canary's declaration-count floor was `assertGreater(count, 1000)` against a
measured census of 1005 at main `ddf332d9`, i.e. four further deletions of
Unicode-bearing declarations before the CI gate step turns red.  #4746 batch 7
deletes seven of them (measured 998 on the batch-7 branch), so the floor is one
batch away from forcing the same recalibration `test_audit_gate.py` needed in
PR #4766 (#4746 Item F1).

The count was never the Unicode guard.  The guard is the universal in
`run_canary`: every Lambda/beta/sigma-bearing declaration must find itself in
its own defining file, which holds at any population size.  The count's only
residual job is anti-vacuity.  So this commit moves the power to assertions that
need no calibration and re-derives the number once, from purpose:

- a fixture tree (through the scanner's own `build_tree`) holding exactly one
  declaration per canary character.  Per-character coverage on the real tree
  rests on three incidental declarations -- beta on exactly one,
  `Inequalities/HighTemp/SimonLiebInfinite.lean:20`, sigma on two in
  `TransferMatrix/MarkedTraceClosedWalk.lean` -- so the claim "the scanner
  handles beta" was guaranteed by an accident of naming.  It is now a fixture.
- two mutations, the direction nothing tested before: no test proved
  `run_canary` can fail at all.  A cleaned text with the Greek letter removed
  (the signature of the tokenizer defect that produced three bad deletion
  sweeps) must raise "cannot find itself", and a tree with no Unicode
  declarations must raise "canary degenerated".
- `CANARY_DECL_FLOOR = 500`, `assertGreaterEqual`.  Derivation, measured on
  `ddf332d9`: census 1005 (Lambda 1002, beta 1, sigma 2); 457 of those live
  inside the #4746 Item A path allow-list and 2 more inside
  `AmbientLattice/SpecialCases` (#4563).  No batch is authorised to delete
  anything else, so the campaign cannot push the census below 1005 - 457 - 2 =
  546 even if every allow-listed module were deleted, and a cascade can only
  promote modules already inside the allow-list.  The measured remaining lane
  (102 fully-dead modules on `ddf332d9`) carries 79 canary-bearing declarations,
  so the realistic floor is 926.  Expected recalibrations for the rest of the
  campaign: zero.

Test-only.  `scripts/dead_candidate_scan.py` is unchanged, so no scan verdict,
report or exit code moves.  111 tests pass via both entry points, including the
CI gate step `python3 scripts/dead_candidate_scan.py --self-test`.

Design: `.self-local/reports/design-canary-floor-f1-2.md`.

Part of #4746

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
phasetr added a commit that referenced this pull request Jul 28, 2026
…re-derive its declaration floor (#4746 Item F1-2) (#4769)

* chore: open branch for #4746 Item F1-2 (canary declaration floor)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(dead-candidate-scan): pair the Unicode canary with fixtures and re-derive its declaration floor

The canary's declaration-count floor was `assertGreater(count, 1000)` against a
measured census of 1005 at main `ddf332d9`, i.e. four further deletions of
Unicode-bearing declarations before the CI gate step turns red.  #4746 batch 7
deletes seven of them (measured 998 on the batch-7 branch), so the floor is one
batch away from forcing the same recalibration `test_audit_gate.py` needed in
PR #4766 (#4746 Item F1).

The count was never the Unicode guard.  The guard is the universal in
`run_canary`: every Lambda/beta/sigma-bearing declaration must find itself in
its own defining file, which holds at any population size.  The count's only
residual job is anti-vacuity.  So this commit moves the power to assertions that
need no calibration and re-derives the number once, from purpose:

- a fixture tree (through the scanner's own `build_tree`) holding exactly one
  declaration per canary character.  Per-character coverage on the real tree
  rests on three incidental declarations -- beta on exactly one,
  `Inequalities/HighTemp/SimonLiebInfinite.lean:20`, sigma on two in
  `TransferMatrix/MarkedTraceClosedWalk.lean` -- so the claim "the scanner
  handles beta" was guaranteed by an accident of naming.  It is now a fixture.
- two mutations, the direction nothing tested before: no test proved
  `run_canary` can fail at all.  A cleaned text with the Greek letter removed
  (the signature of the tokenizer defect that produced three bad deletion
  sweeps) must raise "cannot find itself", and a tree with no Unicode
  declarations must raise "canary degenerated".
- `CANARY_DECL_FLOOR = 500`, `assertGreaterEqual`.  Derivation, measured on
  `ddf332d9`: census 1005 (Lambda 1002, beta 1, sigma 2); 457 of those live
  inside the #4746 Item A path allow-list and 2 more inside
  `AmbientLattice/SpecialCases` (#4563).  No batch is authorised to delete
  anything else, so the campaign cannot push the census below 1005 - 457 - 2 =
  546 even if every allow-listed module were deleted, and a cascade can only
  promote modules already inside the allow-list.  The measured remaining lane
  (102 fully-dead modules on `ddf332d9`) carries 79 canary-bearing declarations,
  so the realistic floor is 926.  Expected recalibrations for the rest of the
  campaign: zero.

Test-only.  `scripts/dead_candidate_scan.py` is unchanged, so no scan verdict,
report or exit code moves.  111 tests pass via both entry points, including the
CI gate step `python3 scripts/dead_candidate_scan.py --self-test`.

Design: `.self-local/reports/design-canary-floor-f1-2.md`.

Part of #4746

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* test(dead-candidate-scan): pin CANARY_CHARS so the per-character fixture is not self-fulfilling

Raised by the codex cross-check of the previous commit: the fixture derives both
its declaration names and its expected `per_char` mapping from
`dcs.CANARY_CHARS`, so it could not detect a shrink of that constant.  Dropping
`β` from it -- the cheapest way to silence the degeneracy abort once the
library's single `β`-bearing declaration is renamed -- would have left every
canary assertion green while the character stopped being checked at all.  That
is the same "derive the calibration from the quantity it calibrates" anti-pattern
the suite already refuses elsewhere.

Also records, in the floor's comment block, that 1005 - 457 - 2 = 546 is an
*authorisation* bound on the #4746 deletion campaign and not an invariant this
file can enforce: a deletion outside the path allow-list is caught by review, not
by the number.

112 tests pass via both entry points.

Part of #4746

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant