refactor(factors): retire the three regeneration tools, verify the frozen bytes instead - #115
Merged
Merged
Conversation
…s instead The D1 baseline was rebuilt by calling the eleven legacy eval runners' private `_load_*_panel` loaders. C6 deletes those runners, so the rebuild could not work again; rather than leave a path that breaks on first use, the capability is retired (owner ruling 2026-07-28) and the baseline is frozen-forever. Retiring it is the safe direction, not merely the available one. The provenance rule was always that this baseline may only be regenerated from the pinned pre-D2 SHA, never from the tree being validated -- rebuilding it here is exactly the empty-reconciliation failure mode this repository committed once. Removing the rebuild removes the only way to break that rule by accident. What replaces it is a real check, not a smaller one. `--verify` recomputes every frozen panel's canonical content hash AND its whole manifest row (rows, date range, symbol count, NaN count, mean, std, file sha256) and checks them against the hashes authored in `docs/factors/` -- the git-tracked side of the split that keeps the bulk panels out of git. That split is load-bearing: a tree regenerated together with its own manifest.json makes the two local witnesses agree with each other, and only the git-tracked table still convicts. Both frozen trees are covered (14 D1 panels + the PR-C corrected jump reference panel); there is no partial mode, because a frozen-forever baseline verified in part is a baseline believed further than it was checked. The manifest-document parser moves to qt/frozen_manifest_doc.py: header-driven, fail-closed on two matching tables, none, a duplicate id or a malformed hash. A parser that silently returned an empty expectation set would pass everything. Also removed with the rebuild: the recipe table, the `--only` selection, the determinism double-run and the eval-artifact reconciliation. The reconciliation outcome survives as a check of the RECORD in manifest.json (every declared field present, every check ok), so a manifest edited to paper over a failure is still caught. The retired flags stay in the parser so the old documented command line answers with the retirement explanation rather than "unrecognized arguments". Tests: 34 (from 33). Six covering the removed machinery are gone with it (determinism subjects, four `--only` selection cases, freezable ids); the "14 frozen factors" property they anchored is now pinned against the git-tracked document itself. Fifteen new parser tests, and one verification test per way a frozen tree can be wrong -- each starting from a tree that verifies GREEN, so a red result is attributable to the tampering.
The D2 rebuild ran the qt.panel_freeze recipes -- the same eleven legacy runner loaders -- so it goes the same way as the freeze (owner ruling 2026-07-28). The comparison, however, never needed those loaders, and it is the part worth keeping. `--verify` re-derives the D2 cell-by-cell verdict from the frozen bytes of BOTH sides: it re-reads the two panel sets from disk through two independent file reads and re-runs compare_panels over all 14 factors. That is a derivation, not a restatement of a recorded result -- if either frozen tree drifts, cells stop matching and the run goes red. The D1 side is authenticated FIRST, by delegating to the freeze's verifier so there is one implementation of "is this the frozen baseline?". Comparing two panel sets while neither has been authenticated reports agreement between two unknowns -- and a test pins exactly that case: two sides tampered identically still agree with each other, and the run still fails. The D2 side is checked against the SAME git-tracked D1 hashes. Not a shortcut: "bit-identical to D1" is precisely what the D2 reconciliation concluded, so the D1 table IS the D2 expectation, and it is the only one of the two in git. manifest_d2.json is still read, for its own per-factor rows and for its recorded all_ok -- a recorded failure must never read as a pass. Tests: 19 (from 6). The six comparator teeth tests are untouched; thirteen new ones cover the retirement and the verification, including that an absent panels_d2 cannot pass by having nothing to compare.
The companion that compared the four ops-rewritten daily factors against their hand-computed anchors imported a legacy runner's precondition check and rebuilt the whole data plane behind it. C6 deletes that runner (owner ruling 2026-07-28). Two things stated plainly rather than folded into the shared retirement wording, because overstating them would be the defect this repository keeps catching: this tool's dependency on the legacy runners was ONE precondition helper, not the eleven private loaders the panel tools called, so what is given up here was not independently doomed; and the four daily factor classes are untouched and keep their own tests -- what is retired is this comparison, not the factors. `--verify` re-derives the RECORDED comparison from its own numbers: for every row it recomputes the relative difference from the stored hand/engine values at the same tolerance and checks the recorded rel_diff, ok flag and all_ok_daily against what those numbers actually imply, plus that every pending anchor got a comparison. A record edited to turn a mismatch into a pass does not survive it. When the record is ABSENT the exit code is non-zero and the wording is "NOT VERIFIED", not "FAILED". Absence of evidence is not evidence of failure, but the exit-code channel is binary, and a verification command that reports success without verifying anything is the empty-reconciliation shape already committed once here. On the tree today that case is live: hand_anchors_d2.json carries 20 pending anchors and no recorded comparison, a later `hand_anchor_rows` run having reset it -- so the command reports NOT VERIFIED rather than pretending. The retirement message composition is now three parts, not two: the shared core (ruling, "no longer rebuilds", pointer to --verify) plus per-tool `why` and `verifies` clauses. A single shared sentence describing what verification checks would have to overstate one of the three tools; this one checks a JSON record, not frozen bytes against a git-tracked manifest, and a test asserts its message does not claim otherwise. Tests: 19 new.
Verification reads two witnesses: the git-tracked document (hashes) and the gitignored manifest.json (the fuller statistical row). Disabling the machine-row comparison broke NO test -- every existing case was also convicted by the document, so the second witness was being read but never depended on. Adds the case only it can see: manifest.json's n_nan is moved while the document is left agreeing with the panel. The test asserts the conviction comes from the machine row AND that the git-tracked side stayed quiet, so it cannot pass for the other reason. With it, disabling that comparison is red.
The eleven per-factor evaluation subcommands were near-identical wrappers around
eleven near-identical runners; D5 C4 replaced them with one parameterised
command. This removes the eleven from the CLI: 433 lines of handler and
subparser, no behaviour of the unified command changed.
The runner MODULES are untouched -- deleting those is the next PR. This one only
stops the CLI offering eleven doors to a thing that has one.
Typing a retired name still produces the route to the new command. argparse's
own answer is "invalid choice", which tells an operator they mistyped rather
than that the command was replaced and by what; the eleven collapsed
mechanically, so the mapping can be printed:
run-eval-valley-price-quantile was retired in D5 C6 ...
python -m qt.cli run-factor-eval --config <config> \
--factor valley_price_quantile_20 [--book-mode decision|close]
That map is DERIVED from factor_eval_reconcile's factor-id/report-name table
(the old names were those report names with dashes), not transcribed -- a second
copy in the CLI is the one nobody would update. The table is made public for it;
its one private call site moves to the public name. The historical list of
eleven names lives in the test, which is where a claim about what used to exist
belongs, and asserts the derivation reproduces exactly those names.
A `run-eval-` name that never existed still falls through to argparse's list of
valid choices rather than getting a fabricated migration.
Retired commands: run-eval-{jump-amount-corr, minute-ideal-amplitude,
amp-marginal-anomaly-vol, volume-peak-count, intraday-amp-cut,
peak-interval-kurtosis, valley-relative-vwap, valley-ridge-vwap-ratio,
ridge-minute-return, valley-price-quantile, peak-ridge-amount-ratio}.
Tests: 27 new.
R16's gate is "before deleting the legacy runner tests, all 59 have a mapping". This runs that reconciliation against the tree rather than re-reading the document, and records the measured result as section 十. Measured: 10 files / 59 tests on disk, 59/59 with a row in the map, 56/56 cited new-suite node-IDs collect today (against 2687 collected), all three family wildcards the size the document claims, every citation resolving to exactly ONE file. R16 holds; the deletion PR has its precondition. The audit is also left behind as a standing guard rather than a one-off run (tests/test_property_test_migration_map.py, 7 tests): a document cannot notice when a test it names is renamed away. Its range is stated in its own docstring — it checks that citations RESOLVE, not that the cited test still asserts the property claimed for it. A renamed test is caught; one gutted in place is not. The legacy-side check goes vacuous the day those files are deleted, deliberately and stated as such: its job is to make an UNMAPPED legacy test impossible while they exist, and a guard that turned red on their deletion would be a booby trap for the very PR this map authorises. Two process notes are recorded because they cost real time and will recur. The audit first passed `--collect-only -q`, which becomes `-qq` on top of the addopts in pyproject and swallows the node-ID listing — it reported a crowd of phantom missing tests. Then excluding the family wildcards with `(?!\*)` was defeated by backtracking: the name pattern gives back its trailing underscore and matches a shorter, still-wrong name, so three family stems still read as missing. `(?![A-Za-z0-9_*])` is the form that holds, and the comment saying so is in the guard. Neither was caught by a test; both were caught by an observation that contradicted the prediction.
The difference catalogue's C6 section required, in advance, that whichever disposition was chosen for these three tools be recorded explicitly and that the D1 manifest's "rerun command" section be updated in the same batch -- otherwise "the document teaches you to run a command that cannot run" recurs, which this repository has already fixed three times (#76/#78/#82). Disposition taken: the second option. Regeneration is retired, the D1/D2 baselines cannot be rebuilt from this repository again, and the three tools stay as verify-only rather than being deleted. - the catalogue's disposition column now records what was actually done, and settles the asymmetric-loss argument it opened: the worry was losing the ability to re-check. That ability did not exist in the old path -- the provenance rule forbids rebuilding the baseline from the tree under validation, which is the empty reconciliation committed here once. `--verify` is a route that did not exist before: 15 panels, canonical hash and full manifest row each, against the expectations in git, in three seconds. - the D1 manifest's rerun command becomes a retirement note plus the verify command, and states the consequence its table now has: the table is READ by the verifier, so editing it changes the verification result and shows up in git diff. - the PR-C reference panel's `--only` recipe is archived as provenance (how that panel came to exist) and marked retired, and its "can be regenerated" cell is corrected to match.
…ed frozen file Record-only, per lead ruling 2026-07-30. No verify logic changes; the C5 harness is not touched. (1) The D2 daily hand-anchor engine comparison was destroyed on 2026-07-25 by a `hand_anchor_rows` rerun, and the tool that refilled it is retired. The entry draws the distinction that matters: the D2 leg's CONCLUSION is still on record (PR #89 and docs/progress/07: "88 hand anchors, 0 mismatches"); what is lost is the ability to re-verify it today. `--verify` therefore exits non-zero on this tree, worded NOT VERIFIED rather than FAILED, and is deliberately kept OUT of the standard gate list -- a red that never turns green trains people to ignore red. (2) That prompted a census of the whole directory, and it found something worse than the one file: 7 of 113 files under artifacts/refactor_baseline carry no hash anywhere. The entry grades them rather than lumping them together -- three manifests are byte-unpinned but content-constrained (verify cross-checks every row against the git document AND the panels), three are derivable renderings, and hand_anchors_d2.json is genuinely unprotected AND is the one that actually got overwritten. The sentence worth keeping: "this change wrote zero bytes of the frozen baseline" is a statement about TODAY'S operation, not about the directory being immutable. It is not immutable. One file in it has already been rewritten. (3) The C5 reconciliation harness reads both the frozen panels (factor_eval_reconcile.py:1539) and hand_anchors_d2.json (:1467) with no hash check at all. Both true sentences are recorded, because either alone misleads: C5's four legs went green against a baseline that was never hash-checked, and (2) proves this directory can be written -- yet today's verify passes 15/15 panels and 14/14 cells, so the panels are intact and the C5 conclusion stands. Fix belongs to the deletion PR, whose subject is making frozen-forever mean it. (4) The three retired tools did not depend on the legacy runners equally: panel_freeze and panel_reconcile import all eleven, while hand_anchors_engine_values used one precondition helper and could have been repointed at the unified runner. The ruling stands -- retiring regeneration is the goal, not an inference from the dependency -- but the asymmetry is recorded: that third retirement really did give up a capability that was not doomed.
…call Defect introduced by this PR and fixed in it. `_check_recorded_reconciliation` iterated the entries PRESENT in manifest.json and failed only when the block was entirely empty -- so deleting one entry of eleven went unnoticed. The verifier could not see a removal. That is the wrong blind spot to ship right now: the next PR is the deletion PR. A check that cannot see removals would be blind precisely during a deletion-themed operation, at the moment it is most needed. And "the expected values are not themselves pinned" is this PR's own subject -- leaving it for later would contradict the thing being built. The expected set is now derived: every row manifest.json lists as a minute panel must have an entry, and nothing else may. Book factors have no entry by design (the eval artifacts carry no coverage fields for them), which is why the set is "minute rows" rather than "all rows". On deriving the expectation from the same document that could be tampered: not usefully tamperable. Deleting the row as well is caught by the panel loop, which iterates the GIT-TRACKED factor list and reports a panel whose machine-manifest row is missing. The two checks close each other, and the code says so. Verified against reality before relying on it: both frozen trees satisfy the invariant exactly (D1 11 minute panels / 11 entries, PR-C 1 / 1), so this is a live constraint rather than a rule the real data was already exempt from -- a third test pins that, skipping only where the gitignored tree is absent. The first version of the deletion test FAILED, correctly: the default fixture has one minute factor, so removing its entry empties the block and the pre-existing "no reconciliation block" message fires -- the test would have passed without the new check existing. Rebuilt with TWO minute factors, which is the real shape (11 becoming 10). Mutation evidence: disabling either direction of the closed set turns the matching test red (16/16 across the suite's mutations, each asserted to have landed and restored by sha).
…aises Review MEDIUM-1. `hand_anchor_rows` printed "(run python -m qt.hand_anchors_engine_values)" on every successful run, and that command now raises. Worse in context: the same function rewrites hand_anchors_d2.json with `daily_pending_engine` and no `daily_engine_compared` -- it IS the action that wiped the record on 2026-07-25. A user had just destroyed the comparison and was then sent down a dead end. It is not one of the eleven runners and the deletion PR would not have touched it. Same shape as #82, and the reviewer found it the way #82 says to: `grep -rn` over the WHOLE repository. I had fixed the two documents I was already looking at and never scanned the feature surface. A guard that only looks where you already looked confirms only what you already know. Scanning properly found more than the reported line. Four stale statements, not one: - the run-summary print (the reported one); - hand_anchors_d2's docstring "Run AFTER `python -m qt.panel_reconcile` produced panels_d2" -- that rebuild is retired too, and panels_d2 is simply there now; - the next docstring line offering the retired companion as a step; - hand_anchor_rows' own docstring claiming those values "are filled in by" the companion. They are not; they stay pending. The pointer is now authored ONCE, as `qt.hand_anchors_d2.ENGINE_COMPARISON_POINTER` plus `pending_engine_line`, and the printer composes it. It lives in hand_anchors_d2 rather than the retired module for a load-bearing reason: hand_anchors_engine_values imports qt.panel_freeze -> data.clean.schema, and the hand side must never load the engine. An author-once home has to be reachable from the pure side, and a test pins that with an AST import check (a substring check tripped on the comment that names data.clean precisely to explain why it stays out). Review MEDIUM-2, in the catalogue: the bare-read site list was missing one and its line numbers had already rotted. Now THREE sites, recorded by SYMBOL -- run_panels_mode, load_anchor_rows, run_anchors_mode. The third is the one that matters most: under `--mode anchors` alone it is the only frozen-panel read there is, so a two-of-three list would have handed the deletion PR a pass that covers two thirds of the surface. The stale line numbers are their own evidence for dropping line numbers: `:1539` was read off main, and this PR's own 711c0a8 had already shifted that file by +3. Also registered, behaviour deliberately unchanged here: that hand_anchor_rows overwrites a file inside the "never overwrite" directory belongs with the frozen-forever work in the deletion PR, alongside wiring verify into all three read sites. Review LOW-1 / NIT-1 in the migration map: the audit's collected-node-ID number held at no commit -- it was measured before that same commit's own seven guard tests existed. Corrected with the reason stated rather than swapped out quietly, and the count is now labelled as context that moves (2694 -> 2697 -> 2701 during this PR) rather than as a claim; the load-bearing numbers are 59/59, 56/56 and zero unresolved. The "56" counting convention is written down so it can be recomputed, next to the reviewer's 58 under a different one -- both with zero unresolved, so the difference is convention, not coverage. Mutation evidence: 19/19 with teeth, three of them new -- restoring the bare pointer, restating it instead of composing, and importing the engine from the hand side each turn the matching test red.
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C6 的第一个 PR：改造。 按 owner 2026-07-28 裁定「退役重生成能力」，把三个工具改成只读验证，D1 基线从此 frozen-forever。本 PR 一个 runner 都不删（删除是下一个 PR，owner 要求改造与删除分开、各带评审）。
为什么这是安全的
这三个工具的重生成路径依赖 11 个旧 runner（
panel_freeze.py直接 import 它们并调其私有_load_*_panel）。删 runner 会让重生成失效——与其留一条跑起来就会坏的代码路径，不如显式退役它，让它只做验证。qt/panel_freeze.pyRegenerationRetiredError；--resume/--only仍被 parser 接受（否则用户拿到unrecognized arguments而非退役说明）docs/factors/哈希表核，gitignoredmanifest.json作第二见证互核。实跑 15/15qt/panel_reconcile.pymax_rel=0.000e+00qt/hand_anchors_engine_values.py新增
qt/frozen_manifest_doc.py（文档表解析器，header 驱动；两张表 / 零张表 / 重复 id / 坏哈希全 raise）。qt/cli.py的 11 个run-eval-*命令收敛到run-factor-eval，敲旧名 rc=1 + 迁移提示；映射从FACTOR_TO_REPORT_NAME派生，不是第二份抄写（评审进程内扰动该表验证：旧名失效、新名生效）。R16 覆盖数非降（删除 PR 的准入证）
10 文件 / 59 条逐条核销：59/59 在映射表里有行、0 条无映射；表引用的新套件 node-ID 全部收集得到；3 个家族通配数目吻合；裸引用全部唯一解析。问题数 0 ⇒ R16 前置条件成立。 结果写进
d5_property_test_migration_map.md§十，并留下常设守卫tests/test_property_test_migration_map.py（射程写进自己的 docstring：只查引用能否解析，改名会抓住、就地掏空不会）。顺带查出来的：冻结目录并不是不可变的
普查
artifacts/refactor_baseline/全部 113 个文件：exec_baseline77 +panels14 +panels_d214 +pr_c1 有 git 里的哈希覆盖，7 个没有。已分级（3 个manifest.json字节未钉但内容三方互核；3 个.md可推导；只有hand_anchors_d2.json真正无保护）。而它恰恰是唯一被实际覆盖过的：2026-07-25 一次
hand_anchor_rows重跑把 engine 比较记录冲掉了（现 20 pending / 0 compared），而填回它的工具本 PR 退役 ⇒hand_anchors --verify在真实树上永久红，措辞NOT VERIFIED (nothing recorded)与 FAILED 区分、未进任何 gates 清单。丢的是可再验证性，不是结论——D2 手算锚那条腿的结论仍在案（PR #89 + 进度文档「88 行分层手算锚 0 失配」）。
⇒ 编目现在写明：「冻结基线一个字节未写」是关于某次操作的陈述，不是关于该目录不可变的陈述。
已登记、留给删除 PR
load_anchor_rows（anchors JSON）/run_panels_mode（冻结 D1 面板）/run_anchors_mode（再读一次求 warmup 网格，--mode anchors单跑时是唯一的冻结面板读入点）。两句实话都在编目里：C5 四腿全绿是在一棵从未经哈希核对的基线上得出的；同时，今天panel_freeze --verify15/15、panel_reconcile --verify14/14max_rel=0.0⇒ 面板此刻完好，C5 结论不受影响。清单用符号名不用行号——本 PR 自己就把:1539挪成了:1542。hand_anchor_rows的 payload 只写daily_pending_engine不写daily_engine_compared（即 07-25 冲掉记录的机制）。panel_freeze/panel_reconcile依赖全部 11 个 runner 的私有 loader ⇒ 删 runner 后必然失效；而hand_anchors_engine_values只用了一个前置检查，本可改指统一 runner 继续活着 ⇒ 这一条退役真的放弃了一个未必注定失效的能力。裁定维持（owner 裁定的是目的，不是从依赖强弱推出的推论），但事实照记。评审揪出的两个 MEDIUM
① 还在教用户跑一条已退役的命令（runtime 输出，不是文档）。
hand_anchor_rows每次跑完 print(run python -m qt.hand_anchors_engine_values)——而它现在 raise；同一函数正是 07-25 冲掉记录的那个 ⇒ 用户刚把记录冲掉，程序就指他去跑死路。实现方照评审的方法扫全仓，自己又多抓 3 处（共 4 处），其中第 4 处是它新加的 author-once 测试当场咬出来的。修法是结构性的：指针 author-once 成
ENGINE_COMPARISON_POINTER+pending_engine_line()，printer 只组合不复述。hand_anchors_d2而非退役模块，这条承重：hand_anchors_engine_values→qt.panel_freeze→data.clean.schema，而手算侧绝不能载入引擎。评审独立实测：import qt.hand_anchors_d2载入 756 个模块，factors.*/data.clean.*恰好 0 个。用 AST import 检查钉住（第一版用子串检查，被一条为解释 data.clean 为何被挡在外而提到 data.clean 的注释绊倒——守卫太粗的又一例）。② 裸读清单少一处且行号已失效（见上，已改三处 + 符号名）。
教训：只扫自己改过的地方的守卫，只能确认你已经知道的事（#82 的复发）
两处 md 改对了、runtime print 漏了；评审扫全仓抓到 1 处，实现方再扫全仓又抓 3 处。清单不全 = 给自己发一张只覆盖三分之二的通行证。
另记本轮第 11 例"不可能失败的测试"：闭集删除测试的第一版用默认 fixture（只有一个 minute 因子），删掉唯一条目会触发既有的"整块为空"分支、在被检验的新检查不存在时也通过。改成 ≥2 个 minute 因子（真实形态 11→10）并断言 problems 里不含旧那句。又一次不是被测试抓住的。
Gates（lead 独立重跑，未采信 subagent 自报）
pytest -p no:warnings2701 passed ·ruffclean · phase0 锚 ic 0.9600 / annual 0.8408 ·validate-config32/32 ·exec_baseline_freeze --verify77/77 @ 45c14aa · secret 0 · 三个重生成入口 rc=1（loud） · 两个--verify仍 OK · 11 个 runner 模块 + 10 个 runner 测试文件 0 改动且仍在 · 冻结基线自 2026-07-25 起 0 写入实现方 mutation 19/19 有牙、评审自写 mutation 全部独立复现；真实数据 mutation 全程跑在
cp -r副本上，两方各自的前后快照证明冻结基线 0 写入。