diff --git a/docs/factors/d1_panel_freeze_manifest.md b/docs/factors/d1_panel_freeze_manifest.md index 03e7609..9609979 100644 --- a/docs/factors/d1_panel_freeze_manifest.md +++ b/docs/factors/d1_panel_freeze_manifest.md @@ -19,15 +19,24 @@ 的结构性预防。具体操作：checkout `3669c90`，把本分支的 `qt/panel_freeze.py` 原样放入 （它只 import、不改任何因子数学模块），再跑下方命令。**任何在 D2 之后的树上直接重跑 本工具得到的"基线"都不是基线**，与冻结哈希不一致时以本文记录的哈希为准。 -- **重跑命令**（cwd = 仓库根，缓存根就位）： +- ⚠️ **重生成能力已于 D5 C6 退役（owner 2026-07-28 裁定），本基线 frozen-forever。** + 重生成路径调的是 11 个旧 eval runner 的私有 `_load_*_panel`，C6 删除它们后这条路必然 + 失效；与其留一条跑起来就会坏的代码，不如显式退役。`python -m qt.panel_freeze` + （不带 `--verify`）现在是**可读的报错**，不是静默 no-op。**从当前树重跑本来就不合法** + （见上一条 provenance 规则），所以失去的不是复核能力。 + +- **现在能跑的是验证**（cwd = 仓库根）： ``` - /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.panel_freeze + /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.panel_freeze --verify ``` - 输出根默认 `artifacts/refactor_baseline`（`--output-root` 可改；`--resume` 语义见 - 模块 docstring——已存在的面板从冻结文件读回并**重新走 process + 对账**后才被接受， - 绝不盲信旧文件）。 + 它对**两棵**冻结树（本基线 14 个面板 + `pr_c_cutoff_fix` 1 个）逐面板重算 canonical + content hash 与整条 manifest 行（rows / 日期范围 / symbol 数 / NaN 数 / mean / std / + file sha256），与**本文 §六表格**（git 里的权威）以及 `manifest.json`（gitignored 的第二 + 见证）双向核对，任一不符即非零退出。多出一个未登记的面板文件同样判负。 + **本文的表格因此不只是记录，而是被程序读取的期望值**——改它就等于改验证结果，会出现在 + `git diff` 里。 ## 二、数据面（与十一因子评估循环同面） diff --git a/docs/factors/d5_property_test_migration_map.md b/docs/factors/d5_property_test_migration_map.md index 0a07f03..f94fe22 100644 --- a/docs/factors/d5_property_test_migration_map.md +++ b/docs/factors/d5_property_test_migration_map.md @@ -168,3 +168,54 @@ eval_peak_ridge_amount_ratio tests/` 无命中）。新套件在此是**净增** | 8 | `tests/test_factor_store_universe.py::test_min_cross_section_gate_still_bites_on_the_read_path` | ✅ | | 9 | `tests/test_exec_basis_eval.py::test_exec_basis_cli_line_survives_an_absent_metric` | ✅ | | 10 | `tests/test_factor_requires_spec_v1.py::test_shipped_factor_declarations_match_the_d0_table` | ✅ | + +## 十、C6 准入核销（2026-07-30，改造 PR 内实跑） + +R16 的门是"删旧 runner 测试文件之前，59 条逐条有映射"。本节是**在树上跑出来的**核销结果， +不是对上面表格的复述。审计脚本 `tmp/context/cc_c6_verify_only/r16_audit.py`（gitignored， +一次性）；**耐久的守卫是 `tests/test_property_test_migration_map.py`（7 条）**，它把同样三个 +问题变成常跑测试。 + +| 问题 | 方法 | 结果 | +|---|---|---| +| 旧套件真的是 10 文件 / 59 条吗 | 逐文件 `grep "^def test"` | **10 / 59** ✅（4+4+4+4+4+10+14+4+7+4，与 §零 口径一致） | +| 59 条**逐条**在本表有行吗 | 26 个 distinct 旧测试名 vs 表格第一列，再按文件展开 | **59 / 59** ✅，0 条无映射 | +| 本表引用的新套件 node-ID 今天还在吗 | 解析全文引用 → 与 `pytest --collect-only` 比对（见下「口径与出处」） | **56 / 56 全部命中** ✅ | +| 家族通配（`…_*（N 条）` 形式）的 N 属实吗 | 按前缀数实际测试数 | **3 / 3 家族数目吻合** ✅（ridge_return 3、valley_ridge 2、neutralization 2） | +| 引用是否唯一解析 | 省略文件名的裸引用按**名字**在全套件查找 | **全部唯一** ✅（无重名歧义——本仓 `tests/` 无 `__init__.py`，重名会静默丢一条） | + +**结论：R16 覆盖数非降的前置条件成立，删除 PR 可以开始。** + +### 口径与出处（评审 LOW-1 / NIT-1 的更正） + +- **本节数字实跑于本 PR 的最后一个代码 commit 之后的工作树**，`pytest --collect-only` 收集到 + **2,701** 个 node-ID。**初稿写的 2,687 在任何一个 commit 上都不成立**：那次审计跑在**它自己 + 那一批 7 条守卫测试落地之前**（`191b443` 上实为 2,694），也就是**审计跑在自己的守卫存在之前**。 + 更正连同原因一起记在这里，而不是把数字悄悄换掉。 +- ⚠️ **收集总数是背景，不是主张**——它随任何新增测试变动（本 PR 期间就走过 + 2,694 → 2,697 → 2,701）。**承重的三个数是 59/59、56/56、0 未解析**，它们与总数无关。 + 引数字时**必须连同它是在哪棵树上量的一起引**（这与本仓「报极值必须连它的窗口一起报」同形）。 +- **「56」的口径**：*去重后、家族通配展开后的 distinct 新套件 node-ID*——同一个 node-ID 在表中 + 被引多次只计一次，`::name_*（N 条）` 展开为它的 N 个成员。评审用另一套口径数到 **58**； + **两者都是 0 条未解析**，所以差的是计数约定、不是覆盖。此处把本表的口径写死，使这个数可复算。 + +**两条如实记录（都不改变结论）**： + +- 审计脚本第一版把 `pytest --collect-only -q` 传成了 `-qq`（`pyproject.toml` 已含 + `addopts="-q"`，见 handoff 陷阱 2），node-ID 列表被吞掉，于是报出一大批"引用了但没收集到"。 + **是"与预期矛盾的观察"暴露的，不是被任何测试抓住的。** +- 第二版用 `(?!\*)` 排除家族通配，被**回溯**绕过：正则把名字末尾的下划线让出来，改匹配一个更短 + 但同样错误的名字，于是三个家族 stem 仍被当成缺失的测试。改成 `(?![A-Za-z0-9_*])` 才成立。 + 守卫里保留了这条注释，因为下一个人写同样的正则会踩同样的坑。 + +**守卫的射程（写进它自己的 docstring）**：它检查引用**能否解析**，不检查被引用的测试是否 +仍然断言本表声称的那条性质。**改名会被抓住，就地掏空不会。** + +**`test_no_legacy_runner_test_is_missing_from_the_map` 在旧文件被删后会变成空过**——这是设计 +如此：它的职责是"只要旧文件还在，就不许存在没被映射的旧测试"，而不是让旧文件活下去；一个会 +因为删除而变红的守卫，对本表要授权的那个 PR 就是个陷阱。 + +**守卫立刻咬到了本节自己**：§十初稿在表格里用一个虚构的双冒号引用当占位符举例，守卫把它当成 +真引用、判定"引用了但不存在"。占位符已改成不含双冒号的描述。**这不是误报**——本表的引用格式 +就是双冒号加测试名，任何写成那个形状的东西都应该指向一个真实测试。写这一段本身又踩了一次 +（复述时把占位符原样抄了回来），第二次才改对。 diff --git a/docs/factors/d5_runner_difference_catalogue.md b/docs/factors/d5_runner_difference_catalogue.md index e8f22f7..bbeb8a9 100644 --- a/docs/factors/d5_runner_difference_catalogue.md +++ b/docs/factors/d5_runner_difference_catalogue.md @@ -231,12 +231,12 @@ aligned_base = sign * gross - base_cost （`tests/__init__.py` 收集陷阱）同一接口。**非测试模块共四个**（实测 `grep -rln "qt\.eval_" --include=*.py | grep -v ^./tests/`）： -| 模块 | 用途 | C6 处置 | +| 模块 | 用途 | C6 处置（**已执行**，改造 PR，2026-07-30） | |---|---|---| -| `qt/cli.py` | 11 个 `_cmd_run_eval_*` 子命令（§五 C13） | 随统一 runner 收敛为一个子命令 | -| **`qt/panel_freeze.py`** | **生产 D1 冻结基线的工具**——调各 runner 私有 `_load_*_panel` + `_build_book_factors`，「零公式重抄」正是靠 import 它们实现的 | **删 runner 会打断「重新生成 / 重新验证 D1 基线」的能力** | -| **`qt/panel_reconcile.py`** | D2 逐格对账工具（同上依赖） | 同上 | -| **`qt/hand_anchors_engine_values.py`** | D2 手算锚的引擎侧取值 | 同上 | +| `qt/cli.py` | 11 个 `_cmd_run_eval_*` 子命令（§五 C13） | ✅ 收敛为 `run-factor-eval` 一个子命令；11 个旧命令名保留**迁移提示**（argparse 的 `invalid choice` 只会让人以为自己打错了） | +| **`qt/panel_freeze.py`** | **生产 D1 冻结基线的工具**——调各 runner 私有 `_load_*_panel` + `_build_book_factors`，「零公式重抄」正是靠 import 它们实现的 | ✅ **改 verify-only**：重生成 loud raise，`--verify` 逐面板重算 canonical hash + 整条 manifest 行，对着 **git 里的**哈希表核 | +| **`qt/panel_reconcile.py`** | D2 逐格对账工具（同上依赖） | ✅ **改 verify-only**：重建退役，**比较仍然真跑**（它本来就不需要 runner）——两侧各自从盘上独立读回、逐格重导出 D2 结论 | +| **`qt/hand_anchors_engine_values.py`** | D2 手算锚的引擎侧取值 | ✅ **改 verify-only**：重算退役，`--verify` 用记录自身的 hand/engine 数重导出 rel_diff / ok / all_ok_daily | ⚠️ **后三个全是 D1/D2 的验收工具**，而 **D5 面板腿的比较对象正是 `panel_freeze.py` 的产物**。 按设计 v3.2 §五第 4 腿的 **provenance 规则**，基线只许从钉住的 pre-D2 SHA 重新生成—— @@ -251,6 +251,17 @@ import 的 runner loader，即便统一 runner 已上线）、**要么显式记 **必须同批更新**——否则就是本项目 #76/#78/#82 那条形态的又一次复发（文档教人跑一条已经 跑不了的命令）。 +**已落实（2026-07-30 改造 PR）**：走的是第二条路——owner 2026-07-28 裁定**退役重生成能力**， +**D1/D2 基线自此不可从本仓再生**，只能依赖已冻结的 artifact + 哈希。三个工具都不删，改成 +只读验证；重生成入口是 **loud raise 而不是静默 no-op**（红线 #9）。要求 ② 同批完成： +`d1_panel_freeze_manifest.md` §一 与 `pr_c_cutoff_fix_reference_panel.md` §四 的「重跑命令」 +已改为退役声明 + 验证命令。 + +**这条不对称损失的账，最后是这样结的**：担心的是「工具已被删、想复核也无从下手」。现在复核 +能力**变强了而不是变弱**——旧的重生成路径从来不能用于复核（provenance 规则禁止从当前树重跑， +从被验证的树重建基线正是本仓犯过一次的空对账），而新的 `--verify` 是一条以前根本不存在的路： +15 个面板逐一重算 canonical hash 与整条 manifest 行，对着 **git 里的**期望值核，3 秒跑完。 + ## 七、与 C5 对账的接口 **本表 §二/§三 = 允许出现的差异白名单；§四 = 必须逐值一致的项；§五 = 已判定归一（对账中 @@ -849,3 +860,110 @@ description 相应改名（`REUSED from ...` → `SHARED taxonomy in ...`）； `classify_panel_differences` / `classify_anchor_row` / `run_panels_mode` / `run_anchors_mode` / `load_anchor_rows` / `frozen_panel_path` / `_build_bundle` **一个字节未变** ⇒ panels 与 anchors 的结论不受影响，**只重跑 reports 腿**。 + +## 六之三、C6 改造期登记的三件事（lead 2026-07-30 裁定，本节只登记不修） + +### (1) D2 手算锚的 engine 比较记录已被冲掉——**丢的是可再验证性，不是结论** + +`artifacts/refactor_baseline/hand_anchors_d2.json` 现在只有 20 条 `daily_pending_engine`、 +**0 条 `daily_engine_compared`**：2026-07-25 18:04 一次 `python -m qt.hand_anchor_rows` 重跑 +（PR #103 的 jump 截断修正配套）把 engine 侧的比较结果整段覆盖掉了，而填回它的 +`qt.hand_anchors_engine_values` 已在 C6 退役。 + +**必须分清的两件事**： + +- **D2 手算锚那条腿的结论仍然在案**——PR #89 正文与 + [`docs/progress/07_factor_layer_refactor.md`](../progress/07_factor_layer_refactor.md) 的 D2 条目 + 白纸黑字记着「**88 行分层手算锚 0 失配**（脚本运行时守卫禁 import 引擎，曾自抓 3 个口径错）」。 + D2 的验收**不因此动摇**。 +- **丢的是「今天再验一次」的能力**。四条 daily 锚（momentum/reversal/liquidity/overnight_mom） + 的 engine 侧数字已不在盘上，退役后也无法重算。 + +因此 `python -m qt.hand_anchors_engine_values --verify` **在今天的树上必然非零退出**，措辞是 +`NOT VERIFIED (nothing recorded)` 而**不是** `FAILED`——"没验"与"验了不过"是两回事，退出码只有 +一位，措辞必须补上这个区分。 + +⚠️ **它不进标准 gates 清单**（lead 裁定）：一条永远不转绿的红，只会训练人忽略红。 + +### (2) 冻结目录里有 7 个**不受任何哈希保护**的文件 + +上一条暴露的问题比它自己大：`hand_anchors_d2.json` 就在 +`artifacts/refactor_baseline/`——那个被反复称作「绝不可覆盖」的目录——**而它确实被覆盖了**。 +于是普查了整个目录（113 个文件）： + +| 类别 | 数量 | 保护来源 | +|---|---|---| +| `exec_baseline/*` | 77 | **git**：`docs/factors/d5_exec_baseline_manifest.json` 逐文件 sha256 | +| `panels/*.parquet` | 14 | **git**：`d1_panel_freeze_manifest.md` §六（canonical + file sha） | +| `panels_d2/*.parquet` | 14 | **git**：同上表（canonical，经 `panel_reconcile --verify`） | +| `pr_c_cutoff_fix/panels/*.parquet` | 1 | **git**：`pr_c_cutoff_fix_reference_panel.md` §五（canonical） | +| **无任何哈希覆盖** | **7** | —— | + +那 7 个是：`hand_anchors_d2.json`、`manifest.json`、`manifest.md`、`manifest_d2.json`、 +`reconcile_d2.md`、`pr_c_cutoff_fix/manifest.json`、`pr_c_cutoff_fix/manifest.md`。 +**它们之间还有程度差别，不要一概而论**： + +- `manifest.json` / `manifest_d2.json` / `pr_c manifest.json` —— **自身字节未被钉，但内容受约束**： + `panel_freeze --verify` 把每一行（canonical / file sha / rows / 日期 / symbol 数 / NaN 数 / + mean / std）与 git 文档、与盘上面板三方互核，header 的 `producing_git_sha` 也与文档核。 + 改内容会被抓，改字节（重排、格式化）不会。 +- `manifest.md` / `reconcile_d2.md` / `pr_c manifest.md` —— 渲染产物，内容可从别处推导，风险低。 +- **`hand_anchors_d2.json` —— 真正无保护**：没有任何一处记着它该长什么样，而它是唯一一个 + **已被实际覆盖**的。 + +**要记住的那句话**：「本次改动冻结基线一个字节未写」是**关于今天这次操作**的陈述， +**不是关于该目录不可变的陈述**。目录本身没有强制不可变性——上面这 7 个文件谁都能改， +其中 1 个已经被改过。 + +### (3) C5 对账 harness 读这两样东西时**不核任何哈希** + +**三处**（`qt/factor_eval_reconcile.py`，**按符号名记，不记行号**——行号会随任何平移失效， +本 PR 自己就是证据：初稿写的 `:1539` 取自 `main`，而本分支的 `711c0a8` 把该文件平移了 +3）： + +| 符号 | 读什么 | 何时是唯一读入点 | +|---|---|---| +| `run_panels_mode` | `pd.read_parquet(frozen_panel_path(...))` 冻结 D1 面板 | `--mode panels` | +| `load_anchor_rows` | `hand_anchors_d2.json` | `--mode anchors` | +| `run_anchors_mode` | 经 `frozen_panel_path()` 再读一次冻结 D1 面板，求 warmup 日期网格 | **`--mode anchors` 单跑时，这是唯一的冻结面板读入点** | + +⚠️ 第三处是评审补的，初稿漏了。**清单不全 = 那一处留着永远不验**——删除 PR 的处置正是 +「把 verify 接进读入点」，接两处而漏一处，等于给自己发一张只覆盖三分之二的通行证。 + +**两句话都必须写下，缺一句都会误导**： + +- **C5 的四腿全绿，是在一棵从未经过哈希核对的基线上得出的。** 而 (2) 已经证明这个目录**能**被写， + 所以这不是假想风险。 +- **同时**：今天 `python -m qt.panel_freeze --verify` 对 15 个面板逐一重算 canonical hash 与整条 + manifest 行、**15/15 通过**，`panel_reconcile --verify` 14/14 `max_rel=0.0` + ⇒ **面板此刻完好，C5 的结论不受影响**。 + +**处置**：修在**下一个 PR（删除 PR）**——那个 PR 的主题正是「让 frozen-forever 名副其实」， +把 `qt.panel_freeze.verify_frozen_panels` 接进 C5 harness 的**三个**读入点正属于它。 +**本 PR 不动 C5 harness。** + +### (3之二) `hand_anchor_rows` 的 payload 只写 pending、不写 compared —— 已知后果，本 PR 不改行为 + +`qt/hand_anchor_rows.py::run_hand_anchors` 每次成功跑完都会把 `hand_anchors_d2.json` 整份重写， +payload 只含 `daily_pending_engine`，**不含 `daily_engine_compared`** ⇒ **它正是 2026-07-25 冲掉 +engine 比较记录的那个动作**。而在本 PR 之前，它跑完还会 print 「run python -m +qt.hand_anchors_engine_values」—— **用户刚把记录冲掉，程序就指他去跑一条现在会 raise 的命令**。 + +- **那行 print 已在本 PR 改掉**（改为指向 `--verify` 并说明记录已无法重建；措辞 + author-once 于 `qt.hand_anchors_d2.ENGINE_COMPARISON_POINTER`，printer 只组合不复述）。 +- **「`hand_anchor_rows` 是否应当被禁止覆盖冻结目录里的记录」属于行为变更**，与 + §六之三(2) 的「该目录没有强制不可变性」是同一件事 ⇒ **并进删除 PR**（与上面三个读入点的 + verify 接线一起做）。**本 PR 不改它的写入行为。** + +### (4) 第三个工具的退役理由：一条如实记录的不对称 + +C6 退役的三个工具对 11 个旧 runner 的依赖**不是一个量级**： + +| 工具 | 对旧 runner 的依赖 | 删 runner 后是否必然失效 | +|---|---|---| +| `qt/panel_freeze.py` | `minute_recipes()` import **全部 11 个模块**并调其私有 `_load_*_panel` | **是** | +| `qt/panel_reconcile.py` | 走 `qt.panel_freeze` 的 recipes，同上 | **是** | +| `qt/hand_anchors_engine_values.py` | 只用了 `qt.eval_jump_amount_corr._check_preconditions` **一个前置检查** | **否**——改指统一 runner 即可继续活着 | + +**裁定维持**（owner 2026-07-28 的裁定是「退役重生成能力、基线 frozen-forever」，那是**目的**， +不是从「依赖 11 个 runner」推出来的推论；依赖强弱不改变裁定）。但如实记下： +**第三条退役真的放弃了一个未必注定失效的能力**，这一点已单独提给 owner。 diff --git a/docs/factors/pr_c_cutoff_fix_reference_panel.md b/docs/factors/pr_c_cutoff_fix_reference_panel.md index cdced0b..19e0b20 100644 --- a/docs/factors/pr_c_cutoff_fix_reference_panel.md +++ b/docs/factors/pr_c_cutoff_fix_reference_panel.md @@ -41,7 +41,7 @@ | 产出 | D1 冻结 run（`main` @ `3669c90`），见 `d1_panel_freeze_manifest.md` | 见下 §四 | | 引擎 | pre-D2 runner loader | **同一条 runner loader**（`qt.panel_freeze` + `--only`） | | 用途 | **已发布内容的忠实记录**；D2 逐位对账的历史参照 | **D5 面板腿对该因子的有效参照** | -| 是否可被覆盖 | **否**（本次改动只新增，一个字节没动） | 可再生（命令在 §四） | +| 是否可被覆盖 | **否**（本次改动只新增，一个字节没动） | **否**（C6 起同样 frozen-forever；产生方式已退役，见 §四） | **为什么「旧引擎 + 截断输入」是干净的 refactor-only 对照**：本次改动是**纯输入截断**—— 被喂进相关系数计算的 bar 少了一批，`compute_jump_amount_corr` 里从 `amp` 到 Pearson @@ -52,10 +52,17 @@ **不要**把两份面板放进同一次对账去「取平均」或「看哪个更接近」：它们是两个不同定义的因子值， 不是同一个量的两次测量。 -## 四、新参照面板的产生方式（可复跑） +## 四、新参照面板的产生方式（**已于 D5 C6 退役**，此处存档） + +⚠️ **下面这条命令不再可用**：owner 2026-07-28 裁定退役重生成能力，`--only` 连同整个冻结 +路径一并退役（它依赖 11 个旧 runner 的私有 loader，C6 删除它们后必然失效）。本面板与 D1 +基线一样 **frozen-forever**；验证走 +`python -m qt.panel_freeze --verify`，它会把本文 §五 表格里的 `canonical_sha256` 当作 +git 里的权威期望值，对盘上这个面板逐格重算核对。原命令留在这里是 provenance 的一部分—— +它记录的是这份面板**当初怎么来的**，不是现在怎么再来一次。 ``` -cd # 缓存根 artifacts/cache/tushare/v1 就位 +cd # 缓存根 artifacts/cache/tushare/v1 就位 [RETIRED, C6] /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.panel_freeze \ --config config/phase_c_jump_amount_corr.yaml \ --output-root artifacts/refactor_baseline/pr_c_cutoff_fix \ diff --git a/qt/cli.py b/qt/cli.py index e8f59da..7a28488 100644 --- a/qt/cli.py +++ b/qt/cli.py @@ -203,195 +203,6 @@ def _cmd_run_phase_i5d_intraday_groups(args: argparse.Namespace) -> int: return 0 -def _cmd_run_eval_jump_amount_corr(args: argparse.Namespace) -> int: - """Run the two real jump-amount-corr factor evaluations (cache-only) + reports.""" - from qt.eval_jump_amount_corr import run_eval_jump_amount_corr - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_jump_amount_corr(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-jump-amount-corr: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_minute_ideal_amplitude(args: argparse.Namespace) -> int: - """Run the two real minute-ideal-amplitude factor evaluations (cache-only) + reports.""" - from qt.eval_minute_ideal_amplitude import run_eval_minute_ideal_amplitude - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_minute_ideal_amplitude(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-minute-ideal-amplitude: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_amp_marginal_anomaly_vol(args: argparse.Namespace) -> int: - """Run the two real amp-marginal-anomaly-vol factor evaluations (cache-only) + reports.""" - from qt.eval_amp_marginal_anomaly_vol import run_eval_amp_marginal_anomaly_vol - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_amp_marginal_anomaly_vol(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-amp-marginal-anomaly-vol: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_volume_peak_count(args: argparse.Namespace) -> int: - """Run the two real volume-peak-count factor evaluations (cache-only) + reports.""" - from qt.eval_volume_peak_count import run_eval_volume_peak_count - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_volume_peak_count(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-volume-peak-count: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_intraday_amp_cut(args: argparse.Namespace) -> int: - """Run the two real intraday-amp-cut factor evaluations (cache-only) + reports.""" - from qt.eval_intraday_amp_cut import run_eval_intraday_amp_cut - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_intraday_amp_cut(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-intraday-amp-cut: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_peak_interval_kurtosis(args: argparse.Namespace) -> int: - """Run the two real peak-interval-kurtosis evaluations (cache-only) + reports.""" - from qt.eval_peak_interval_kurtosis import run_eval_peak_interval_kurtosis - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_peak_interval_kurtosis(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-peak-interval-kurtosis: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_valley_relative_vwap(args: argparse.Namespace) -> int: - """Run the two real valley-relative-VWAP evaluations (cache-only) + reports.""" - from qt.eval_valley_relative_vwap import run_eval_valley_relative_vwap - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_valley_relative_vwap(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-valley-relative-vwap: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - def _fmt_metric(value: object, spec: str = ".4f") -> str: """Format a metric that a SKIPPED section may legitimately leave absent. @@ -409,173 +220,6 @@ def _fmt_metric(value: object, spec: str = ".4f") -> str: return str(value) -def _cmd_run_eval_ridge_minute_return(args: argparse.Namespace) -> int: - """Run the two real ridge-minute-return evaluations (cache-only) + reports.""" - from qt.eval_ridge_minute_return import run_eval_ridge_minute_return - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_ridge_minute_return(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - net = ( - " ".join( - f"{m:g}x={v:+.6f}" for m, v in sorted(nb["net_long_short_by_cost"].items()) - ) - or "n/a" - ) - print( - f"OK run-eval-ridge-minute-return: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - # Ridge bars are structurally scarce and the return guard narrows them further; - # the realized distribution and the day-validity rate are surfaced so a coverage - # regression is visible. - f"{result.ridge_coverage.render()}\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - # sign=-1 makes the frozen layer's aligned_spread_* unreliable (it adds costs back - # instead of deducting them); the sign-agnostic net spreads are printed instead. - f"net long-short by cost (aligned_spread_* UNRELIABLE at sign=-1): {net}\n" - f"turnover={_fmt_metric(nb['long_short_turnover'])} " - f"rank_autocorr_lag1={_fmt_metric(nb['rank_autocorr_lag1'])} " - f"half_life={_fmt_metric(nb['half_life_periods'], '.2f')}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_peak_ridge_amount_ratio(args: argparse.Namespace) -> int: - """Run the two real peak/ridge amount-ratio evaluations (cache-only) + reports.""" - from qt.eval_peak_ridge_amount_ratio import run_eval_peak_ridge_amount_ratio - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_peak_ridge_amount_ratio(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - net = ( - " ".join( - f"{m:g}x={v:+.6f}" for m, v in sorted(nb["net_long_short_by_cost"].items()) - ) - or "n/a" - ) - print( - f"OK run-eval-peak-ridge-amount-ratio: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - # PEAK bars are the structurally scarce leg of THIS pair (the reverse of PR-J); - # the realized distribution, the day-validity rate and the counterfactual at a - # peak floor of 10 are surfaced so the lowered gate is a number, not a claim. - f"{result.peak_coverage.render()}\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - # sign=+1, so the frozen layer's aligned-spread cost-sign defect does NOT apply - # to this factor; net spreads are still printed for sibling comparability. - f"net long-short by cost: {net}\n" - f"ic_rank={_fmt_metric(nb['ic_mean'])} " - f"ic_pearson={_fmt_metric(nb['ic_pearson_mean'])} " - f"monotonicity={_fmt_metric(nb['monotonicity_spearman'])}\n" - f"turnover={_fmt_metric(nb['long_short_turnover'])} " - f"rank_autocorr_lag1={_fmt_metric(nb['rank_autocorr_lag1'])} " - f"half_life={_fmt_metric(nb['half_life_periods'], '.2f')}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_valley_price_quantile(args: argparse.Namespace) -> int: - """Run the two real valley-price-quantile evaluations (cache-only) + reports.""" - from qt.eval_valley_price_quantile import run_eval_valley_price_quantile - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_valley_price_quantile(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - net = ( - " ".join( - f"{m:g}x={v:+.6f}" for m, v in sorted(nb["net_long_short_by_cost"].items()) - ) - or "n/a" - ) - print( - f"OK run-eval-valley-price-quantile: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - # The reversal neutralization is the structural novelty of this factor; a - # neutralization that silently ate the panel shows up here as a number. - # render() is the single home of this line (catalogue section 3 one-site - # normalization; it used to be an inline f-string here). - f"{result.neutralization.render()}\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - # sign=+1, so the frozen layer's aligned-spread cost-sign defect does NOT apply - # to this factor; net spreads are still printed for sibling comparability. - f"net long-short by cost: {net}\n" - # The PR-K review's regularity, first tested here on a positive-sign factor. - f"ic_rank={_fmt_metric(nb['ic_mean'])} " - f"ic_pearson={_fmt_metric(nb['ic_pearson_mean'])} " - f"monotonicity={_fmt_metric(nb['monotonicity_spearman'])}\n" - f"turnover={_fmt_metric(nb['long_short_turnover'])} " - f"rank_autocorr_lag1={_fmt_metric(nb['rank_autocorr_lag1'])} " - f"half_life={_fmt_metric(nb['half_life_periods'], '.2f')}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - -def _cmd_run_eval_valley_ridge_vwap_ratio(args: argparse.Namespace) -> int: - """Run the two real valley/ridge VWAP-ratio evaluations (cache-only) + reports.""" - from qt.eval_valley_ridge_vwap_ratio import run_eval_valley_ridge_vwap_ratio - from qt.exec_basis_eval import format_exec_basis_line - - try: - result = run_eval_valley_ridge_vwap_ratio(args.config) - except (ConfigError, ValueError, FileNotFoundError) as exc: - print(f"ERROR: {exc}", file=sys.stderr) - return 1 - nb, wb = result.no_book_metrics, result.with_book_metrics - print( - f"OK run-eval-valley-ridge-vwap-ratio: covered={result.covered_symbols}/" - f"{result.requested_symbols}, stk_mins_live_calls={result.minute_live_calls}, " - f"factor_rows={result.factor_rows} ({result.elapsed:.1f}s)\n" - # Ridge bars are structurally scarce; the realized distribution and the - # day-validity rate are surfaced so a coverage regression is visible. - f"{result.ridge_coverage.render()}\n" - f"no-book: {nb['deployment']} (predictive={nb['predictive']}) " - f"ic_mean={nb['ic_mean']:.4f} ic_ir={nb['ic_ir']:.3f} N_eff={nb['effective_samples']:.1f}\n" - f"with-book: {wb['deployment']} (incremental={wb['incremental']}) " - f"incr_ic_ir={wb['incremental_ic_ir']:.3f}\n" - f"reports: {result.reports.no_book_md} | {result.reports.with_book_md}\n" - f"dashboards: {result.reports.no_book_dashboard} | " - f"{result.reports.with_book_dashboard}" - ) - print(format_exec_basis_line(result.exec_basis)) - return 0 - - def _cmd_run_factor_eval(args: argparse.Namespace) -> int: """Run the unified exec-only factor evaluation (D5 C4).""" from qt.exec_basis_eval import format_exec_basis_line @@ -832,83 +476,6 @@ def build_parser() -> argparse.ArgumentParser: p_i5d.add_argument("--config", required=True, help="Path to the YAML config.") p_i5d.set_defaults(func=_cmd_run_phase_i5d_intraday_groups) - p_jac = sub.add_parser( - "run-eval-jump-amount-corr", - help="Run the first real factor evaluation (jump-amount-corr, CSI500, cache-only).", - ) - p_jac.add_argument("--config", required=True, help="Path to the YAML config.") - p_jac.set_defaults(func=_cmd_run_eval_jump_amount_corr) - - p_mia = sub.add_parser( - "run-eval-minute-ideal-amplitude", - help="Run the minute-ideal-amplitude factor evaluation (CSI500, cache-only).", - ) - p_mia.add_argument("--config", required=True, help="Path to the YAML config.") - p_mia.set_defaults(func=_cmd_run_eval_minute_ideal_amplitude) - - p_amav = sub.add_parser( - "run-eval-amp-marginal-anomaly-vol", - help="Run the amp-marginal-anomaly-vol factor evaluation (CSI500, cache-only).", - ) - p_amav.add_argument("--config", required=True, help="Path to the YAML config.") - p_amav.set_defaults(func=_cmd_run_eval_amp_marginal_anomaly_vol) - - p_vpc = sub.add_parser( - "run-eval-volume-peak-count", - help="Run the volume-peak-count factor evaluation (CSI500, cache-only).", - ) - p_vpc.add_argument("--config", required=True, help="Path to the YAML config.") - p_vpc.set_defaults(func=_cmd_run_eval_volume_peak_count) - - p_iac = sub.add_parser( - "run-eval-intraday-amp-cut", - help="Run the intraday-amp-cut factor evaluation (CSI500, cache-only).", - ) - p_iac.add_argument("--config", required=True, help="Path to the YAML config.") - p_iac.set_defaults(func=_cmd_run_eval_intraday_amp_cut) - - p_pik = sub.add_parser( - "run-eval-peak-interval-kurtosis", - help="Run the peak-interval-kurtosis factor evaluation (CSI500, cache-only).", - ) - p_pik.add_argument("--config", required=True, help="Path to the YAML config.") - p_pik.set_defaults(func=_cmd_run_eval_peak_interval_kurtosis) - - p_vrv = sub.add_parser( - "run-eval-valley-relative-vwap", - help="Run the valley-relative-VWAP factor evaluation (CSI500, cache-only).", - ) - p_vrv.add_argument("--config", required=True, help="Path to the YAML config.") - p_vrv.set_defaults(func=_cmd_run_eval_valley_relative_vwap) - - p_vrr = sub.add_parser( - "run-eval-valley-ridge-vwap-ratio", - help="Run the valley/ridge VWAP-ratio factor evaluation (CSI500, cache-only).", - ) - p_vrr.add_argument("--config", required=True, help="Path to the YAML config.") - p_vrr.set_defaults(func=_cmd_run_eval_valley_ridge_vwap_ratio) - - p_rmr = sub.add_parser( - "run-eval-ridge-minute-return", - help="Run the ridge-minute-return factor evaluation (CSI500, cache-only).", - ) - p_rmr.add_argument("--config", required=True, help="Path to the YAML config.") - p_rmr.set_defaults(func=_cmd_run_eval_ridge_minute_return) - - p_vpq = sub.add_parser( - "run-eval-valley-price-quantile", - help="Run the valley-price-quantile factor evaluation (CSI500, cache-only).", - ) - p_vpq.add_argument("--config", required=True, help="Path to the YAML config.") - p_vpq.set_defaults(func=_cmd_run_eval_valley_price_quantile) - - p_pra = sub.add_parser( - "run-eval-peak-ridge-amount-ratio", - help="Run the peak/ridge amount-ratio factor evaluation (CSI500, cache-only).", - ) - p_pra.add_argument("--config", required=True, help="Path to the YAML config.") - p_pra.set_defaults(func=_cmd_run_eval_peak_ridge_amount_ratio) - p_fe = sub.add_parser( "run-factor-eval", help="Run the UNIFIED exec-only factor evaluation (D5 C4, cache-only).", @@ -961,8 +528,55 @@ def build_parser() -> argparse.ArgumentParser: return parser +#: Prefix of the eleven per-factor evaluation subcommands retired in D5 C6. +RETIRED_EVAL_PREFIX = "run-eval-" + + +def retired_eval_commands() -> dict[str, str]: + """Retired subcommand name -> the factor id ``run-factor-eval`` wants. + + DERIVED, not transcribed: the old command names were the frozen exec report + names with underscores turned into dashes, and that factor-id/report-name + map already exists as one closed table. Restating eleven pairs here would be + a second copy, and the copy in the CLI is the one nobody would update. + """ + from qt.factor_eval_reconcile import FACTOR_TO_REPORT_NAME + + return { + f"{RETIRED_EVAL_PREFIX}{report_name.replace('_', '-')}": factor_id + for factor_id, report_name in FACTOR_TO_REPORT_NAME.items() + } + + +def _retired_command_hint(name: str) -> str | None: + """The migration message for a retired subcommand, or None if unknown. + + argparse's own answer to a removed subcommand is ``invalid choice``, which + tells an operator that they typed something wrong — not that the command was + replaced, nor by what. The eleven per-factor runners collapsed into ONE + parameterised command, so the mapping is mechanical and worth printing. + """ + factor_id = retired_eval_commands().get(name) + if factor_id is None: + return None + return ( + f"{name} was retired in D5 C6: the eleven per-factor evaluation commands " + f"collapsed into one. Use instead:\n" + f" python -m qt.cli run-factor-eval --config " + f"--factor {factor_id} [--book-mode decision|close]\n" + "It is exec-only (the close basis left the evaluation contract in D5) and " + "writes factor_eval_* artifacts rather than eval_*." + ) + + def main(argv: list[str] | None = None) -> int: """Parse args and dispatch. Returns a process exit code.""" + argv = list(sys.argv[1:] if argv is None else argv) + if argv and argv[0].startswith(RETIRED_EVAL_PREFIX): + hint = _retired_command_hint(argv[0]) + if hint is not None: + print(f"ERROR: {hint}", file=sys.stderr) + return 1 parser = build_parser() args = parser.parse_args(argv) return args.func(args) diff --git a/qt/factor_eval_reconcile.py b/qt/factor_eval_reconcile.py index 6fe225a..8c8b1a5 100644 --- a/qt/factor_eval_reconcile.py +++ b/qt/factor_eval_reconcile.py @@ -379,7 +379,10 @@ class is CEILED at ``(lookback_depth - 1) x |frozen symbols|`` cells #: factor_id -> the frozen exec artifact's report name (qt.exec_baseline_freeze #: FACTORS). Closed map: an unknown factor id is a readable error, never a guess. -_FACTOR_TO_REPORT_NAME: dict[str, str] = { +#: PUBLIC because it is the single source for the retired-command hints in +#: qt.cli too (D5 C6) -- the eleven old ``run-eval-*`` names were these report +#: names with dashes, so the CLI derives them here rather than keeping a copy. +FACTOR_TO_REPORT_NAME: dict[str, str] = { "jump_amount_corr_20": "jump_amount_corr", "minute_ideal_amp_10": "minute_ideal_amplitude", "amp_marginal_anomaly_vol_20": "amp_marginal_anomaly_vol", @@ -392,7 +395,7 @@ class is CEILED at ``(lookback_depth - 1) x |frozen symbols|`` cells "valley_price_quantile_20": "valley_price_quantile", "peak_ridge_amount_ratio_20": "peak_ridge_amount_ratio", } -assert set(_FACTOR_TO_REPORT_NAME.values()) == set(REPORT_FACTOR_NAMES) +assert set(FACTOR_TO_REPORT_NAME.values()) == set(REPORT_FACTOR_NAMES) #: Registered JSON additions (catalogue §七 spec 16->20 keys; §七之二 contract #: v1.0/v1.1). ``corrections`` and ``spec.requires`` are matched as PREFIXES @@ -1477,7 +1480,7 @@ def load_anchor_rows(factor_id: str, repo_root: Path) -> list[dict]: # --------------------------------------------------------------------------- # def _report_name(factor_id: str) -> str: try: - return _FACTOR_TO_REPORT_NAME[factor_id] + return FACTOR_TO_REPORT_NAME[factor_id] except KeyError: raise ReconciliationError( f"{factor_id!r} has no frozen exec artifact name mapping." diff --git a/qt/frozen_manifest_doc.py b/qt/frozen_manifest_doc.py new file mode 100644 index 0000000..0661b66 --- /dev/null +++ b/qt/frozen_manifest_doc.py @@ -0,0 +1,180 @@ +"""Parsers for the git-tracked frozen-baseline manifest DOCUMENTS. + +The bulk artifacts of the factor-layer refactor baselines are gitignored; their +expected hashes are authored in Markdown documents under ``docs/factors/`` that +call themselves the authoritative manifest. Keeping the hashes in git and the +bytes out of it is what gives verification teeth: whoever regenerates a frozen +tree together with its machine manifest still cannot move the expectations +without it showing up in ``git diff``. + +So the expectations are READ FROM those documents rather than transcribed into +code — a transcription is a second copy, and the copy in code is the one nobody +updates. + +Parsing is header-driven and fail-closed. The table is located by requiring the +columns ``factor_id`` and ``canonical_sha256`` (documents carry several other +tables), every other column is read BY NAME, and anything ambiguous — two +matching tables, none, a duplicate factor id, a malformed hash — is a readable +error. A parser that silently returned an empty expectation set would make the +verifier pass everything. + +Consumers: ``qt.panel_freeze`` (D1 baseline + the PR-C reference panel) and +``qt.panel_reconcile`` (the D2 rebuild, which is expected to be bit-identical to +the D1 baseline and is therefore verified against the SAME document). +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from pathlib import Path + + +_HEX64 = re.compile(r"^[0-9a-f]{64}$") +_SHA40 = re.compile(r"[0-9a-f]{40}") +#: Cell decorations the documents legitimately carry (bold emphasis on the +#: numbers a reader should notice, code ticks around hashes). +_CELL_NOISE = re.compile(r"[`*]") + + +@dataclass(frozen=True) +class DocManifestRow: + """One factor's expectations as authored in the git-tracked manifest doc. + + Fields absent from a given document are ``None`` — the PR-C reference table + carries no ``kind`` or ``file_sha256`` column, and a verifier that invented + values for them would be checking its own invention. + """ + + factor_id: str + canonical_sha256: str + kind: str | None = None + rows: int | None = None + n_symbols: int | None = None + n_nan: int | None = None + mean: float | None = None + std: float | None = None + file_sha256: str | None = None + + +def _split_table_row(line: str) -> list[str]: + cells = line.strip().strip("|").split("|") + return [_CELL_NOISE.sub("", cell).strip() for cell in cells] + + +def _is_separator(cells: list[str]) -> bool: + return bool(cells) and all(set(cell) <= set("-: ") and "-" in cell for cell in cells) + + +def parse_doc_manifest(path: Path | str) -> dict[str, DocManifestRow]: + """Parse the factor-hash table out of a git-tracked manifest document. + + Header-driven, not position-driven: the table is located by requiring the + columns ``factor_id`` and ``canonical_sha256``, and every other column is + read by NAME. Two matching tables, zero matching tables, a duplicate factor + id, or a hash that is not 64 lowercase hex are all readable errors — a + verifier whose expectations parsed to an empty dict would "pass" everything. + """ + path = Path(path) + lines = path.read_text(encoding="utf-8").splitlines() + tables: list[dict[str, DocManifestRow]] = [] + index = 0 + while index < len(lines) - 1: + cells = _split_table_row(lines[index]) if "|" in lines[index] else [] + if ( + "factor_id" in cells + and "canonical_sha256" in cells + and "|" in lines[index + 1] + and _is_separator(_split_table_row(lines[index + 1])) + ): + columns = {name: position for position, name in enumerate(cells)} + rows: dict[str, DocManifestRow] = {} + cursor = index + 2 + while cursor < len(lines) and lines[cursor].lstrip().startswith("|"): + values = _split_table_row(lines[cursor]) + cursor += 1 + if len(values) != len(cells): + raise ValueError( + f"{path.name}: manifest table row has {len(values)} cells but " + f"the header has {len(cells)}: {values!r}" + ) + row = _doc_row(path, columns, values) + if row.factor_id in rows: + raise ValueError( + f"{path.name}: duplicate factor id {row.factor_id!r} in the " + "manifest table." + ) + rows[row.factor_id] = row + if not rows: + raise ValueError(f"{path.name}: manifest table has no data rows.") + tables.append(rows) + index = cursor + continue + index += 1 + if len(tables) != 1: + raise ValueError( + f"{path.name}: expected exactly ONE table carrying factor_id + " + f"canonical_sha256; found {len(tables)}. The verifier refuses to guess " + "which one is the manifest." + ) + return tables[0] + + +def _doc_row(path: Path, columns: dict[str, int], values: list[str]) -> DocManifestRow: + def cell(name: str) -> str | None: + if name not in columns: + return None + text = values[columns[name]] + return text or None + + canonical = cell("canonical_sha256") or "" + if not _HEX64.match(canonical): + raise ValueError( + f"{path.name}: canonical_sha256 {canonical!r} is not 64 lowercase hex." + ) + file_hash = cell("file_sha256") + if file_hash is not None and not _HEX64.match(file_hash): + raise ValueError( + f"{path.name}: file_sha256 {file_hash!r} is not 64 lowercase hex." + ) + factor_id = cell("factor_id") + if not factor_id: + raise ValueError(f"{path.name}: manifest table row has an empty factor_id.") + + def as_int(name: str) -> int | None: + text = cell(name) + return None if text is None else int(text) + + def as_float(name: str) -> float | None: + text = cell(name) + return None if text is None else float(text) + + return DocManifestRow( + factor_id=factor_id, + canonical_sha256=canonical, + kind=cell("kind"), + rows=as_int("rows"), + n_symbols=as_int("n_symbols"), + n_nan=as_int("n_nan"), + mean=as_float("mean"), + std=as_float("std"), + file_sha256=file_hash, + ) + + +def parse_doc_producing_sha(path: Path | str) -> str | None: + """The producing SHA the document pins, if it states one. + + Read from the document rather than transcribed into this module: the SHA + would otherwise exist in two places, and the copy in code is the one nobody + updates. + """ + for line in Path(path).read_text(encoding="utf-8").splitlines(): + lowered = line.lower() + if "producing" in lowered and ("sha" in lowered or "git_sha" in lowered): + found = _SHA40.search(line) + if found: + return found.group(0) + return None + + diff --git a/qt/hand_anchor_rows.py b/qt/hand_anchor_rows.py index 90cac08..725cb04 100644 --- a/qt/hand_anchor_rows.py +++ b/qt/hand_anchor_rows.py @@ -4,9 +4,11 @@ of :mod:`qt.hand_anchors_d2`, compares each against the NEW-ENGINE value read from the ``panels_d2`` parquet files (file reads — never engine imports), and writes the selection + results JSON. The four ops-rewritten daily factors that -have no frozen panel are SELECTED and hand-computed here too; their engine-side -values are filled in by ``qt.hand_anchors_engine_values`` (the only module of -the trio allowed to import the engine). +have no frozen panel are SELECTED and hand-computed here too, and left in +``daily_pending_engine``. The companion that USED to fill in their engine-side +values was retired in D5 C6, so those rows now stay pending: see +``qt.hand_anchors_d2.ENGINE_COMPARISON_POINTER``, which this module prints +rather than restating. Selection sources (all engine-free): the ``panels_d2`` files give the finite / NaN pattern (warm-up ends, interior rows); the ``adj_factor`` cache gives the @@ -455,8 +457,7 @@ def compare(factor_id, cls, key, hand, note=""): out_path.write_text(json.dumps(payload, indent=2), encoding="utf-8") n_bad = sum(1 for r in results if not r["ok"]) print(f"hand anchors (frozen 14): {len(results)} rows, {n_bad} mismatches -> {out_path}") - print(f"daily rows pending engine comparison: {len(daily_rows)} " - f"(run python -m qt.hand_anchors_engine_values)") + print(H.pending_engine_line(len(daily_rows))) return 0 if n_bad == 0 else 1 diff --git a/qt/hand_anchors_d2.py b/qt/hand_anchors_d2.py index 8756933..8e5732f 100644 --- a/qt/hand_anchors_d2.py +++ b/qt/hand_anchors_d2.py @@ -32,10 +32,14 @@ class — (a) the warm-up END (the symbol's first finite value; verified by hand * qfq for the daily factors: raw close x af / af(symbol's last panel day) — the ``front_adjust`` anchor convention, re-derived here from its definition. -Run AFTER ``python -m qt.panel_reconcile`` produced ``panels_d2``: +``panels_d2`` is FROZEN (D5 C6): ``qt.panel_reconcile`` no longer rebuilds it, +so the files this module reads are simply there. Run: python -m qt.hand_anchors_d2 # select + hand-compute + compare (14) - python -m qt.hand_anchors_engine_values # compare the 4 daily factors + +The companion that filled in the four daily factors' engine values is RETIRED +(D5 C6) -- see ``ENGINE_COMPARISON_POINTER`` below for what to run instead and +why those rows can no longer be completed. """ from __future__ import annotations @@ -54,6 +58,27 @@ class — (a) the warm-up END (the symbol's first finite value; verified by hand WINDOW_LO = pd.Timestamp("2021-07-01") WINDOW_HI = pd.Timestamp("2026-06-30") TOL = 1e-12 + +#: The ONE authored sentence about what an operator can still do with the daily +#: engine comparison, and what they cannot. ``qt.hand_anchor_rows`` composes it +#: rather than restating it: a regex cannot assert that no other sentence points +#: at the retired command, but "there is no other sentence" can (methodology #2). +#: +#: It lives HERE, not in ``qt.hand_anchors_engine_values``, because that module +#: imports ``qt.panel_freeze`` -> ``data.clean.schema``, and the hand side must +#: never load the engine (``assert_no_engine_imports`` below). An author-once +#: home has to be reachable from the pure side. +ENGINE_COMPARISON_POINTER = ( + "the engine-side comparison is RETIRED (D5 C6) and these rows can no longer " + "be completed from this tree; run `python -m qt.hand_anchors_engine_values " + "--verify` to re-check whatever comparison is already recorded" +) + + +def pending_engine_line(n_pending: int) -> str: + """The run-summary line about daily rows awaiting an engine comparison.""" + return f"daily rows pending engine comparison: {n_pending} — {ENGINE_COMPARISON_POINTER}" + SEED = 20260724 # ---- pinned definition constants, restated as PLAIN NUMBERS (the point of a diff --git a/qt/hand_anchors_engine_values.py b/qt/hand_anchors_engine_values.py index 8f27af5..2e63ea3 100644 --- a/qt/hand_anchors_engine_values.py +++ b/qt/hand_anchors_engine_values.py @@ -1,98 +1,224 @@ -"""Engine-side comparison for the four daily hand anchors (D2, R12 companion). +"""Daily hand-anchor engine comparison — VERIFY-ONLY since D5 C6. ``qt.hand_anchors_d2`` / ``qt.hand_anchor_rows`` hand-compute momentum_20 / reversal_20 / liquidity_20 / overnight_mom_20 anchor rows WITHOUT importing the engine and leave them in ``hand_anchors_d2.json`` under -``daily_pending_engine``. This companion is the one place ALLOWED to import -the engine: it rebuilds the freeze data plane (same config, cache-only), runs -the ops-rewritten factor classes, and compares each pending row at the same -<= 1e-12 relative tolerance. The independence requirement binds the HAND side, -not the comparer (module docstring of hand_anchors_d2). - -Run AFTER ``python -m qt.hand_anchor_rows``: - - python -m qt.hand_anchors_engine_values +``daily_pending_engine``. This companion used to be the one place ALLOWED to +import the engine: it rebuilt the freeze data plane (same config, cache-only), +ran the ops-rewritten factor classes, and wrote each comparison back into the +JSON under ``daily_engine_compared`` / ``all_ok_daily``. + +REGENERATION IS RETIRED (owner ruling 2026-07-28, D5 C6) +--------------------------------------------------------- +The rebuild imported a legacy runner's precondition check and re-derived the +whole data plane behind it; C6 deletes that runner. Two honest notes about the +scope of this particular retirement, because overstating it would be the very +defect this repository keeps catching: + +* the dependency on the legacy runners was THIN here — one precondition helper — + not the eleven private loaders that ``qt.panel_freeze`` called. Retiring this + tool therefore gives up a capability that was not, on its own, doomed; +* the four daily factor classes themselves are untouched and keep their own + tests. What is retired is this hand-vs-engine COMPARISON, not the factors. + +What ``--verify`` does +---------------------- +It re-derives the recorded comparison: for every row of +``daily_engine_compared`` it recomputes the relative difference from the +recorded ``hand`` / ``engine`` values at the same tolerance and checks the +recorded ``rel_diff`` / ``ok`` / ``all_ok_daily`` against what those values +actually imply, and checks that every pending anchor got a comparison. So a +record edited to turn a mismatch into a pass does not survive; a record whose +NUMBERS were never produced cannot be re-created here at all. + +When the record is ABSENT the exit code is non-zero and the wording is "NOT +VERIFIED", not "FAILED" — absence of evidence is not evidence of failure, but a +verification command that reports success without verifying anything is the +empty-reconciliation shape this repository has already committed once. + +Run:: + + python -m qt.hand_anchors_engine_values --verify """ from __future__ import annotations +import argparse import json +import math import sys +from dataclasses import dataclass from pathlib import Path -import numpy as np - from qt.hand_anchors_d2 import OUT_JSON, TOL +from qt.panel_freeze import RegenerationRetiredError, retirement_message + +#: The four ops-rewritten daily factors this comparison covers. +DAILY_FACTORS = ("momentum_20", "reversal_20", "liquidity_20", "overnight_mom_20") +#: Fields every recorded comparison row must carry. +COMPARED_ROW_FIELDS = ("factor_id", "class", "date", "symbol", "hand", "engine", + "rel_diff", "ok") + + +def relative_difference(hand: float, engine: float) -> float: + """The recorded comparison's own rule, restated once and used by the checker.""" + if math.isfinite(hand) and math.isfinite(engine): + denominator = max(abs(hand), abs(engine)) + return abs(hand - engine) / denominator if denominator > 0 else 0.0 + if math.isnan(hand) and math.isnan(engine): + return 0.0 + return float("inf") + + +@dataclass(frozen=True) +class AnchorRecordVerification: + """Outcome of checking the recorded daily hand-anchor comparison.""" + + path: Path + n_compared: int + n_pending: int + problems: tuple[str, ...] + recorded: bool + + @property + def ok(self) -> bool: + return self.recorded and not self.problems + + +def verify_recorded_comparison(path: Path | str = OUT_JSON) -> AnchorRecordVerification: + """Re-derive the recorded daily hand-anchor comparison from its own numbers.""" + path = Path(path) + problems: list[str] = [] + if not path.exists(): + return AnchorRecordVerification( + path=path, + n_compared=0, + n_pending=0, + problems=(f"anchor record not found: {path}",), + recorded=False, + ) + payload = json.loads(path.read_text(encoding="utf-8")) + pending = payload.get("daily_pending_engine", []) or [] + compared = payload.get("daily_engine_compared", []) or [] + if not compared: + return AnchorRecordVerification( + path=path, + n_compared=0, + n_pending=len(pending), + problems=( + f"{path.name} carries no daily_engine_compared rows " + f"({len(pending)} anchor(s) still pending). The comparison that " + "would fill them in is retired, so this record cannot be " + "completed from this tree.", + ), + recorded=False, + ) + n_bad = 0 + for position, row in enumerate(compared): + missing = [field for field in COMPARED_ROW_FIELDS if field not in row] + if missing: + problems.append(f"row {position}: missing field(s) {missing}") + continue + label = f"{row['factor_id']} {row['date']} {row['symbol']}" + if row["factor_id"] not in DAILY_FACTORS: + problems.append(f"row {position}: unexpected factor id {row['factor_id']!r}") + expected_rel = relative_difference(float(row["hand"]), float(row["engine"])) + recorded_rel = float(row["rel_diff"]) + if not _same_number(expected_rel, recorded_rel): + problems.append( + f"{label}: recorded rel_diff {recorded_rel!r} but hand/engine imply " + f"{expected_rel!r}" + ) + expected_ok = expected_rel <= TOL + if bool(row["ok"]) != expected_ok: + problems.append( + f"{label}: recorded ok={row['ok']!r} but rel {expected_rel!r} vs " + f"tolerance {TOL!r} implies {expected_ok}" + ) + n_bad += 0 if expected_ok else 1 + + recorded_all_ok = payload.get("all_ok_daily") + if recorded_all_ok is None: + problems.append("all_ok_daily is absent from the record") + elif bool(recorded_all_ok) != (n_bad == 0): + problems.append( + f"all_ok_daily is recorded as {recorded_all_ok!r} but the rows imply " + f"{n_bad == 0} ({n_bad} mismatching row(s))" + ) + elif n_bad: + problems.append( + f"{n_bad} recorded row(s) exceed the tolerance — the daily anchors did " + "not agree with the engine." + ) -def main(argv: list[str] | None = None) -> int: - import pandas as pd - - from factors.compute.candidates import ( - LiquidityFactor, - OvernightMomentumFactor, - ReversalFactor, + compared_keys = {(r.get("factor_id"), r.get("date"), r.get("symbol")) for r in compared} + for row in pending: + key = (row.get("factor_id"), row.get("date"), row.get("symbol")) + if key not in compared_keys: + problems.append(f"pending anchor never compared: {key}") + + return AnchorRecordVerification( + path=path, + n_compared=len(compared), + n_pending=len(pending), + problems=tuple(problems), + recorded=True, ) - from factors.compute.momentum import MomentumFactor - from qt.config import load_config - from qt.eval_jump_amount_corr import _check_preconditions - from qt.panel_freeze import DEFAULT_CONFIG, PANEL_STORE_NAME - from qt.pipeline import _build_cache, _build_universe, _load_panel, _make_logger - - payload = json.loads(Path(OUT_JSON).read_text(encoding="utf-8")) - pending = payload.get("daily_pending_engine", []) - if not pending: - print("no pending daily rows; run qt.hand_anchor_rows first") - return 1 - cfg = load_config(DEFAULT_CONFIG) - _check_preconditions(cfg) - cfg = cfg.model_copy( - update={"data": cfg.data.model_copy(update={"output_name": PANEL_STORE_NAME})} + +def _same_number(left: float, right: float) -> bool: + if math.isnan(left) and math.isnan(right): + return True + return left == right + + +def run_engine_comparison(*args, **kwargs): + """RETIRED (D5 C6). Always raises :class:`RegenerationRetiredError`.""" + raise RegenerationRetiredError( + retirement_message( + "python -m qt.hand_anchors_engine_values", + "the daily hand-anchor engine comparison", + "The rebuild imported a legacy eval runner's precondition check and " + "re-derived the whole data plane behind it; C6 deletes that runner. " + "The four daily factor classes are untouched — what is retired is " + "this comparison, not the factors.", + "it re-derives the RECORDED comparison in hand_anchors_d2.json from " + "its own hand/engine numbers. It cannot produce numbers that were " + "never recorded, and says so rather than reporting success.", + ) ) - logger = _make_logger( - Path(cfg.output.log_dir) / "hand_anchors_engine.log", - name="qt.hand_anchors_engine", + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--record", default=str(OUT_JSON)) + parser.add_argument( + "--verify", + action="store_true", + help="re-derive the recorded daily hand-anchor comparison from its own numbers", ) - cache = _build_cache(cfg) - _, symbols = _build_universe(cfg, logger, cache) - panel = _load_panel(cfg, symbols, logger, cache) - - factories = { - "momentum_20": MomentumFactor(window=20), - "reversal_20": ReversalFactor(window=20), - "liquidity_20": LiquidityFactor(window=20), - "overnight_mom_20": OvernightMomentumFactor(window=20), - } - values = {name: f.compute(panel) for name, f in factories.items()} + args = parser.parse_args(argv) - n_bad = 0 - out_rows = [] - for row in pending: - series = values[row["factor_id"]] - key = (pd.Timestamp(row["date"]), row["symbol"]) - engine = float(series.loc[key]) if key in series.index else float("nan") - hand = row["hand"] - if np.isfinite(hand) and np.isfinite(engine): - denom = max(abs(hand), abs(engine)) - rel = abs(hand - engine) / denom if denom > 0 else 0.0 - elif np.isnan(hand) and np.isnan(engine): - rel = 0.0 - else: - rel = float("inf") - ok = rel <= TOL - n_bad += 0 if ok else 1 - out_rows.append({**row, "engine": engine, "rel_diff": rel, "ok": bool(ok)}) - print( - f"{'OK ' if ok else 'FAIL'} {row['factor_id']:18s} {row['class']:12s} " - f"{row['date']} {row['symbol']} hand={hand!r} engine={engine!r} rel={rel:.2e}" - ) + if not args.verify: + try: + run_engine_comparison() + except RegenerationRetiredError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 - payload["daily_engine_compared"] = out_rows - payload["all_ok_daily"] = n_bad == 0 - Path(OUT_JSON).write_text(json.dumps(payload, indent=2), encoding="utf-8") - print(f"daily anchors: {len(out_rows)} rows, {n_bad} mismatches -> {OUT_JSON}") - return 0 if n_bad == 0 else 1 + result = verify_recorded_comparison(args.record) + print( + f"{result.path}: {result.n_compared} recorded comparison(s), " + f"{result.n_pending} pending anchor(s)" + ) + for problem in result.problems: + print(f" PROBLEM {problem}") + if not result.recorded: + print("daily hand-anchor verification: NOT VERIFIED (nothing recorded)") + return 1 + print("daily hand-anchor verification: " + ("OK" if result.ok else "FAILED")) + return 0 if result.ok else 1 if __name__ == "__main__": # pragma: no cover - thin CLI shim diff --git a/qt/panel_freeze.py b/qt/panel_freeze.py index bb32ef0..88ed60f 100644 --- a/qt/panel_freeze.py +++ b/qt/panel_freeze.py @@ -1,31 +1,36 @@ -"""D1 end-of-stage RAW factor-value panel freeze (the pre-D2 baseline for D5). - -Freezes the closing 14 factors' RAW daily factor-value panels (11 minute-derived -factors + the 3 confirmed book factors) on the eleven-factor evaluation data -plane (CSI500 ``000905.SH``, 2021-07-01 .. 2026-06-30, CACHE-ONLY). ``main`` at -the producing SHA still carries the pre-refactor factor math bit-for-bit (the D1 -registry PR was dispatch-only), so these panels are the ONLY baseline the D5 -cell-by-cell reconciliation can compare against once D2 rewrites the math. - -NO FACTOR MATH IS REIMPLEMENTED HERE. Every panel is produced by calling the -SAME loader chain the ``qt/eval_*.py`` runners call, with the SAME constants, -imported FROM the runner modules themselves (identity with the runner call -sites, not a transcription of them): - -* minute factors — each runner's private ``_load_*_panel`` (per-symbol - cache-only 1min read -> ``data.clean`` ``compute_*`` -> daily aggregation), - then the runner's own restriction to the daily panel dates - (``load.factor[dates.isin(panel_dates)]``). The frozen value is the RAW - restricted series, BEFORE ``_process_factors`` (z-score / neutralization are - shared machinery, not factor math). -* book factors — the runners' ``_build_book_factors()`` + - ``factor.compute(panel)`` on the enriched daily panel, frozen RAW. - -PROVENANCE RULE (design v3.2 §5 leg 4): regenerating this baseline is only -legitimate from a checkout of the pinned pre-D2 producing SHA recorded in the -manifest — NEVER from current code. Rebuilding the baseline from the same tree -that is being validated would make the D5 reconciliation structurally unable to -fail (the ``compare_postmerge.py`` empty-reconciliation failure mode). +"""D1 RAW factor-value panel baseline — VERIFY-ONLY since D5 C6. + +The 14 closing factors' RAW daily factor-value panels (11 minute-derived + 3 +confirmed book factors) were frozen once, on the eleven-factor evaluation data +plane (CSI500 ``000905.SH``, 2021-07-01 .. 2026-06-30, CACHE-ONLY), from ``main`` +at the producing SHA recorded in ``docs/factors/d1_panel_freeze_manifest.md``. +They are the ONLY baseline the D5 cell-by-cell reconciliation compares against. + +REGENERATION IS RETIRED (owner ruling 2026-07-28, D5 C6) +--------------------------------------------------------- +This module used to rebuild those panels by calling the eleven legacy +``qt/eval_*.py`` runners' private ``_load_*_panel`` loaders. C6 deletes those +runners, so the rebuild could not work again; rather than leave a code path that +breaks the moment it is used, the capability is retired outright and the +baseline is **frozen-forever**. What remains is verification: read the frozen +bytes and check them against the manifest. + +That direction is the safe one. The provenance rule (design v3.2 §5 leg 4) was +always that regenerating this baseline is legitimate ONLY from a checkout of the +pinned pre-D2 SHA — never from current code, because rebuilding the baseline +from the tree being validated makes the D5 reconciliation structurally unable to +fail (the ``compare_postmerge.py`` empty-reconciliation failure mode this repo +committed once). Retiring the rebuild removes the only way to violate that rule +by accident. + +Where the authority lives +------------------------- +The bulk panels are gitignored; **the hashes are in git**, in the manifest +document's table. That split is what gives verification teeth: someone who +regenerates the panels and their machine manifest cannot also silently move the +expected hashes, because those show up in ``git diff``. The gitignored +``manifest.json`` is checked too — as a second, richer witness (row counts, +NaN counts, mean/std, file sha256) that must AGREE with the git-tracked table. Canonical content hash ---------------------- @@ -46,7 +51,14 @@ hashed (content equality, not labelling). NaN payload bits are collapsed on purpose; +0.0 and -0.0 stay distinct (a real IEEE value difference). -Run: ``python -m qt.panel_freeze`` (deliberately NOT registered in qt/cli). +Run (deliberately NOT registered in qt/cli, as before):: + + python -m qt.panel_freeze --verify + +There is no partial verification. A frozen-forever baseline verified in part is +a baseline you believe more than you checked, so ``--verify`` always covers +every panel in both frozen trees (the D1 baseline and the PR-C corrected +``jump_amount_corr_20`` reference panel). """ from __future__ import annotations @@ -54,36 +66,37 @@ import argparse import hashlib import json -import logging import os -import subprocess -import time -from collections.abc import Sequence +import sys from dataclasses import dataclass from pathlib import Path -from typing import Callable import numpy as np import pandas as pd from data.clean.schema import DATE_LEVEL, SYMBOL_LEVEL +from qt.frozen_manifest_doc import ( + DocManifestRow, + parse_doc_manifest, + parse_doc_producing_sha, +) CANONICAL_HASH_VERSION = "qt.panel_freeze canonical v1" DEFAULT_CONFIG = "config/phase_c_jump_amount_corr.yaml" DEFAULT_OUTPUT_ROOT = "artifacts/refactor_baseline" -#: PanelStore artifact name for THIS run — content-identical to the runners' -#: per-run panel (same cache, same window), under a freeze-specific name so the -#: freeze never clobbers an accepted eval run's ``artifacts/data`` artifact. +#: PanelStore artifact name the freeze used — kept because the frozen +#: ``manifest.json`` header records it and the verifier reads that header. PANEL_STORE_NAME = "d1_panel_freeze_daily" -#: Determinism double-run subjects (task §2): two minute factors — including -#: valley_price_quantile, the one loader that also consumes the daily panel — -#: plus one book factor. -DETERMINISM_FACTORS = ("jump_amount_corr_20", "valley_price_quantile_20", "value_ep") -#: data_coverage payload fields reconciled against the shipped eval JSONs. -#: They describe the PROCESSED series at the evaluator boundary (see -#: ``analytics/eval/standard.py::data_coverage``), so the reconciliation -#: re-derives that boundary from the frozen RAW panel via the runners' own -#: ``_process_factors`` — the frozen artifact itself stays raw. +#: The git-tracked authority for the D1 baseline (14 panels). +D1_MANIFEST_DOC = "docs/factors/d1_panel_freeze_manifest.md" +#: The git-tracked authority for the PR-C corrected jump reference panel (1). +PR_C_MANIFEST_DOC = "docs/factors/pr_c_cutoff_fix_reference_panel.md" +#: Sub-root of the PR-C reference panel, relative to the baseline root. +PR_C_SUBDIR = "pr_c_cutoff_fix" +#: data_coverage payload fields the freeze reconciled against the shipped eval +#: JSONs BEFORE accepting each panel. The reconciliation itself is history; the +#: field list is not, because ``manifest.json`` records the per-field outcome +#: and verification re-reads that record. RECONCILE_INT_FIELDS = ( "panel_rows", "evaluation_periods", @@ -93,6 +106,42 @@ ) RECONCILE_FLOAT_FIELDS = ("factor_nan_rate",) +#: The single authored statement of what C6 retired. Every tool that retired a +#: regeneration path COMPOSES this rather than restating it: a regex cannot +#: assert that no other sentence says the same thing, but "there is no other +#: sentence" can (CLAUDE.md methodology #2). +RETIREMENT_RULING = "owner ruling 2026-07-28, D5 C6" + + +class RegenerationRetiredError(RuntimeError): + """Raised when a retired regeneration entry point is invoked. + + A retired capability must fail LOUDLY and READABLY (redline #9). Letting the + call succeed as a no-op, or die on a bare ``ImportError`` from a deleted + runner, both leave the caller guessing whether anything happened. + """ + + +def retirement_message(tool: str, product: str, why: str, verifies: str) -> str: + """The retirement error text, composed from one authored core plus two + per-tool clauses. + + Only what is true of ALL THREE retired tools is shared: the ruling, the fact + that this repository no longer rebuilds the product, and the pointer to + ``--verify``. Both ``why`` (what the rebuild depended on) and ``verifies`` + (what verification actually checks) are supplied by the caller, because the + three tools differ on both counts — they did not lean on the legacy runners + to the same degree, and they do not verify the same kind of thing. A single + shared sentence covering those would have to overstate one of them, which is + this repository's recurring defect: text that describes a check other than + the one performed. + """ + return ( + f"{tool}: regeneration is RETIRED ({RETIREMENT_RULING}). This repository " + f"no longer rebuilds {product}. {why} " + f"Run `{tool} --verify` instead: {verifies}" + ) + # --------------------------------------------------------------------------- # # Canonical content hash + atomic write (pure, network-free, unit-tested) @@ -169,6 +218,12 @@ def atomic_write_parquet(panel: pd.Series | pd.DataFrame, path: Path | str) -> s tmp-file + ``os.replace`` in the target directory: readers never observe a partial file, and a failed write leaves no tmp residue (and never touches an existing target). + + This is the byte-layout contract the frozen panels were laid down with. No + production caller remains after C6 (the freeze that used it is retired); it + is kept because it DEFINES the on-disk form the verifier reads, and because + a verifier that cannot construct a valid frozen tree cannot be given a + negative test. """ series = _as_single_series(panel).sort_index(kind="mergesort") target = Path(path) @@ -231,7 +286,12 @@ def manifest_row( file_name: str, ) -> dict: """One manifest record. mean/std are float64 ``Series.mean()``/``std(ddof=1)`` - with NaN skipped (pandas defaults), reported at full precision.""" + with NaN skipped (pandas defaults), reported at full precision. + + Still live after C6: verification RE-DERIVES each row from the frozen panel + and compares it field by field with the recorded one, so a manifest whose + hash was patched but whose statistics were not is caught as well. + """ series = _as_single_series(panel) dates = series.index.get_level_values(DATE_LEVEL) return { @@ -251,7 +311,12 @@ def manifest_row( def render_manifest_markdown(header: dict, rows: list[dict]) -> str: - """Deterministic Markdown manifest: a header block + one row per factor.""" + """Deterministic Markdown manifest: a header block + one row per factor. + + This is the format definition of the frozen ``manifest.md``; ``--verify + --show-manifest`` re-renders the table from the panels actually on disk so + it can be eyeballed (or diffed) against the frozen one. + """ lines = ["# D1 panel freeze manifest", ""] for key in sorted(header): lines.append(f"- **{key}**: {header[key]}") @@ -269,577 +334,422 @@ def render_manifest_markdown(header: dict, rows: list[dict]) -> str: return "\n".join(lines) -def atomic_write_text(text: str, path: Path | str) -> None: - """tmp + ``os.replace`` text write (same atomicity contract as the parquet).""" - target = Path(path) - target.parent.mkdir(parents=True, exist_ok=True) - tmp = target.with_name(target.name + ".tmp") - try: - tmp.write_text(text, encoding="utf-8") - os.replace(tmp, target) - finally: - tmp.unlink(missing_ok=True) - - # --------------------------------------------------------------------------- # -# Recipes: the runner call sites, referenced (not transcribed) +# Verification # --------------------------------------------------------------------------- # @dataclass(frozen=True) -class MinuteRecipe: - """One minute factor's runner-owned build chain. - - ``build`` closes over the runner module's loader AND its constants, so the - call is identical to the runner's own call site by construction. - """ +class PanelVerification: + """One frozen panel's verification outcome (no secrets).""" factor_id: str - stem: str # the runner's _REPORT_STEM (names the eval JSON artifacts) - spec: object - build: Callable[[object, list[str], pd.DataFrame, logging.Logger], object] + file: str + rows: int + canonical_sha256: str + problems: tuple[str, ...] + @property + def ok(self) -> bool: + return not self.problems -def minute_recipes() -> list[MinuteRecipe]: - """The 11 minute factors, each bound to its runner's loader + constants. - Imports live inside the function so importing ``qt.panel_freeze`` for the - network-free unit tests stays light. +@dataclass(frozen=True) +class VerifyResult: + """A whole frozen tree's verification outcome.""" + + root: Path + doc: Path + panels: tuple[PanelVerification, ...] + problems: tuple[str, ...] + producing_git_sha: str | None + rendered_manifest: str | None = None + + @property + def ok(self) -> bool: + return not self.problems and all(panel.ok for panel in self.panels) + + @property + def n_ok(self) -> int: + return sum(1 for panel in self.panels if panel.ok) + + +def _load_machine_manifest(root: Path) -> dict: + """The gitignored ``manifest.json`` document, or ``{}`` when absent.""" + path = root / "manifest.json" + if not path.exists(): + return {} + return json.loads(path.read_text(encoding="utf-8")) + + +def _check_recorded_reconciliation(document: dict, problems: list[str]) -> None: + """The freeze reconciled each minute panel against its shipped eval JSON and + recorded the per-field outcome. Verification re-reads that record. + + This is deliberately a check of the RECORD, not a re-run: re-running it + needs the processed series, which needs the data plane, which needs the + loaders C6 deletes. What can still be asserted is that the record says what + a passing reconciliation says — every declared field present, every check + ``ok`` — so a manifest edited to paper over a failure is caught. """ - import qt.eval_amp_marginal_anomaly_vol as e - import qt.eval_intraday_amp_cut as g - import qt.eval_jump_amount_corr as c - import qt.eval_minute_ideal_amplitude as d - import qt.eval_peak_interval_kurtosis as h - import qt.eval_peak_ridge_amount_ratio as m - import qt.eval_ridge_minute_return as k - import qt.eval_valley_price_quantile as ell - import qt.eval_valley_relative_vwap as i - import qt.eval_valley_ridge_vwap_ratio as j - import qt.eval_volume_peak_count as f - - recipes: list[MinuteRecipe] = [] - - def add(module, factor, build) -> None: - spec = factor.spec - recipes.append( - MinuteRecipe( - factor_id=spec.factor_id, - stem=module._REPORT_STEM, - spec=spec, - build=build, - ) + recorded = document.get("reconciliation", {}) + if not recorded: + problems.append( + "manifest.json carries no reconciliation block; the frozen panels' " + "agreement with the shipped eval artifacts is unwitnessed." ) - - jac = c.JumpAmountCorrFactor(lookback_days=c.JUMP_LOOKBACK_DAYS) - add(c, jac, lambda cfg, symbols, panel, logger, _s=jac.spec: c._load_jump_factor_panel( - cfg, symbols, _s, logger, - lookback_days=c.JUMP_LOOKBACK_DAYS, min_pairs=c.JUMP_MIN_PAIRS, - )) - - mia = d.MinuteIdealAmplitudeFactor(lookback_days=d.IDEAL_AMP_LOOKBACK_DAYS) - add(d, mia, lambda cfg, symbols, panel, logger, _s=mia.spec: d._load_minute_ideal_amp_panel( - cfg, symbols, _s, logger, - lookback_days=d.IDEAL_AMP_LOOKBACK_DAYS, lam=d.IDEAL_AMP_LAMBDA, - min_minutes=d.IDEAL_AMP_MIN_MINUTES, - )) - - amav = e.AmpMarginalAnomalyVolFactor(lookback_days=e.AMP_ANOMALY_LOOKBACK_DAYS) - add(e, amav, lambda cfg, symbols, panel, logger, _s=amav.spec: e._load_amp_anomaly_vol_panel( - cfg, symbols, _s, logger, - lookback_days=e.AMP_ANOMALY_LOOKBACK_DAYS, min_pool=e.AMP_ANOMALY_MIN_POOL, - min_selected=e.AMP_ANOMALY_MIN_SELECTED, sigma_k=e.AMP_ANOMALY_SIGMA_K, - )) - - vpc = f.VolumePeakCountFactor(lookback_days=f.VOLUME_PRV_LOOKBACK_DAYS) - add(f, vpc, lambda cfg, symbols, panel, logger, _s=vpc.spec: f._load_volume_peak_count_panel( - cfg, symbols, _s, logger, - lookback_days=f.VOLUME_PRV_LOOKBACK_DAYS, baseline_days=f.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=f.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=f.VOLUME_PRV_SIGMA_K, - min_valid_days=f.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=f.VOLUME_PRV_MIN_CLASSIFIABLE, - )) - - iac = g.IntradayAmpCutFactor(lookback_days=g.AMP_CUT_LOOKBACK_DAYS) - add(g, iac, lambda cfg, symbols, panel, logger, _s=iac.spec: g._load_amp_cut_panel( - cfg, symbols, _s, logger, - lookback_days=g.AMP_CUT_LOOKBACK_DAYS, lam=g.AMP_CUT_LAMBDA, - min_day_minutes=g.AMP_CUT_MIN_DAY_MINUTES, min_valid_days=g.AMP_CUT_MIN_VALID_DAYS, - min_cross_section=g.AMP_CUT_MIN_CROSS_SECTION, - )) - - pik = h.PeakIntervalKurtosisFactor(lookback_days=h.PEAK_INTERVAL_LOOKBACK_DAYS) - add(h, pik, lambda cfg, symbols, panel, logger, _s=pik.spec: h._load_peak_interval_kurtosis_panel( - cfg, symbols, _s, logger, - lookback_days=h.PEAK_INTERVAL_LOOKBACK_DAYS, baseline_days=h.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=h.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=h.VOLUME_PRV_SIGMA_K, - min_valid_days=h.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=h.VOLUME_PRV_MIN_CLASSIFIABLE, - min_intervals=h.PEAK_INTERVAL_MIN_INTERVALS, - )) - - vrv = i.ValleyRelativeVwapFactor(lookback_days=i.VALLEY_VWAP_LOOKBACK_DAYS) - add(i, vrv, lambda cfg, symbols, panel, logger, _s=vrv.spec: i._load_valley_relative_vwap_panel( - cfg, symbols, _s, logger, - lookback_days=i.VALLEY_VWAP_LOOKBACK_DAYS, baseline_days=i.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=i.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=i.VOLUME_PRV_SIGMA_K, - min_valid_days=i.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=i.VOLUME_PRV_MIN_CLASSIFIABLE, - min_valley_bars=i.VALLEY_VWAP_MIN_VALLEY_BARS, - )) - - vrr = j.ValleyRidgeVwapRatioFactor(lookback_days=j.VALLEY_RIDGE_LOOKBACK_DAYS) - add(j, vrr, lambda cfg, symbols, panel, logger, _s=vrr.spec: j._load_valley_ridge_vwap_ratio_panel( - cfg, symbols, _s, logger, - lookback_days=j.VALLEY_RIDGE_LOOKBACK_DAYS, baseline_days=j.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=j.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=j.VOLUME_PRV_SIGMA_K, - min_valid_days=j.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=j.VOLUME_PRV_MIN_CLASSIFIABLE, - min_valley_bars=j.VALLEY_RIDGE_MIN_VALLEY_BARS, - min_ridge_bars=j.VALLEY_RIDGE_MIN_RIDGE_BARS, - )) - - rmr = k.RidgeMinuteReturnFactor(lookback_days=k.RIDGE_RETURN_LOOKBACK_DAYS) - add(k, rmr, lambda cfg, symbols, panel, logger, _s=rmr.spec: k._load_ridge_minute_return_panel( - cfg, symbols, _s, logger, - lookback_days=k.RIDGE_RETURN_LOOKBACK_DAYS, baseline_days=k.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=k.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=k.VOLUME_PRV_SIGMA_K, - min_valid_days=k.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=k.VOLUME_PRV_MIN_CLASSIFIABLE, - min_ridge_bars=k.RIDGE_RETURN_MIN_RIDGE_BARS, - )) - - vpq = ell.ValleyPriceQuantileFactor(lookback_days=ell.VALLEY_QUANTILE_LOOKBACK_DAYS) - # The ONE loader that also consumes the daily panel (its reversal - # neutralization needs daily closes) — the runner passes it positionally. - add(ell, vpq, lambda cfg, symbols, panel, logger, _s=vpq.spec: ell._load_valley_price_quantile_panel( - cfg, symbols, _s, panel, logger, - lookback_days=ell.VALLEY_QUANTILE_LOOKBACK_DAYS, - baseline_days=ell.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=ell.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=ell.VOLUME_PRV_SIGMA_K, - min_valid_days=ell.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=ell.VOLUME_PRV_MIN_CLASSIFIABLE, - min_valley_bars=ell.VALLEY_QUANTILE_MIN_VALLEY_BARS, - min_cross_section=ell.VALLEY_QUANTILE_MIN_CROSS_SECTION, - reversal_days=ell.VALLEY_QUANTILE_REVERSAL_DAYS, - )) - - pra = m.PeakRidgeAmountRatioFactor(lookback_days=m.PEAK_RIDGE_LOOKBACK_DAYS) - add(m, pra, lambda cfg, symbols, panel, logger, _s=pra.spec: m._load_peak_ridge_amount_ratio_panel( - cfg, symbols, _s, logger, - lookback_days=m.PEAK_RIDGE_LOOKBACK_DAYS, baseline_days=m.VOLUME_PRV_BASELINE_DAYS, - baseline_min_obs=m.VOLUME_PRV_BASELINE_MIN_OBS, sigma_k=m.VOLUME_PRV_SIGMA_K, - min_valid_days=m.VOLUME_PRV_MIN_VALID_DAYS, - min_classifiable=m.VOLUME_PRV_MIN_CLASSIFIABLE, - min_peak_bars=m.PEAK_RIDGE_MIN_PEAK_BARS, - min_ridge_bars=m.PEAK_RIDGE_MIN_RIDGE_BARS, - )) - - return recipes - - -def freezable_factor_ids() -> frozenset[str]: - """Every factor id this freeze can produce (3 book + 11 minute). - - Derived from the recipes and the runners' own book builder, so it cannot - drift from what the freeze actually writes. - """ - from qt.eval_jump_amount_corr import _build_book_factors - - return frozenset( - {f.name for f in _build_book_factors()} - | {r.factor_id for r in minute_recipes()} - ) - - -def resolve_selection(only: Sequence[str] | None) -> tuple[str, ...] | None: - """Validate a ``--only`` selection against the ids the freeze can produce. - - ``None`` (freeze everything) passes straight through — the default path is - untouched. A named id that this freeze does not produce is a readable error - rather than a silently empty output root: "froze 0 panels" reported as - success is how a correctness rerun quietly produces nothing. - """ - if only is None: - return None - selected = tuple(only) - if not selected: - raise ValueError( - "--only was given an empty selection; omit it to freeze all factors." + return + + # CLOSED SET, not "whatever happens to be here". Iterating only the entries + # present cannot see a DELETED one -- and the PR that follows this one is a + # deletion PR, so a verifier blind to removals would be blind exactly when it + # is needed. Every minute panel was reconciled before the freeze accepted it + # (book factors carry no coverage fields in the eval artifacts, so they have + # no entry by design); anything else means the record was edited. + # + # The expected set comes from this same document's rows, which are themselves + # tamperable -- but not usefully: deleting a row too is caught by the panel + # loop, which iterates the GIT-TRACKED factor list and reports a panel whose + # machine-manifest row is absent. The two checks close each other. + expected = { + str(row["factor_id"]) + for row in document.get("rows", []) + if str(row.get("kind")) == "minute" + } + for factor_id in sorted(expected - set(recorded)): + problems.append( + f"reconciliation has no entry for {factor_id}, which manifest.json " + "lists as a minute panel — every minute panel was reconciled before " + "being frozen, so an absent entry means the record lost one." ) - known = freezable_factor_ids() - unknown = [fid for fid in selected if fid not in known] - if unknown: - raise ValueError( - f"--only names factor id(s) this freeze does not produce: {unknown}; " - f"known ids are {sorted(known)}." + for factor_id in sorted(set(recorded) - expected): + problems.append( + f"reconciliation carries an entry for {factor_id}, which is not a " + "minute panel in this manifest." ) - return selected + + for factor_id in sorted(recorded): + entry = recorded[factor_id] or {} + artifact = entry.get("artifact") + if not artifact: + problems.append(f"reconciliation[{factor_id}] names no eval artifact") + checks = entry.get("checks", {}) + for field in RECONCILE_INT_FIELDS + RECONCILE_FLOAT_FIELDS: + check = checks.get(field) + if check is None: + problems.append( + f"reconciliation[{factor_id}] is missing the {field!r} check" + ) + elif not check.get("ok"): + problems.append( + f"reconciliation[{factor_id}].{field} was recorded as NOT ok " + f"({check.get('artifact')!r} vs {check.get('frozen')!r}) — a " + "divergent panel must never have been frozen." + ) -# --------------------------------------------------------------------------- # -# Artifact reconciliation (against the shipped eval JSONs) -# --------------------------------------------------------------------------- # -def reconcile_with_eval_artifact( - stem: str, - processed: pd.Series, - declared_symbols: list[str], - reports_dir: Path, -) -> dict: - """Check the frozen panel's coverage against ``{stem}_no_book.json``. - - The JSON's ``data_coverage`` fields describe the PROCESSED factor series at - the evaluator boundary; ``processed`` here was re-derived from the frozen - RAW panel through the runners' own ``_process_factors``, so equality proves - the frozen panel is the same series the shipped evaluation consumed. Any - mismatch raises — a divergent panel must NOT be recorded (task: stop and - investigate, never freeze a discrepancy). +def verify_frozen_panels( + root: Path | str, + doc_path: Path | str, + *, + show_manifest: bool = False, +) -> VerifyResult: + """Check a frozen panel tree against its git-tracked manifest document. + + Per panel, four independent statements must hold: + + 1. the recomputed canonical content hash equals the hash AUTHORED IN GIT; + 2. it also equals the hash in the gitignored ``manifest.json`` (so a tree + regenerated together with its machine manifest is still caught: the two + witnesses would agree with each other and disagree with git); + 3. the whole manifest row re-derived from the panel (rows / date range / + symbol count / NaN count / mean / std / file sha256) equals the recorded + one — a patched hash with stale statistics does not survive this; + 4. the document's own columns (rows / n_symbols / n_nan / mean / std where + the document carries them) agree as well. + + Plus tree-level statements: the set of ``*.parquet`` files equals the set of + factor ids in the document (an EXTRA panel is a problem, not a bonus), and + the recorded producing SHA equals the one the document pins. """ - json_path = reports_dir / f"{stem}_no_book.json" - if not json_path.exists(): - raise FileNotFoundError( - f"eval artifact {json_path.name} not found under {reports_dir} — the " - "reconciliation target is missing; refusing to freeze unreconciled." + root = Path(root) + doc_path = Path(doc_path) + problems: list[str] = [] + + expected = parse_doc_manifest(doc_path) + document = _load_machine_manifest(root) + if not document: + problems.append( + f"machine manifest missing: {root / 'manifest.json'} — the frozen " + "tree is incomplete (the git-tracked document alone cannot witness " + "row counts, NaN counts, mean/std or the file hashes)." ) - document = json.loads(json_path.read_text(encoding="utf-8")) - coverage = None - for section in document.get("sections", []): - if section.get("name") == "data_coverage": - coverage = section.get("payload", {}) - if not coverage: - raise ValueError(f"{json_path.name} carries no data_coverage payload.") - - values = np.asarray(processed.to_numpy(), dtype=float) - finite = np.isfinite(values) - total = int(len(processed)) - evaluated = pd.unique(processed.index.get_level_values(SYMBOL_LEVEL)) - ours: dict[str, object] = { - "panel_rows": total, - "evaluation_periods": int( - pd.unique(processed.index.get_level_values(DATE_LEVEL)).size - ), - "symbols_evaluated": int(len(evaluated)), - "universe_symbols_declared": int(len(declared_symbols)), - "dropped_symbols_count": len( - set(map(str, declared_symbols)) - set(map(str, evaluated)) - ), - # the eval JSON writer rounds floats to 6 decimals (data.quality clean_value) - "factor_nan_rate": round(1.0 - finite.sum() / total, 6) if total else None, - } - checks: dict[str, dict] = {} - mismatches: list[str] = [] - for field in RECONCILE_INT_FIELDS + RECONCILE_FLOAT_FIELDS: - expected = coverage.get(field) - actual = ours[field] - ok = expected == actual - checks[field] = {"artifact": expected, "frozen": actual, "ok": bool(ok)} - if not ok: - mismatches.append(f"{field}: artifact={expected!r} frozen={actual!r}") - if mismatches: - raise ValueError( - f"frozen panel disagrees with {json_path.name} — NOT freezing: " - + "; ".join(mismatches) + else: + _check_recorded_reconciliation(document, problems) + machine_rows = {str(row["factor_id"]): row for row in document.get("rows", [])} + header = document.get("header", {}) + + doc_sha = parse_doc_producing_sha(doc_path) + recorded_sha = header.get("producing_git_sha") + if doc_sha and recorded_sha and doc_sha != recorded_sha: + problems.append( + f"producing SHA mismatch: {doc_path.name} pins {doc_sha} but " + f"manifest.json records {recorded_sha} — this tree was not produced " + "by the run the document describes." ) - return {"artifact": json_path.name, "checks": checks} - -# --------------------------------------------------------------------------- # -# Freeze orchestration -# --------------------------------------------------------------------------- # -@dataclass(frozen=True) -class FreezeResult: - """Immutable summary of one panel-freeze run (no secrets).""" - - output_root: Path - manifest_json: Path - manifest_md: Path - rows: tuple[dict, ...] - header: dict - elapsed: float - - -def _git_head_sha() -> str: - """Best-effort producing SHA (the manifest doc records the authoritative one).""" - try: - out = subprocess.run( - ["git", "rev-parse", "HEAD"], - capture_output=True, text=True, check=True, timeout=10, + panels_dir = root / "panels" + if not panels_dir.is_dir(): + problems.append(f"frozen panels directory not found: {panels_dir}") + return VerifyResult( + root=root, + doc=doc_path, + panels=(), + problems=tuple(problems), + producing_git_sha=recorded_sha, ) - return out.stdout.strip() - except Exception: # noqa: BLE001 - provenance is best-effort here - return "UNKNOWN" + on_disk = {path.stem for path in panels_dir.glob("*.parquet")} + for extra in sorted(on_disk - set(expected)): + problems.append( + f"unregistered panel on disk: {extra}.parquet is not in " + f"{doc_path.name}; the frozen inventory is defined by the document." + ) -def run_panel_freeze( - config_path: str = DEFAULT_CONFIG, - output_root: str | Path = DEFAULT_OUTPUT_ROOT, - *, - resume: bool = False, - only: Sequence[str] | None = None, -) -> FreezeResult: - """Freeze all 14 RAW factor panels + verify determinism + reconcile artifacts. - - ``only`` restricts the freeze to the named factor ids (default ``None`` = - all 14, byte-identical to before). It exists so a SINGLE factor whose - definition was corrected can be re-frozen into its OWN output root as a - reference panel, without a second freeze implementation and without - re-reading the 279M-row minute cache for the thirteen that did not change. - The shared data plane is built from the FULL book either way, so the daily - panel a filtered run computes on is identical to the full run's. - - ``resume=True`` completes an interrupted freeze: a minute factor whose panel - file already exists is NOT rebuilt from the minute cache — its RAW series is - read back FROM the frozen file and then pushed through the SAME processing + - eval-artifact reconciliation before being accepted into the manifest (a - stale or corrupted file fails loudly; nothing is reused unverified). Its - canonical hash is therefore the hash of the file CONTENT. Book factors are - always recomputed (cheap). The determinism double-run still rebuilds its - subjects end-to-end via the runner loaders, so a resumed subject is compared - fresh-build-vs-frozen-file — a strictly stronger check than two in-process - builds. Resumed factor ids are disclosed in the manifest header. - """ - from qt.config import load_config - from qt.eval_jump_amount_corr import _build_book_factors, _check_preconditions - from qt.pipeline import ( - _build_cache, - _build_universe, - _load_panel, - _log_run_cache_stats, - _make_logger, - _maybe_enrich_covariates, - _maybe_enrich_value, - _process_factors, - ) + verifications: list[PanelVerification] = [] + rendered_rows: list[dict] = [] + for factor_id in sorted(expected): + verification, rendered = _verify_one_panel( + factor_id, expected[factor_id], panels_dir, machine_rows + ) + verifications.append(verification) + if rendered is not None: + rendered_rows.append(rendered) - started = time.monotonic() - # Validate the selection BEFORE the expensive data plane: a typo must cost a - # readable error, not an hour of cache reads followed by an empty freeze. - selected = resolve_selection(only) - cfg = load_config(config_path) - _check_preconditions(cfg) # tushare + cache-only + PIT index + neutralize, as the runners - # Freeze-specific PanelStore name: identical content, never clobbers an - # accepted eval run's per-run panel artifact. - cfg = cfg.model_copy( - update={"data": cfg.data.model_copy(update={"output_name": PANEL_STORE_NAME})} + rendered_manifest = ( + render_manifest_markdown(dict(header), rendered_rows) if show_manifest else None ) - - out_root = Path(output_root) - panels_dir = out_root / "panels" - log_path = Path(cfg.output.log_dir) / "panel_freeze.log" - logger = _make_logger(log_path, name="qt.panel_freeze") - logger.info("panel freeze: config=%s output_root=%s", config_path, out_root) - - # -- shared data plane, the runners' exact preamble --------------------- # - cache = _build_cache(cfg) - universe, symbols = _build_universe(cfg, logger, cache) - panel = _load_panel(cfg, symbols, logger, cache) - book_factors = _build_book_factors() - panel = _maybe_enrich_value(cfg, panel, symbols, book_factors, logger, cache) - panel = _maybe_enrich_covariates(cfg, panel, symbols, logger, cache) - _log_run_cache_stats(cache, logger) - panel_dates = pd.Index( - pd.unique(panel.index.get_level_values(DATE_LEVEL)), name=DATE_LEVEL + return VerifyResult( + root=root, + doc=doc_path, + panels=tuple(verifications), + problems=tuple(problems), + producing_git_sha=recorded_sha, + rendered_manifest=rendered_manifest, ) - rows: list[dict] = [] - reconciliations: dict[str, dict] = {} - canonical_by_id: dict[str, str] = {} - live_calls_total = 0 - def _keep(factor_id: str) -> bool: - return selected is None or factor_id in selected - - # -- book factors (raw, pre-processing) --------------------------------- # - kept_book = [factor for factor in book_factors if _keep(factor.name)] - book_raw = ( - pd.concat( - [factor.compute(panel).rename(factor.name) for factor in kept_book], axis=1 - ) - if kept_book - else pd.DataFrame() - ) - for factor in kept_book: - raw = book_raw[factor.name].rename(factor.name) - canonical = canonical_content_hash(raw) - target = panels_dir / f"{factor.name}.parquet" - sha = atomic_write_parquet(raw, target) - rows.append(manifest_row(factor.name, "book", raw, canonical, sha, target.name)) - canonical_by_id[factor.name] = canonical - logger.info("frozen book %s: rows=%d canonical=%s", factor.name, len(raw), canonical) - - # -- minute factors (raw, the runners' own loader chains) ---------------- # - resumed: list[str] = [] - kept_recipes = [r for r in minute_recipes() if _keep(r.factor_id)] - for recipe in kept_recipes: - target = panels_dir / f"{recipe.factor_id}.parquet" - if resume and target.exists(): - raw = read_frozen_panel(target, recipe.factor_id) - resumed.append(recipe.factor_id) - covered_note = "resumed-from-file" - else: - load = recipe.build(cfg, symbols, panel, logger) - live_calls = int(load.live_calls) - live_calls_total += live_calls - if live_calls != 0: - raise RuntimeError( - f"{recipe.factor_id}: stk_mins_live_calls={live_calls} != 0 — the " - "freeze is cache-only by contract." - ) - raw = load.factor[ - load.factor.index.get_level_values(DATE_LEVEL).isin(panel_dates) - ].rename(recipe.factor_id) - covered_note = f"covered={len(load.covered)}/{load.requested}" - # reconcile BEFORE accepting (a divergent panel is never frozen/kept) - processed = _process_factors( - cfg, raw.to_frame(recipe.factor_id), panel - )[recipe.factor_id] - reconciliations[recipe.factor_id] = reconcile_with_eval_artifact( - recipe.stem, processed, symbols, Path(cfg.output.report_dir) - ) - canonical = canonical_content_hash(raw) - if recipe.factor_id in resumed: - sha = file_sha256(target) - else: - sha = atomic_write_parquet(raw, target) - rows.append( - manifest_row(recipe.factor_id, "minute", raw, canonical, sha, target.name) +def _verify_one_panel( + factor_id: str, + expected: DocManifestRow, + panels_dir: Path, + machine_rows: dict[str, dict], +) -> tuple[PanelVerification, dict | None]: + problems: list[str] = [] + path = panels_dir / f"{factor_id}.parquet" + if not path.exists(): + return ( + PanelVerification( + factor_id=factor_id, + file=path.name, + rows=0, + canonical_sha256="", + problems=(f"frozen panel missing from disk: {path}",), + ), + None, ) - canonical_by_id[recipe.factor_id] = canonical - logger.info( - "frozen minute %s: rows=%d %s canonical=%s (reconciled vs %s)", - recipe.factor_id, len(raw), covered_note, canonical, - reconciliations[recipe.factor_id]["artifact"], + try: + panel = read_frozen_panel(path, factor_id) + except (ValueError, TypeError, OSError) as exc: + return ( + PanelVerification( + factor_id=factor_id, + file=path.name, + rows=0, + canonical_sha256="", + problems=(f"frozen panel unreadable: {type(exc).__name__}: {exc}",), + ), + None, ) - # -- determinism double-run (2 minute + 1 book) -------------------------- # - determinism: dict[str, dict] = {} - determinism_subjects = tuple(fid for fid in DETERMINISM_FACTORS if _keep(fid)) - for factor_id in determinism_subjects: - rebuilt = _rebuild_for_determinism( - factor_id, cfg, symbols, panel, panel_dates, logger + canonical = canonical_content_hash(panel) + derived = manifest_row( + factor_id, + str(expected.kind or (machine_rows.get(factor_id, {}) or {}).get("kind", "")), + panel, + canonical, + file_sha256(path), + path.name, + ) + + if canonical != expected.canonical_sha256: + problems.append( + f"canonical content hash {canonical} != the git-tracked " + f"{expected.canonical_sha256} — these are NOT the frozen bytes." ) - second = canonical_content_hash(rebuilt) - first = canonical_by_id[factor_id] - determinism[factor_id] = {"first": first, "second": second, "ok": second == first} - if second != first: - raise RuntimeError( - f"determinism double-run FAILED for {factor_id}: {first} != {second}" + for field, want in ( + ("rows", expected.rows), + ("n_symbols", expected.n_symbols), + ("n_nan", expected.n_nan), + ("mean", expected.mean), + ("std", expected.std), + ): + if want is not None and derived[field] != want: + problems.append( + f"{field}: git-tracked document says {want!r}, panel gives " + f"{derived[field]!r}" + ) + if expected.file_sha256 is not None and derived["file_sha256"] != expected.file_sha256: + problems.append( + f"file sha256 {derived['file_sha256']} != the git-tracked " + f"{expected.file_sha256} — the file was rewritten" + + ( + " (its CONTENT still matches, so this is a re-write, not a value " + "change — but a frozen-forever file has no legitimate reason to " + "be rewritten)." + if canonical == expected.canonical_sha256 + else "." ) - logger.info("determinism double-run ok: %s %s", factor_id, first) - - header = { - "producing_git_sha": _git_head_sha(), - "config": str(config_path), - "universe": f"{cfg.universe.type}:{cfg.universe.index_code}", - "window": f"{cfg.data.start}..{cfg.data.end}", - "cache_root": str(cfg.data.cache.root_dir), - "panel_store_name": PANEL_STORE_NAME, - "panel_rows": int(len(panel)), - "panel_dates": int(len(panel_dates)), - "universe_symbols": int(len(symbols)), - "stk_mins_live_calls": int(live_calls_total), - "resumed_factors": ",".join(resumed) if resumed else "none", - "selected_factors": "all" if selected is None else ",".join(selected), - "elapsed_seconds": round(time.monotonic() - started, 1), - "generated_utc": pd.Timestamp.now(tz="UTC").strftime("%Y-%m-%d %H:%M:%S"), - "canonical_hash_version": CANONICAL_HASH_VERSION, - "determinism_double_run": "; ".join( - f"{fid}:{'ok' if determinism[fid]['ok'] else 'FAIL'}" - for fid in determinism_subjects ) - or "none (no determinism subject selected)", - "artifact_reconciliation": ( - f"{len(reconciliations)}/{len(kept_recipes)} minute factors reconciled " - "against eval_*_no_book.json data_coverage (book factors carry no " - "coverage fields in the eval artifacts — disclosed, not skipped silently)" - ), - } - manifest = { - "header": header, - "rows": rows, - "reconciliation": reconciliations, - "determinism": determinism, - } - manifest_json = out_root / "manifest.json" - manifest_md = out_root / "manifest.md" - atomic_write_text(json.dumps(manifest, indent=2, sort_keys=True), manifest_json) - atomic_write_text(render_manifest_markdown(header, rows), manifest_md) - logger.info( - "panel freeze complete: %d panels, %ss, manifest=%s", - len(rows), header["elapsed_seconds"], manifest_json, - ) - return FreezeResult( - output_root=out_root, - manifest_json=manifest_json, - manifest_md=manifest_md, - rows=tuple(rows), - header=header, - elapsed=time.monotonic() - started, + recorded = machine_rows.get(factor_id) + if recorded is None: + problems.append("no row for this factor in manifest.json") + else: + for field in MANIFEST_ROW_FIELDS: + if field not in recorded: + problems.append(f"manifest.json row is missing field {field!r}") + continue + if recorded[field] != derived[field]: + problems.append( + f"manifest.json {field}: recorded {recorded[field]!r}, panel " + f"gives {derived[field]!r}" + ) + + return ( + PanelVerification( + factor_id=factor_id, + file=path.name, + rows=int(derived["rows"]), + canonical_sha256=canonical, + problems=tuple(problems), + ), + derived, ) -def _rebuild_for_determinism( - factor_id: str, - cfg, - symbols: list[str], - panel: pd.DataFrame, - panel_dates: pd.Index, - logger: logging.Logger, -) -> pd.Series: - """Second, independent build of one factor's RAW panel (same chains). - - Minute factors re-run the runner loader end-to-end (a fresh cache-store - read); book factors re-instantiate ``_build_book_factors()`` and recompute - from the shared enriched panel (the panel load itself is once-per-process, - exactly as in the runners; its own determinism is covered by the cache - equivalence suites). +def verify_all( + baseline_root: Path | str = DEFAULT_OUTPUT_ROOT, + *, + show_manifest: bool = False, +) -> list[VerifyResult]: + """Verify BOTH frozen panel trees: the D1 baseline and the PR-C reference.""" + baseline_root = Path(baseline_root) + repo_root = repo_root_for(baseline_root) + return [ + verify_frozen_panels( + baseline_root, repo_root / D1_MANIFEST_DOC, show_manifest=show_manifest + ), + verify_frozen_panels( + baseline_root / PR_C_SUBDIR, + repo_root / PR_C_MANIFEST_DOC, + show_manifest=show_manifest, + ), + ] + + +def repo_root_for(baseline_root: Path | str) -> Path: + """Where to look for the git-tracked manifest documents. + + ``docs/`` sits next to the repository root, while the baseline root may be + an arbitrary directory (a copy under test, say). Walk up from the baseline + root looking for ``docs/factors``; fall back to this module's own repository + so a copied tree still verifies against the documents in git. """ - from qt.eval_jump_amount_corr import _build_book_factors + for candidate in (baseline_root, *baseline_root.parents): + if (candidate / D1_MANIFEST_DOC).exists(): + return candidate + return Path(__file__).resolve().parent.parent + - for factor in _build_book_factors(): - if factor.name == factor_id: - return factor.compute(panel).rename(factor_id) - for recipe in minute_recipes(): - if recipe.factor_id == factor_id: - load = recipe.build(cfg, symbols, panel, logger) - return load.factor[ - load.factor.index.get_level_values(DATE_LEVEL).isin(panel_dates) - ].rename(factor_id) - raise KeyError(f"unknown determinism factor_id {factor_id!r}") +# --------------------------------------------------------------------------- # +# Retired regeneration entry point +# --------------------------------------------------------------------------- # +def run_panel_freeze(*args, **kwargs): + """RETIRED (D5 C6). Always raises :class:`RegenerationRetiredError`. + + Kept as a stub rather than deleted outright so that a stale caller — a + script, a notebook, a copied command line — gets the retirement explanation + instead of an ``AttributeError`` that reads like a packaging accident. + """ + raise RegenerationRetiredError( + retirement_message( + "python -m qt.panel_freeze", + "the D1 RAW factor-value panel baseline, which is frozen-forever", + "The rebuild called the eleven legacy eval runners' private " + "`_load_*_panel` loaders, which C6 deletes; and rebuilding this " + "baseline from the tree under validation is exactly the " + "empty-reconciliation failure mode the provenance rule forbids.", + "it recomputes every frozen panel's canonical content hash and " + "manifest row and checks them against the hashes authored in " + "`docs/factors/`.", + ) + ) def main(argv: list[str] | None = None) -> int: - """CLI: ``python -m qt.panel_freeze [--config ...] [--output-root ...]``.""" + """CLI: ``python -m qt.panel_freeze --verify``. + + The retired flags are still ACCEPTED by the parser so that someone typing + the old documented command gets the retirement explanation rather than + ``unrecognized arguments: --resume``, which reads like a version mismatch. + """ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument("--config", default=DEFAULT_CONFIG) - parser.add_argument("--output-root", default=DEFAULT_OUTPUT_ROOT) + parser.add_argument("--config", default=DEFAULT_CONFIG, help=argparse.SUPPRESS) + parser.add_argument("--output-root", default=DEFAULT_OUTPUT_ROOT, help=argparse.SUPPRESS) + parser.add_argument("--resume", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--only", default=None, help=argparse.SUPPRESS) parser.add_argument( - "--resume", + "--baseline-root", + default=None, + help="frozen baseline root (default: the --output-root location)", + ) + parser.add_argument( + "--verify", action="store_true", - help=( - "complete an interrupted freeze: existing minute panels are read back " - "from their frozen files and re-verified (processing + eval-artifact " - "reconciliation) instead of being rebuilt from the minute cache" - ), + help="verify the frozen panels against the git-tracked manifest documents", ) parser.add_argument( - "--only", - default=None, - help=( - "comma-separated factor ids to freeze (default: all 14). Used to " - "re-freeze a SINGLE corrected factor into its own --output-root; an " - "unknown id is a readable error, never a silently empty freeze" - ), + "--show-manifest", + action="store_true", + help="also print the manifest table re-derived from the panels on disk", ) args = parser.parse_args(argv) - only = None if args.only is None else tuple(x for x in args.only.split(",") if x) - result = run_panel_freeze( - args.config, args.output_root, resume=args.resume, only=only - ) - print(f"panels frozen: {len(result.rows)} -> {result.output_root / 'panels'}") - print(f"manifest: {result.manifest_json}") - print(f"stk_mins_live_calls: {result.header['stk_mins_live_calls']}") - print(f"elapsed_seconds: {result.header['elapsed_seconds']}") - return 0 + + if not args.verify: + try: + run_panel_freeze() + except RegenerationRetiredError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + + root = Path(args.baseline_root or args.output_root) + results = verify_all(root, show_manifest=args.show_manifest) + failed = False + for result in results: + print( + f"{result.root}: verified {result.n_ok}/{len(result.panels)} panels " + f"against {result.doc.name} (producing sha " + f"{result.producing_git_sha or 'unrecorded'})" + ) + for problem in result.problems: + print(f" PROBLEM {problem}") + for panel in result.panels: + for problem in panel.problems: + print(f" PROBLEM {panel.factor_id}: {problem}") + if result.rendered_manifest: + print(result.rendered_manifest) + failed = failed or not result.ok + print("panel freeze verification: " + ("FAILED" if failed else "OK")) + return 1 if failed else 0 if __name__ == "__main__": # pragma: no cover - thin CLI shim diff --git a/qt/panel_reconcile.py b/qt/panel_reconcile.py index 768a672..7aefe6c 100644 --- a/qt/panel_reconcile.py +++ b/qt/panel_reconcile.py @@ -1,10 +1,9 @@ -"""D2 cell-by-cell reconciliation of the migrated engine vs the D1 frozen baseline. +"""D2 cell-by-cell panel reconciliation — VERIFY-ONLY since D5 C6. -Design v3.2 §五 leg 4 (R11), pulled forward into D2 by the task card: rebuild -all 14 closing factors' RAW panels on the CURRENT tree (the ``qt.panel_freeze`` -recipes route through the ``data.clean`` shims, so they exercise the migrated -``factors.compute.minute`` math automatically), write them to a SEPARATE -directory, and compare each panel CELL BY CELL against the frozen D1 baseline: +D2 rewrote the factor math into ``factors.compute.minute``. To show the rewrite +changed no value, all 14 closing factors' RAW panels were rebuilt on the D2 tree +into ``artifacts/refactor_baseline/panels_d2`` and compared CELL BY CELL against +the D1 frozen baseline: * the (date, symbol) index sets must be IDENTICAL; * the NaN sets must be IDENTICAL (a NaN-set change would be an undeclared @@ -13,46 +12,76 @@ ``|new - old| / max(|old|, |new|)`` must be <= 1e-12 (the float-reordering budget; any excess means STOP AND FIX THE ENGINE, never widen the budget). +It passed 14/14 at ``max_rel_diff = 0.0`` — bit-identical, canonical hashes +equal. Both sides are now frozen. + +REGENERATION IS RETIRED (owner ruling 2026-07-28, D5 C6) +--------------------------------------------------------- +The rebuild ran the ``qt.panel_freeze`` recipes, i.e. the eleven legacy eval +runners' private ``_load_*_panel`` loaders, which C6 deletes. Retired rather +than left to break on first use. + +What survives is the part that never needed the runners: **the comparison +itself**. ``--verify`` re-derives the D2 verdict from the frozen bytes of both +sides — it re-reads the two panel sets from disk and re-runs +:func:`compare_panels` over all 14 factors. That is a real check, not a +restatement of a recorded one: if either frozen tree drifted, the cells no +longer match and the run goes red. + Anti-empty-reconciliation provenance (the ``compare_postmerge.py`` lesson): -* the frozen side is NEVER regenerated here — it is read from - ``artifacts/refactor_baseline/panels`` exactly as D1 froze it, and each - frozen file's canonical content hash is first checked against the D1 - ``manifest.json`` (a stale/clobbered baseline fails loudly before any - comparison is trusted); -* the comparison reads BOTH sides from disk through two independent file - reads — the freshly built panel is written to - ``artifacts/refactor_baseline/panels_d2`` first and read back, so no shared - in-memory object can make the equality vacuous. - -Run: ``python -m qt.panel_reconcile`` (deliberately NOT registered in qt/cli; -``--resume`` reads back already-written panels_d2 files after checking that -they were produced at the SAME git SHA recorded in ``manifest_d2.json``). +* neither side is ever regenerated here; +* both sides are read from disk through two INDEPENDENT file reads, so no + shared in-memory object can make the equality vacuous; +* before any cell is compared, both sides are checked against manifests — the + D2 panels against the git-tracked D1 document (they are expected to be + bit-identical to it, which IS the D2 result) and against their own + ``manifest_d2.json``. + +Run (deliberately NOT registered in qt/cli, as before):: + + python -m qt.panel_reconcile --verify """ from __future__ import annotations import argparse import json -import time +import sys from dataclasses import dataclass from pathlib import Path import numpy as np import pandas as pd -from data.clean.schema import DATE_LEVEL +from qt.frozen_manifest_doc import parse_doc_manifest from qt.panel_freeze import ( + D1_MANIFEST_DOC, DEFAULT_CONFIG, PANEL_STORE_NAME, - atomic_write_parquet, - atomic_write_text, + RegenerationRetiredError, canonical_content_hash, - minute_recipes, read_frozen_panel, - _git_head_sha, + repo_root_for, + retirement_message, + verify_frozen_panels, ) +__all__ = [ + "CellComparison", + "D2_SUBDIR", + "DEFAULT_BASELINE_ROOT", + "DEFAULT_CONFIG", + "PANEL_STORE_NAME", + "RELATIVE_TOLERANCE", + "ReconcileVerification", + "compare_panels", + "main", + "render_report", + "run_panel_reconcile", + "verify_d2_reconciliation", +] + DEFAULT_BASELINE_ROOT = "artifacts/refactor_baseline" D2_SUBDIR = "panels_d2" RELATIVE_TOLERANCE = 1e-12 @@ -123,29 +152,6 @@ def compare_panels(frozen: pd.Series, new: pd.Series, factor_id: str) -> CellCom ) -def _verify_frozen_against_manifest(baseline_root: Path) -> dict[str, str]: - """Check every frozen panel's canonical hash against the D1 manifest. - - Returns {factor_id: canonical_hash}. Any mismatch raises — a clobbered or - regenerated baseline must never be silently reconciled against. - """ - manifest = json.loads((baseline_root / "manifest.json").read_text(encoding="utf-8")) - frozen_hashes: dict[str, str] = {} - for row in manifest["rows"]: - factor_id = row["factor_id"] - path = baseline_root / "panels" / row["file"] - frozen = read_frozen_panel(path, factor_id) - actual = canonical_content_hash(frozen) - if actual != row["canonical_sha256"]: - raise RuntimeError( - f"frozen baseline {row['file']} canonical hash {actual} does not " - f"match the D1 manifest ({row['canonical_sha256']}) — the baseline " - "has been altered since the freeze; refusing to reconcile." - ) - frozen_hashes[factor_id] = actual - return frozen_hashes - - def render_report(rows: list[CellComparison], header: dict) -> str: lines = ["# D2 panel reconciliation vs the D1 frozen baseline", ""] for key in sorted(header): @@ -169,162 +175,221 @@ def render_report(rows: list[CellComparison], header: dict) -> str: return "\n".join(lines) -def run_panel_reconcile( - config_path: str = DEFAULT_CONFIG, - baseline_root: str | Path = DEFAULT_BASELINE_ROOT, - *, - resume: bool = False, -) -> list[CellComparison]: - """Rebuild the 14 RAW panels on the current tree and reconcile vs the freeze.""" - from qt.config import load_config - from qt.eval_jump_amount_corr import _build_book_factors, _check_preconditions - from qt.pipeline import ( - _build_cache, - _build_universe, - _load_panel, - _log_run_cache_stats, - _make_logger, - _maybe_enrich_covariates, - _maybe_enrich_value, - ) +# --------------------------------------------------------------------------- # +# Verification +# --------------------------------------------------------------------------- # +@dataclass(frozen=True) +class ReconcileVerification: + """Outcome of re-deriving the D2 verdict from the two frozen panel sets.""" + + baseline_root: Path + comparisons: tuple[CellComparison, ...] + problems: tuple[str, ...] + d1_ok: bool + recorded_all_ok: bool | None - started = time.monotonic() - root = Path(baseline_root) - d2_dir = root / D2_SUBDIR - d2_dir.mkdir(parents=True, exist_ok=True) - head_sha = _git_head_sha() - - # provenance for --resume: a panels_d2 file may only be reused if it was - # produced at the SAME git SHA (else it might carry an older engine). - manifest_d2_path = root / "manifest_d2.json" - prior: dict = {} - if manifest_d2_path.exists(): - prior = json.loads(manifest_d2_path.read_text(encoding="utf-8")) - if resume and prior and prior.get("producing_git_sha") != head_sha: - raise RuntimeError( - f"--resume refused: existing panels_d2 were produced at " - f"{prior.get('producing_git_sha')} but HEAD is {head_sha}; delete " - f"{d2_dir} to rebuild from scratch." + @property + def ok(self) -> bool: + return ( + self.d1_ok + and not self.problems + and bool(self.comparisons) + and all(c.ok for c in self.comparisons) ) - frozen_hashes = _verify_frozen_against_manifest(root) - cfg = load_config(config_path) - _check_preconditions(cfg) - cfg = cfg.model_copy( - update={"data": cfg.data.model_copy(update={"output_name": PANEL_STORE_NAME})} - ) - log_path = Path(cfg.output.log_dir) / "panel_reconcile.log" - logger = _make_logger(log_path, name="qt.panel_reconcile") - logger.info("panel reconcile: config=%s baseline_root=%s sha=%s", - config_path, root, head_sha) - - cache = _build_cache(cfg) - universe, symbols = _build_universe(cfg, logger, cache) - panel = _load_panel(cfg, symbols, logger, cache) - book_factors = _build_book_factors() - panel = _maybe_enrich_value(cfg, panel, symbols, book_factors, logger, cache) - panel = _maybe_enrich_covariates(cfg, panel, symbols, logger, cache) - _log_run_cache_stats(cache, logger) - panel_dates = pd.Index( - pd.unique(panel.index.get_level_values(DATE_LEVEL)), name=DATE_LEVEL - ) +def verify_d2_reconciliation( + baseline_root: Path | str = DEFAULT_BASELINE_ROOT, + doc_path: Path | str | None = None, +) -> ReconcileVerification: + """Re-derive the D2 cell-by-cell verdict from the frozen bytes of both sides. - results: list[CellComparison] = [] - d2_rows: list[dict] = [] + Order matters. The D1 side is verified FIRST, against the git-tracked + document (delegated to :func:`qt.panel_freeze.verify_frozen_panels`, so + there is one implementation of "is this the frozen baseline?"). Comparing + two panel sets while neither has been authenticated would report agreement + between two unknowns. - def _reconcile_one(factor_id: str, raw: pd.Series | None) -> None: - target = d2_dir / f"{factor_id}.parquet" - if raw is None: # resume path: reuse the already-written d2 panel - logger.info("resume: reading back %s", target.name) - else: - atomic_write_parquet(raw.rename(factor_id), target) - new = read_frozen_panel(target, factor_id) # file read #1 - frozen = read_frozen_panel(root / "panels" / f"{factor_id}.parquet", factor_id) # #2 - comp = compare_panels(frozen, new, factor_id) - results.append(comp) - d2_rows.append( - {"factor_id": factor_id, "canonical_sha256": canonical_content_hash(new), - "frozen_sha256": frozen_hashes[factor_id], "rows": comp.rows} - ) - logger.info( - "reconciled %s: ok=%s max_rel=%.3e nan_frozen_only=%d nan_new_only=%d", - factor_id, comp.ok, comp.max_rel_diff, - comp.nan_only_in_frozen, comp.nan_only_in_new, + The D2 side is then checked against the SAME git-tracked hashes. That is not + a shortcut: "bit-identical to D1" is precisely what the D2 reconciliation + concluded, so the D1 table IS the D2 expectation, and it is the only one of + the two that lives in git. + + ``doc_path`` overrides which manifest document supplies the expectations; + it exists so the tests can build a small frozen tree with its own document + instead of being forced through the 14 real factors. + """ + baseline_root = Path(baseline_root) + doc = Path(doc_path) if doc_path is not None else _doc_path(baseline_root) + problems: list[str] = [] + + d1 = verify_frozen_panels(baseline_root, doc) + for problem in d1.problems: + problems.append(f"D1 side: {problem}") + for panel in d1.panels: + for problem in panel.problems: + problems.append(f"D1 side {panel.factor_id}: {problem}") + + expected = parse_doc_manifest(doc) + d2_dir = baseline_root / D2_SUBDIR + if not d2_dir.is_dir(): + problems.append(f"D2 panel directory not found: {d2_dir}") + return ReconcileVerification( + baseline_root=baseline_root, + comparisons=(), + problems=tuple(problems), + d1_ok=d1.ok, + recorded_all_ok=None, ) - # book factors (raw, pre-processing) — same call shape as the freeze - for factor in book_factors: - target = d2_dir / f"{factor.name}.parquet" - if resume and target.exists(): - _reconcile_one(factor.name, None) - else: - _reconcile_one(factor.name, factor.compute(panel).rename(factor.name)) - - # minute factors — the freeze recipes, now routed through the D2 shims - live_total = 0 - for recipe in minute_recipes(): - target = d2_dir / f"{recipe.factor_id}.parquet" - if resume and target.exists(): - _reconcile_one(recipe.factor_id, None) + on_disk = {path.stem for path in d2_dir.glob("*.parquet")} + for extra in sorted(on_disk - set(expected)): + problems.append(f"unregistered D2 panel on disk: {extra}.parquet") + + recorded_rows, recorded_all_ok = _read_manifest_d2(d2_dir.parent, problems) + + comparisons: list[CellComparison] = [] + for factor_id in sorted(expected): + d2_path = d2_dir / f"{factor_id}.parquet" + d1_path = baseline_root / "panels" / f"{factor_id}.parquet" + if not d2_path.exists() or not d1_path.exists(): + problems.append( + f"{factor_id}: cannot compare — missing " + f"{'D2' if not d2_path.exists() else 'D1'} panel on disk." + ) continue - load = recipe.build(cfg, symbols, panel, logger) - live = int(load.live_calls) - live_total += live - if live != 0: - raise RuntimeError( - f"{recipe.factor_id}: stk_mins_live_calls={live} != 0 — the " - "reconciliation is cache-only by contract." + # TWO independent file reads; no shared in-memory object. + new = read_frozen_panel(d2_path, factor_id) + frozen = read_frozen_panel(d1_path, factor_id) + comparison = compare_panels(frozen, new, factor_id) + comparisons.append(comparison) + + d2_hash = canonical_content_hash(new) + if d2_hash != expected[factor_id].canonical_sha256: + problems.append( + f"{factor_id}: D2 panel canonical hash {d2_hash} != the git-tracked " + f"{expected[factor_id].canonical_sha256}. D2 concluded the rewrite " + "was bit-identical, so these must agree." ) - raw = load.factor[ - load.factor.index.get_level_values(DATE_LEVEL).isin(panel_dates) - ].rename(recipe.factor_id) - _reconcile_one(recipe.factor_id, raw) - - header = { - "producing_git_sha": head_sha, - "config": str(config_path), - "baseline_root": str(root), - "relative_tolerance": repr(RELATIVE_TOLERANCE), - "stk_mins_live_calls": live_total, - "panels": len(results), - "all_ok": all(r.ok for r in results), - "elapsed_seconds": round(time.monotonic() - started, 1), - "generated_utc": pd.Timestamp.now(tz="UTC").strftime("%Y-%m-%d %H:%M:%S"), - } - atomic_write_text( - json.dumps({"producing_git_sha": head_sha, "rows": d2_rows, "header": header}, - indent=2, sort_keys=True), - manifest_d2_path, + recorded = recorded_rows.get(factor_id) + if recorded is None: + problems.append(f"{factor_id}: no row in manifest_d2.json") + else: + if recorded.get("canonical_sha256") != d2_hash: + problems.append( + f"{factor_id}: manifest_d2.json records canonical " + f"{recorded.get('canonical_sha256')}, panel gives {d2_hash}" + ) + if recorded.get("rows") != comparison.rows: + problems.append( + f"{factor_id}: manifest_d2.json records rows " + f"{recorded.get('rows')!r}, panel has {comparison.rows}" + ) + + if recorded_all_ok is False: + problems.append( + "manifest_d2.json records all_ok=False — the frozen D2 reconciliation " + "did not pass, and nothing downstream may treat it as if it had." + ) + return ReconcileVerification( + baseline_root=baseline_root, + comparisons=tuple(comparisons), + problems=tuple(problems), + d1_ok=d1.ok, + recorded_all_ok=recorded_all_ok, + ) + + +def _doc_path(baseline_root: Path) -> Path: + return repo_root_for(baseline_root) / D1_MANIFEST_DOC + + +def _read_manifest_d2( + root: Path, problems: list[str] +) -> tuple[dict[str, dict], bool | None]: + path = root / "manifest_d2.json" + if not path.exists(): + problems.append(f"D2 manifest missing: {path}") + return {}, None + document = json.loads(path.read_text(encoding="utf-8")) + rows = {str(row["factor_id"]): row for row in document.get("rows", [])} + return rows, document.get("header", {}).get("all_ok") + + +# --------------------------------------------------------------------------- # +# Retired regeneration entry point +# --------------------------------------------------------------------------- # +def run_panel_reconcile(*args, **kwargs): + """RETIRED (D5 C6). Always raises :class:`RegenerationRetiredError`. + + Kept as a stub so a stale caller gets the retirement explanation rather than + an ``AttributeError`` that reads like a packaging accident. + """ + raise RegenerationRetiredError( + retirement_message( + "python -m qt.panel_reconcile", + "the D2 rebuilt panel set (`panels_d2`), which is frozen-forever", + "The rebuild ran the qt.panel_freeze recipes, i.e. the eleven legacy " + "eval runners' private `_load_*_panel` loaders, which C6 deletes.", + "the COMPARISON never needed those loaders, so it still runs — it " + "re-reads both frozen panel sets from disk and re-derives the " + "cell-by-cell D2 verdict.", + ) ) - atomic_write_text(render_report(results, header), root / "reconcile_d2.md") - logger.info("panel reconcile complete: all_ok=%s (%ss)", - header["all_ok"], header["elapsed_seconds"]) - return results def main(argv: list[str] | None = None) -> int: + """CLI: ``python -m qt.panel_reconcile --verify``. + + ``--resume`` is still accepted by the parser so the old documented command + produces the retirement explanation rather than ``unrecognized arguments``, + which reads like a version mismatch. + """ parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) - parser.add_argument("--config", default=DEFAULT_CONFIG) + parser.add_argument("--config", default=DEFAULT_CONFIG, help=argparse.SUPPRESS) parser.add_argument("--baseline-root", default=DEFAULT_BASELINE_ROOT) + parser.add_argument("--resume", action="store_true", help=argparse.SUPPRESS) parser.add_argument( - "--resume", action="store_true", - help="reuse already-written panels_d2 files (same-SHA guarded)", + "--doc", + default=None, + help="manifest document supplying the expected hashes " + f"(default: {D1_MANIFEST_DOC})", ) - args = parser.parse_args(argv) - results = run_panel_reconcile( - args.config, args.baseline_root, resume=args.resume + parser.add_argument( + "--verify", + action="store_true", + help="re-derive the D2 cell-by-cell verdict from the frozen bytes", ) - for r in results: + args = parser.parse_args(argv) + + if not args.verify: + try: + run_panel_reconcile() + except RegenerationRetiredError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + + result = verify_d2_reconciliation(args.baseline_root, args.doc) + for comparison in result.comparisons: print( - f"{'OK ' if r.ok else 'MISMATCH'} {r.factor_id}: rows={r.rows} " - f"max_rel={r.max_rel_diff:.3e} max_abs={r.max_abs_diff:.3e} " - f"nan_frozen_only={r.nan_only_in_frozen} nan_new_only={r.nan_only_in_new} " - f"hash_equal={r.hashes_equal}" + f"{'OK ' if comparison.ok else 'MISMATCH'} {comparison.factor_id}: " + f"rows={comparison.rows} max_rel={comparison.max_rel_diff:.3e} " + f"max_abs={comparison.max_abs_diff:.3e} " + f"nan_frozen_only={comparison.nan_only_in_frozen} " + f"nan_new_only={comparison.nan_only_in_new} " + f"hash_equal={comparison.hashes_equal}" ) - ok = all(r.ok for r in results) - print(f"reconcile: {'ALL OK' if ok else 'MISMATCH — stop and fix the engine'}") - return 0 if ok else 1 + for problem in result.problems: + print(f" PROBLEM {problem}") + print( + f"D1 side verified: {result.d1_ok}; recorded all_ok: {result.recorded_all_ok}; " + f"{len(result.comparisons)} factors re-compared" + ) + print( + "panel reconcile verification: " + + ("OK" if result.ok else "FAILED — stop and investigate the frozen trees") + ) + return 0 if result.ok else 1 if __name__ == "__main__": # pragma: no cover - thin CLI shim diff --git a/tests/fixtures/frozen_baseline.py b/tests/fixtures/frozen_baseline.py new file mode 100644 index 0000000..47523b3 --- /dev/null +++ b/tests/fixtures/frozen_baseline.py @@ -0,0 +1,173 @@ +"""Builders for a SYNTHETIC frozen-baseline tree (D1 / D2 verify-only tests). + +The real baseline is 109 MB of parquet and must never be written to, so the +verification tests operate on a miniature tree with the same shape: a +``panels/`` directory, a machine ``manifest.json`` (rows + reconciliation +block), an optional ``panels_d2/`` with its ``manifest_d2.json``, and a +git-tracked-style Markdown manifest document carrying the expected hashes. + +The tree is built through the production writer (``atomic_write_parquet``) and +the production row/renderer helpers, so a synthetic tree is a valid frozen tree +by construction rather than by transcription. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import numpy as np +import pandas as pd + +from data.clean.schema import DATE_LEVEL, SYMBOL_LEVEL +from qt.panel_freeze import ( + RECONCILE_FLOAT_FIELDS, + RECONCILE_INT_FIELDS, + atomic_write_parquet, + canonical_content_hash, + manifest_row, +) + +#: Two factors is enough to show that a problem is attributed to ONE of them. +DEFAULT_FACTORS = {"alpha_20": "minute", "book_x": "book"} +DATES = ("2024-01-02", "2024-01-03", "2024-01-04") +SYMBOLS = ("000001.SZ", "600000.SH") + + +def make_panel(factor_id: str, *, offset: float = 0.0, nan_last: bool = True) -> pd.Series: + """A small, deterministic (date, symbol) factor panel.""" + keys = [(pd.Timestamp(d), s) for d in DATES for s in SYMBOLS] + values = [offset + 0.5 * position for position in range(len(keys))] + if nan_last: + values[-1] = float("nan") + index = pd.MultiIndex.from_tuples(keys, names=[DATE_LEVEL, SYMBOL_LEVEL]) + return pd.Series(np.asarray(values, dtype="float64"), index=index, name=factor_id) + + +def render_doc(rows: list[dict], *, producing_sha: str, include_file_sha: bool = True) -> str: + """A Markdown manifest document in the shape the real ones use. + + Deliberately includes a decoy table BEFORE the manifest table (the real + documents carry several), so the header-driven table finder is exercised + rather than a "first table wins" shortcut. + """ + columns = ["factor_id", "kind", "rows", "n_symbols", "n_nan", "mean", "std", + "canonical_sha256"] + if include_file_sha: + columns.append("file_sha256") + lines = [ + "# synthetic manifest", + "", + f"- **producing SHA**: `{producing_sha}`", + "", + "| item | value |", + "|---|---|", + "| universe | synthetic |", + "", + "| " + " | ".join(columns) + " |", + "|" + "|".join("---" for _ in columns) + "|", + ] + for row in rows: + cells = [] + for column in columns: + value = row[column] + cells.append(repr(value) if isinstance(value, float) else str(value)) + lines.append("| " + " | ".join(cells) + " |") + lines.append("") + return "\n".join(lines) + + +def build_frozen_tree( + root: Path, + *, + factors: dict[str, str] | None = None, + producing_sha: str = "0" * 40, + with_d2: bool = False, + include_file_sha: bool = True, +) -> Path: + """Write a complete synthetic frozen tree; return the manifest document path.""" + factors = dict(factors or DEFAULT_FACTORS) + panels_dir = root / "panels" + rows: list[dict] = [] + for position, (factor_id, kind) in enumerate(factors.items()): + panel = make_panel(factor_id, offset=float(position)) + target = panels_dir / f"{factor_id}.parquet" + file_hash = atomic_write_parquet(panel, target) + rows.append( + manifest_row( + factor_id, kind, panel, canonical_content_hash(panel), file_hash, + target.name, + ) + ) + + reconciliation = { + row["factor_id"]: { + "artifact": f"eval_{row['factor_id']}_no_book.json", + "checks": { + field: {"artifact": 1, "frozen": 1, "ok": True} + for field in RECONCILE_INT_FIELDS + RECONCILE_FLOAT_FIELDS + }, + } + for row in rows + if factors[row["factor_id"]] == "minute" + } + (root / "manifest.json").write_text( + json.dumps( + { + "header": {"producing_git_sha": producing_sha, "panel_rows": 6}, + "rows": rows, + "reconciliation": reconciliation, + }, + indent=2, + sort_keys=True, + ), + encoding="utf-8", + ) + + if with_d2: + d2_dir = root / "panels_d2" + d2_rows = [] + for position, factor_id in enumerate(factors): + panel = make_panel(factor_id, offset=float(position)) + atomic_write_parquet(panel, d2_dir / f"{factor_id}.parquet") + d2_rows.append( + { + "factor_id": factor_id, + "canonical_sha256": canonical_content_hash(panel), + "frozen_sha256": canonical_content_hash(panel), + "rows": int(len(panel)), + } + ) + (root / "manifest_d2.json").write_text( + json.dumps( + { + "producing_git_sha": producing_sha, + "header": {"producing_git_sha": producing_sha, "all_ok": True}, + "rows": d2_rows, + }, + indent=2, + sort_keys=True, + ), + encoding="utf-8", + ) + + doc_path = root / "manifest_doc.md" + doc_path.write_text( + render_doc(rows, producing_sha=producing_sha, include_file_sha=include_file_sha), + encoding="utf-8", + ) + return doc_path + + +def rewrite_panel(root: Path, factor_id: str, panel: pd.Series, *, d2: bool = False) -> None: + """Overwrite one panel of a synthetic tree (the tampering primitive).""" + subdir = "panels_d2" if d2 else "panels" + atomic_write_parquet(panel.rename(factor_id), root / subdir / f"{factor_id}.parquet") + + +def patch_manifest(root: Path, name: str, mutate) -> None: + """Read/modify/write one of the synthetic tree's JSON manifests.""" + path = root / name + document = json.loads(path.read_text(encoding="utf-8")) + mutate(document) + path.write_text(json.dumps(document, indent=2, sort_keys=True), encoding="utf-8") diff --git a/tests/test_cli_retired_eval_commands.py b/tests/test_cli_retired_eval_commands.py new file mode 100644 index 0000000..f680c09 --- /dev/null +++ b/tests/test_cli_retired_eval_commands.py @@ -0,0 +1,91 @@ +"""The eleven per-factor `run-eval-*` commands collapsed into `run-factor-eval`. + +Two things are pinned here. First, the eleven names really are gone from the +parser — a retirement that left them registered would be a rename, not a +convergence. Second, typing one of them still produces the migration route: +argparse answers a removed subcommand with ``invalid choice``, which tells an +operator they mistyped rather than that the command was replaced and by what. + +The historical list of retired names lives in THIS file on purpose. The +production map is derived from ``FACTOR_TO_REPORT_NAME`` so there is no second +copy to drift; a literal list is still needed to assert that the derivation +produces exactly the names that used to exist, and a test is where a historical +claim belongs. +""" + +from __future__ import annotations + +import pytest + +from qt.cli import build_parser, main, retired_eval_commands +from qt.factor_eval_reconcile import FACTOR_TO_REPORT_NAME + +#: The eleven subcommands qt/cli.py registered before D5 C6 (git history). +RETIRED_NAMES = ( + "run-eval-jump-amount-corr", + "run-eval-minute-ideal-amplitude", + "run-eval-amp-marginal-anomaly-vol", + "run-eval-volume-peak-count", + "run-eval-intraday-amp-cut", + "run-eval-peak-interval-kurtosis", + "run-eval-valley-relative-vwap", + "run-eval-valley-ridge-vwap-ratio", + "run-eval-ridge-minute-return", + "run-eval-valley-price-quantile", + "run-eval-peak-ridge-amount-ratio", +) + + +def _registered_commands() -> set[str]: + parser = build_parser() + actions = [a for a in parser._actions if getattr(a, "choices", None)] + return set(actions[0].choices) + + +def test_the_derived_map_is_exactly_the_eleven_names_that_used_to_exist(): + assert set(retired_eval_commands()) == set(RETIRED_NAMES) + assert len(RETIRED_NAMES) == 11 + + +def test_every_retired_name_maps_to_a_real_factor_id(): + mapping = retired_eval_commands() + assert set(mapping.values()) == set(FACTOR_TO_REPORT_NAME) + assert mapping["run-eval-valley-price-quantile"] == "valley_price_quantile_20" + # the one whose command name is NOT its factor id with dashes + assert mapping["run-eval-minute-ideal-amplitude"] == "minute_ideal_amp_10" + + +@pytest.mark.parametrize("name", RETIRED_NAMES) +def test_the_retired_name_is_no_longer_a_subcommand(name: str): + assert name not in _registered_commands() + + +def test_the_unified_command_is_the_one_that_remains(): + commands = _registered_commands() + assert "run-factor-eval" in commands + assert not any(c.startswith("run-eval-") for c in commands) + + +@pytest.mark.parametrize("name", RETIRED_NAMES) +def test_a_retired_name_gets_the_migration_route_not_invalid_choice(name, capsys): + assert main([name, "--config", "config/factor_eval_csi500.yaml"]) == 1 + err = capsys.readouterr().err + assert "retired in D5 C6" in err + assert "run-factor-eval" in err + assert f"--factor {retired_eval_commands()[name]}" in err + assert "invalid choice" not in err + + +def test_an_unknown_run_eval_name_still_falls_through_to_argparse(capsys): + """The hint is for names that really existed. Anything else is a typo and + should get argparse's list of valid choices, not a fabricated migration.""" + with pytest.raises(SystemExit): + main(["run-eval-not-a-real-factor"]) + assert "invalid choice" in capsys.readouterr().err + + +def test_the_hint_does_not_hijack_an_unrelated_command(capsys): + """Only the retired prefix is intercepted; everything else parses normally.""" + with pytest.raises(SystemExit): + main(["run-factor-eval"]) # missing required --config/--factor + assert "retired in D5 C6" not in capsys.readouterr().err diff --git a/tests/test_factor_eval_reconcile.py b/tests/test_factor_eval_reconcile.py index 1a6fd7a..a901e25 100644 --- a/tests/test_factor_eval_reconcile.py +++ b/tests/test_factor_eval_reconcile.py @@ -1980,12 +1980,12 @@ def test_the_registered_description_pairs_match_the_frozen_artifacts_exactly(): DEFAULT_MANIFEST, FrozenExecBaseline, ) - from qt.factor_eval_reconcile import _FACTOR_TO_REPORT_NAME + from qt.factor_eval_reconcile import FACTOR_TO_REPORT_NAME repo = Path(".").resolve() baseline = FrozenExecBaseline(repo / DEFAULT_FROZEN_ROOT, repo / DEFAULT_MANIFEST) for factor_id, (old_text, _new_text) in REGISTERED_SPEC_DESCRIPTION_REWRITES.items(): - frozen = baseline.report_json(_FACTOR_TO_REPORT_NAME[factor_id], "no_book") + frozen = baseline.report_json(FACTOR_TO_REPORT_NAME[factor_id], "no_book") assert frozen["spec"]["description"] == old_text, factor_id diff --git a/tests/test_frozen_manifest_doc.py b/tests/test_frozen_manifest_doc.py new file mode 100644 index 0000000..10aab69 --- /dev/null +++ b/tests/test_frozen_manifest_doc.py @@ -0,0 +1,147 @@ +"""Tests for the git-tracked frozen-baseline manifest document parser. + +The parser is the load-bearing half of "the hashes live in git": if it silently +returned an empty or partial expectation set, ``--verify`` would pass a tampered +tree while looking busy. So every failure mode it is supposed to be loud about +gets a test, and the two REAL documents are parsed here as a coupling check — +a reformat that breaks the table stops the suite, not a verification run. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +from qt.frozen_manifest_doc import parse_doc_manifest, parse_doc_producing_sha +from qt.panel_freeze import D1_MANIFEST_DOC, PR_C_MANIFEST_DOC + +REPO = Path(__file__).resolve().parents[1] + +HEADER = "| factor_id | rows | canonical_sha256 |" +SEPARATOR = "|---|---|---|" +HASH_A = "a" * 64 +HASH_B = "b" * 64 + + +def _doc(tmp_path: Path, body: str, name: str = "doc.md") -> Path: + path = tmp_path / name + path.write_text(body, encoding="utf-8") + return path + + +def test_parses_the_table_and_reads_columns_by_name(tmp_path: Path): + path = _doc( + tmp_path, + "\n".join( + [ + "# heading", + "", + "| item | value |", # decoy table without the required columns + "|---|---|", + "| universe | CSI500 |", + "", + HEADER, + SEPARATOR, + f"| alpha_20 | 10 | `{HASH_A}` |", + f"| beta_20 | **20** | {HASH_B} |", + ] + ), + ) + rows = parse_doc_manifest(path) + assert set(rows) == {"alpha_20", "beta_20"} + assert rows["alpha_20"].canonical_sha256 == HASH_A + assert rows["beta_20"].rows == 20 # bold markers stripped, read by column NAME + + +def test_absent_columns_are_none_rather_than_invented(tmp_path: Path): + """The PR-C table carries no ``kind`` / ``file_sha256``; a verifier that made + values up for them would be checking its own invention.""" + path = _doc( + tmp_path, + "\n".join([HEADER, SEPARATOR, f"| alpha_20 | 10 | {HASH_A} |"]), + ) + row = parse_doc_manifest(path)["alpha_20"] + assert row.kind is None and row.file_sha256 is None and row.n_nan is None + assert row.rows == 10 + + +def test_two_matching_tables_is_a_readable_error(tmp_path: Path): + path = _doc( + tmp_path, + "\n".join( + [HEADER, SEPARATOR, f"| alpha_20 | 10 | {HASH_A} |", "", + HEADER, SEPARATOR, f"| alpha_20 | 10 | {HASH_B} |"] + ), + ) + with pytest.raises(ValueError, match="exactly ONE table"): + parse_doc_manifest(path) + + +def test_no_matching_table_is_a_readable_error(tmp_path: Path): + path = _doc(tmp_path, "# nothing here\n\n| a | b |\n|---|---|\n| 1 | 2 |\n") + with pytest.raises(ValueError, match="exactly ONE table"): + parse_doc_manifest(path) + + +def test_duplicate_factor_id_is_rejected(tmp_path: Path): + path = _doc( + tmp_path, + "\n".join( + [HEADER, SEPARATOR, f"| alpha_20 | 10 | {HASH_A} |", + f"| alpha_20 | 10 | {HASH_B} |"] + ), + ) + with pytest.raises(ValueError, match="duplicate factor id"): + parse_doc_manifest(path) + + +@pytest.mark.parametrize( + "bad", ["not-a-hash", "A" * 64, "a" * 63, ""], +) +def test_a_malformed_hash_is_rejected(tmp_path: Path, bad: str): + path = _doc(tmp_path, "\n".join([HEADER, SEPARATOR, f"| alpha_20 | 10 | {bad} |"])) + with pytest.raises(ValueError, match="64 lowercase hex"): + parse_doc_manifest(path) + + +def test_a_ragged_row_is_rejected(tmp_path: Path): + path = _doc(tmp_path, "\n".join([HEADER, SEPARATOR, f"| alpha_20 | {HASH_A} |"])) + with pytest.raises(ValueError, match="cells but"): + parse_doc_manifest(path) + + +def test_producing_sha_is_read_from_the_document(tmp_path: Path): + sha = "3" * 40 + path = _doc(tmp_path, f"- **producing SHA**: `{sha}` (some note)\n") + assert parse_doc_producing_sha(path) == sha + + +def test_producing_sha_absent_reads_as_none(tmp_path: Path): + assert parse_doc_producing_sha(_doc(tmp_path, "# no provenance line\n")) is None + + +# --------------------------------------------------------------------------- # +# Coupling to the REAL git-tracked documents +# --------------------------------------------------------------------------- # +def test_the_real_d1_document_lists_the_fourteen_frozen_factors(): + rows = parse_doc_manifest(REPO / D1_MANIFEST_DOC) + assert len(rows) == 14 + assert {"value_ep", "value_bp", "volatility_20"} <= set(rows) # the 3 book factors + assert sum(1 for row in rows.values() if row.kind == "minute") == 11 + assert all(row.file_sha256 is not None for row in rows.values()) + + +def test_the_real_pr_c_document_lists_the_one_corrected_panel(): + rows = parse_doc_manifest(REPO / PR_C_MANIFEST_DOC) + assert set(rows) == {"jump_amount_corr_20"} + # The corrected panel is a DIFFERENT factor definition from the D1 row of the + # same name; if these ever hashed equal, one of the two documents is wrong. + d1 = parse_doc_manifest(REPO / D1_MANIFEST_DOC)["jump_amount_corr_20"] + assert rows["jump_amount_corr_20"].canonical_sha256 != d1.canonical_sha256 + + +def test_both_real_documents_pin_a_producing_sha(): + for doc in (D1_MANIFEST_DOC, PR_C_MANIFEST_DOC): + sha = parse_doc_producing_sha(REPO / doc) + assert sha is not None and len(sha) == 40, doc diff --git a/tests/test_hand_anchor_record.py b/tests/test_hand_anchor_record.py new file mode 100644 index 0000000..f9aa488 --- /dev/null +++ b/tests/test_hand_anchor_record.py @@ -0,0 +1,313 @@ +"""Tests for the daily hand-anchor record verification (VERIFY-ONLY since C6). + +The comparison that produced ``daily_engine_compared`` is retired, so what is +left to check is whether the RECORD says what its own numbers imply. These tests +pin both directions: an honest record passes, and every way of editing it into a +false pass comes back red. + +The absence case gets its own test because it is the one with a judgement call +in it: nothing recorded exits NON-ZERO with the wording "NOT VERIFIED". Absence +of evidence is not evidence of failure, but the exit-code channel is binary and +a verification command that reports success without verifying anything is the +empty-reconciliation shape this repository has already committed once. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from qt.hand_anchors_d2 import TOL, pending_engine_line +from qt.hand_anchors_engine_values import ( + DAILY_FACTORS, + AnchorRecordVerification, + main, + relative_difference, + run_engine_comparison, + verify_recorded_comparison, +) +from qt.panel_freeze import RegenerationRetiredError + + +def _row(factor_id="momentum_20", hand=1.0, engine=1.0, **overrides) -> dict: + rel = relative_difference(hand, engine) + row = { + "factor_id": factor_id, + "class": "random", + "date": "2024-05-30", + "symbol": "000001.SZ", + "hand": hand, + "engine": engine, + "rel_diff": rel, + "ok": rel <= TOL, + } + row.update(overrides) + return row + + +def _record(tmp_path: Path, compared: list[dict], pending: list[dict] | None = None, + all_ok: bool | None = None) -> Path: + if all_ok is None: + all_ok = all(bool(row.get("ok")) for row in compared) + path = tmp_path / "hand_anchors_d2.json" + path.write_text( + json.dumps( + { + "seed": 20260724, + "tolerance": TOL, + "daily_pending_engine": pending or [], + "daily_engine_compared": compared, + "all_ok_daily": all_ok, + } + ), + encoding="utf-8", + ) + return path + + +# --------------------------------------------------------------------------- # +# Retired regeneration entry point +# --------------------------------------------------------------------------- # +def test_the_engine_comparison_raises_instead_of_quietly_doing_nothing(): + with pytest.raises(RegenerationRetiredError, match="RETIRED"): + run_engine_comparison() + + +def test_the_retirement_message_does_not_overstate_what_was_lost(): + """This tool leaned on ONE precondition helper, not the eleven private + loaders, and its factors are untouched. Saying otherwise would be the same + class of defect as a report claiming a check it did not run.""" + with pytest.raises(RegenerationRetiredError) as caught: + run_engine_comparison() + text = str(caught.value) + assert "precondition check" in text + assert "four daily factor classes are untouched" in text + assert "python -m qt.hand_anchors_engine_values --verify" in text + # It must NOT claim to check frozen bytes against a git-tracked manifest — + # that is what the two panel tools do, not this one. + assert "git-tracked manifest" not in text + + +def test_a_bare_invocation_is_non_zero_and_explains_itself(capsys): + assert main([]) == 1 + assert "RETIRED" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- # +# Verification of the record +# --------------------------------------------------------------------------- # +def test_an_honest_record_verifies_green(tmp_path: Path): + path = _record(tmp_path, [_row(factor_id=fid) for fid in DAILY_FACTORS]) + result = verify_recorded_comparison(path) + assert result.ok and result.n_compared == 4 and not result.problems + + +def test_a_flipped_ok_flag_is_convicted(tmp_path: Path): + """The record claims a row passed; its own hand/engine numbers say it did + not. This is the edit that a verify-only tool exists to catch.""" + honest = _row(hand=1.0, engine=2.0) + assert honest["ok"] is False # the numbers really do disagree + path = _record(tmp_path, [{**honest, "ok": True}], all_ok=True) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("recorded ok=True" in p for p in result.problems) + + +def test_a_doctored_rel_diff_is_convicted(tmp_path: Path): + path = _record(tmp_path, [_row(hand=1.0, engine=2.0, rel_diff=0.0, ok=True)], + all_ok=True) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("imply" in p for p in result.problems) + + +def test_a_recorded_mismatch_is_reported_even_when_labelled_consistently(tmp_path: Path): + """Row honestly says ok=False and all_ok_daily is honestly False. The record + is self-consistent — and it records a FAILURE, which must not read green.""" + path = _record(tmp_path, [_row(hand=1.0, engine=2.0)]) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("exceed the tolerance" in p for p in result.problems) + + +def test_an_all_ok_flag_that_contradicts_the_rows_is_convicted(tmp_path: Path): + path = _record(tmp_path, [_row(hand=1.0, engine=2.0)], all_ok=True) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("all_ok_daily is recorded as True" in p for p in result.problems) + + +def test_a_pending_anchor_that_was_never_compared_is_convicted(tmp_path: Path): + """A partially filled record must not pass on the strength of the rows that + ARE there.""" + pending = [{"factor_id": "liquidity_20", "date": "2024-05-30", "symbol": "600000.SH"}] + path = _record(tmp_path, [_row()], pending=pending) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("never compared" in p for p in result.problems) + + +def test_a_row_missing_fields_is_convicted(tmp_path: Path): + honest = _row() + honest.pop("engine") + path = _record(tmp_path, [honest], all_ok=True) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("missing field(s)" in p for p in result.problems) + + +def test_an_unexpected_factor_id_is_convicted(tmp_path: Path): + path = _record(tmp_path, [_row(factor_id="not_a_daily_factor")]) + result = verify_recorded_comparison(path) + assert not result.ok + assert any("unexpected factor id" in p for p in result.problems) + + +def test_nothing_recorded_is_not_verified_and_exits_non_zero(tmp_path: Path, capsys): + path = _record(tmp_path, [], pending=[{"factor_id": "momentum_20", + "date": "2024-05-30", + "symbol": "000001.SZ"}]) + result = verify_recorded_comparison(path) + assert isinstance(result, AnchorRecordVerification) + assert not result.recorded and not result.ok and result.n_pending == 1 + assert main(["--verify", "--record", str(path)]) == 1 + assert "NOT VERIFIED (nothing recorded)" in capsys.readouterr().out + + +def test_an_absent_record_file_is_not_verified(tmp_path: Path): + result = verify_recorded_comparison(tmp_path / "nope.json") + assert not result.recorded and not result.ok + + +def test_verify_writes_nothing_into_the_record(tmp_path: Path): + path = _record(tmp_path, [_row(factor_id=fid) for fid in DAILY_FACTORS]) + before = (path.stat().st_mtime_ns, path.stat().st_size) + assert main(["--verify", "--record", str(path)]) == 0 + assert (path.stat().st_mtime_ns, path.stat().st_size) == before + + +@pytest.mark.parametrize( + "hand, engine, expected", + [ + (1.0, 1.0, 0.0), + (0.0, 0.0, 0.0), + (float("nan"), float("nan"), 0.0), + (1.0, float("nan"), float("inf")), + (2.0, 1.0, 0.5), + ], +) +def test_the_relative_difference_rule_matches_the_recorded_one(hand, engine, expected): + assert relative_difference(hand, engine) == expected + + +# --------------------------------------------------------------------------- # +# The run-summary pointer (review MEDIUM-1) +# --------------------------------------------------------------------------- # +def test_the_pending_line_does_not_send_anyone_to_a_command_that_raises(): + """`hand_anchor_rows` printed "(run python -m qt.hand_anchors_engine_values)" + on every successful run — and that command now raises. + + Worse in context: the same function writes `daily_pending_engine` WITHOUT + `daily_engine_compared`, so it is the thing that empties the record. A user + had just wiped the comparison and was then pointed down a dead end. + """ + line = pending_engine_line(20) + assert "20" in line + assert "python -m qt.hand_anchors_engine_values --verify" in line + assert "RETIRED" in line + assert "no longer be completed" in line + # the bare invocation must not appear as an instruction + assert "hand_anchors_engine_values)" not in line + assert "run python -m qt.hand_anchors_engine_values\n" not in line + + +def test_the_pointer_is_composed_by_the_printer_not_restated(): + """Author-once, structurally: the printing module must carry NO literal of + its own. A regex cannot assert that no other sentence points at the retired + command; "there is no other sentence in this file" can.""" + source = (Path(__file__).resolve().parents[1] / "qt" / "hand_anchor_rows.py").read_text() + # The INVOCATION is what must exist once. Naming the module in prose is + # fine and useful; telling someone to run it is the thing that goes stale. + assert "python -m qt.hand_anchors_engine_values" not in source + assert "pending_engine_line" in source + + +def test_the_pointer_lives_where_the_pure_side_can_reach_it(): + """It is in `hand_anchors_d2`, not in the retired module, because the latter + imports the engine and the hand side must never load it. If this ever moves, + `hand_anchor_rows` would drag `data.clean` in and break hand-anchor purity.""" + import qt.hand_anchors_d2 as pure + + assert isinstance(pure.ENGINE_COMPARISON_POINTER, str) + # AST, not substring: the module has a COMMENT naming data.clean to explain + # why it stays out, and a text match cannot tell that from an import. + import ast + + for name in ("hand_anchors_d2.py", "hand_anchor_rows.py"): + path = Path(__file__).resolve().parents[1] / "qt" / name + for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))): + targets: list[str] = [] + if isinstance(node, ast.Import): + targets = [alias.name for alias in node.names] + elif isinstance(node, ast.ImportFrom): + targets = [node.module or ""] + for target in targets: + assert not target.startswith(("factors", "data.clean")), ( + f"{name} imports the engine ({target}); the hand side must not" + ) + + +RETIRED_INVOCATIONS = ( + "python -m qt.panel_freeze", + "python -m qt.panel_reconcile", + "python -m qt.hand_anchors_engine_values", +) +#: Sites allowed to name a retired invocation WITHOUT `--verify`, each because +#: the tool is the SUBJECT of a retirement statement rather than an instruction. +#: Reviewed one by one; anything new must be looked at, not appended blindly. +NAMES_A_RETIRED_TOOL_AS_SUBJECT = { + "qt/panel_freeze.py", # the `tool` argument of retirement_message + "qt/panel_reconcile.py", # ditto + "qt/hand_anchors_engine_values.py", # ditto + "docs/factors/d1_panel_freeze_manifest.md", # "(不带 --verify) 现在是可读的报错" + "docs/factors/pr_c_cutoff_fix_reference_panel.md", # archived [RETIRED, C6] block +} + + +def test_no_new_file_starts_instructing_people_to_run_a_retired_tool(): + """An inventory net, NOT a meaning check — range stated so it is not trusted + further than it reaches. + + It cannot tell an instruction from a description; it only notices when a NEW + file starts naming a retired invocation without `--verify`. That is worth + having because the defect it follows was a pre-existing line in a file the + author never scanned: fixing the two documents and missing the runtime print + is exactly the #82 shape — a guard that only looks where you already looked + confirms only what you already know. `docs/progress/` is excluded as a + historical archive: it records what was true then. + """ + repo = Path(__file__).resolve().parents[1] + offenders: list[str] = [] + for path in sorted(repo.rglob("*")): + if path.suffix not in {".py", ".md", ".yaml", ".txt"} or not path.is_file(): + continue + relative = path.relative_to(repo).as_posix() + if relative.startswith(("docs/progress/", ".git/", "tests/")): + continue + text = path.read_text(encoding="utf-8") + for call in RETIRED_INVOCATIONS: + start = text.find(call) + while start != -1: + # WINDOW, not line: these sentences wrap, so `--verify` routinely + # lands on the following source line. + window = text[start : start + len(call) + 40] + if "--verify" not in window and relative not in NAMES_A_RETIRED_TOOL_AS_SUBJECT: + number = text.count("\n", 0, start) + 1 + offenders.append(f"{relative}:{number}: {call}") + start = text.find(call, start + 1) + assert not offenders, ( + "these name a retired invocation without --verify, from a file not on the " + "reviewed subject list:\n " + "\n ".join(offenders) + ) diff --git a/tests/test_panel_freeze.py b/tests/test_panel_freeze.py index 2bed9dc..75110c1 100644 --- a/tests/test_panel_freeze.py +++ b/tests/test_panel_freeze.py @@ -1,24 +1,24 @@ -"""Network-free tests for qt.panel_freeze (the D1 raw-panel freeze tool). +"""Network-free tests for qt.panel_freeze (VERIFY-ONLY since D5 C6). -Synthetic panels only — the real-data obligations (determinism double-run, -artifact reconciliation, live-call zero) are RUN-time verifications recorded in -the freeze manifest, not unit tests. Here we pin the pure machinery: +Synthetic panels only. Two halves: -* canonical content hash: sensitive to any value / index change, row-order - independent, NaN-payload independent, +0/-0 distinct, loud on malformed input; -* atomic parquet write: readers never see a partial file, failed writes leave - no tmp residue and never touch an existing target; -* manifest row / renderer: field-complete, full-precision, deterministic; -* eval-artifact reconciliation: exact-match discipline — a mismatch raises. +* the pure machinery that DESCRIBES the frozen artifacts — canonical content + hash (sensitive to any value / index change, row-order independent, + NaN-payload independent, +0/-0 distinct, loud on malformed input), the atomic + parquet write it was laid down with, the manifest row and its renderer; +* verification — every way a frozen tree can be wrong must come back red, and + the retired regeneration entry point must fail loudly rather than no-op. Every invariance claim here is paired with a sensitivity assertion on the same axis (the shuffle test alone could be satisfied by a constant hash; the -value/index sensitivity tests kill that degenerate implementation). +value/index sensitivity tests kill that degenerate implementation). The same +rule applies to the verification tests: each starts from a tree that verifies +GREEN, so a red result is attributable to the tampering and not to a fixture +that never verified in the first place. """ from __future__ import annotations -import json import struct from pathlib import Path @@ -28,17 +28,24 @@ from data.clean.schema import DATE_LEVEL, SYMBOL_LEVEL from qt.panel_freeze import ( - DETERMINISM_FACTORS, MANIFEST_ROW_FIELDS, + RegenerationRetiredError, atomic_write_parquet, canonical_content_hash, file_sha256, - freezable_factor_ids, + main, manifest_row, read_frozen_panel, - reconcile_with_eval_artifact, render_manifest_markdown, - resolve_selection, + retirement_message, + run_panel_freeze, + verify_frozen_panels, +) +from tests.fixtures.frozen_baseline import ( + build_frozen_tree, + make_panel, + patch_manifest, + rewrite_panel, ) @@ -240,114 +247,299 @@ def test_render_manifest_markdown_deterministic_and_full_precision(): assert repr(row["mean"]) in text -def test_determinism_subjects_cover_two_minute_and_one_book(): - assert "value_ep" in DETERMINISM_FACTORS # the book subject - minute = [f for f in DETERMINISM_FACTORS if f.endswith("_20")] - assert len(minute) >= 2 # two minute subjects (incl. the panel-consuming loader) # --------------------------------------------------------------------------- # -# eval-artifact reconciliation +# Retired regeneration entry point # --------------------------------------------------------------------------- # -def _write_eval_json(reports_dir: Path, stem: str, payload: dict) -> None: - document = {"sections": [{"name": "data_coverage", "payload": payload}]} - reports_dir.mkdir(parents=True, exist_ok=True) - (reports_dir / f"{stem}_no_book.json").write_text( - json.dumps(document), encoding="utf-8" - ) +def test_regeneration_raises_instead_of_quietly_doing_nothing(): + with pytest.raises(RegenerationRetiredError, match="RETIRED"): + run_panel_freeze() -def _matching_payload(processed: pd.Series, declared: list[str]) -> dict: - values = processed.to_numpy(dtype=float) - evaluated = set(processed.index.get_level_values(SYMBOL_LEVEL)) - return { - "panel_rows": int(len(processed)), - "evaluation_periods": int( - pd.unique(processed.index.get_level_values(DATE_LEVEL)).size - ), - "symbols_evaluated": len(evaluated), - "universe_symbols_declared": len(declared), - "dropped_symbols_count": len(set(declared) - evaluated), - "factor_nan_rate": round(1.0 - np.isfinite(values).sum() / len(values), 6), - } - +def test_the_retirement_message_names_the_ruling_the_tool_and_the_way_forward(): + with pytest.raises(RegenerationRetiredError) as caught: + run_panel_freeze("config/whatever.yaml") + text = str(caught.value) + assert "owner ruling 2026-07-28, D5 C6" in text + assert "python -m qt.panel_freeze --verify" in text + assert "no longer rebuilds" in text -def test_reconcile_passes_on_exact_match(tmp_path: Path): - processed = _panel([1.0, np.nan, 3.0, 4.0], KEYS) - declared = ["000001.SZ", "600000.SH", "999999.SZ"] - _write_eval_json(tmp_path, "eval_x", _matching_payload(processed, declared)) - out = reconcile_with_eval_artifact("eval_x", processed, declared, tmp_path) - assert out["artifact"] == "eval_x_no_book.json" - assert all(check["ok"] for check in out["checks"].values()) +def test_the_retirement_core_is_composed_not_restated(): + """The shared sentence is authored once and every tool composes it. -def test_reconcile_raises_on_mismatch(tmp_path: Path): - processed = _panel([1.0, np.nan, 3.0, 4.0], KEYS) - declared = ["000001.SZ", "600000.SH"] - payload = _matching_payload(processed, declared) - payload["panel_rows"] += 1 # the artifact claims one more row than we froze - _write_eval_json(tmp_path, "eval_x", payload) - with pytest.raises(ValueError, match="panel_rows"): - reconcile_with_eval_artifact("eval_x", processed, declared, tmp_path) + A regex cannot assert that no other sentence says the same thing; "there is + no other sentence" can (CLAUDE.md methodology #2). So: the core must come + out of ``retirement_message``, and the per-tool clauses must be the ONLY + difference between the three messages. + """ + composed = retirement_message("tool-x", "product-y", "WHY.", "VERIFIES.") + assert composed.startswith("tool-x: regeneration is RETIRED") + assert "WHY." in composed and "VERIFIES." in composed + source = Path(__file__).resolve().parents[1] / "qt" / "panel_freeze.py" + body = source.read_text(encoding="utf-8") + assert body.count("regeneration is RETIRED (") == 1 -def test_reconcile_raises_on_nan_rate_mismatch(tmp_path: Path): - processed = _panel([1.0, np.nan, 3.0, 4.0], KEYS) - declared = ["000001.SZ", "600000.SH"] - payload = _matching_payload(processed, declared) - payload["factor_nan_rate"] = round(payload["factor_nan_rate"] + 1e-6, 6) - _write_eval_json(tmp_path, "eval_x", payload) - with pytest.raises(ValueError, match="factor_nan_rate"): - reconcile_with_eval_artifact("eval_x", processed, declared, tmp_path) +def test_the_old_command_line_still_parses_so_it_gets_the_explanation(capsys): + """``--resume`` / ``--only`` were the documented regeneration flags. Dropping + them from the parser would answer the old command with ``unrecognized + arguments``, which reads like a broken install rather than a retirement.""" + assert main(["--resume", "--only", "value_ep"]) == 1 + assert "RETIRED" in capsys.readouterr().err -def test_reconcile_missing_artifact_is_loud(tmp_path: Path): - processed = _panel([1.0, 2.0, 3.0, 4.0], KEYS) - with pytest.raises(FileNotFoundError, match="no_book.json"): - reconcile_with_eval_artifact("eval_x", processed, ["000001.SZ"], tmp_path) - - -def test_reconcile_missing_payload_is_loud(tmp_path: Path): - (tmp_path / "eval_x_no_book.json").write_text( - json.dumps({"sections": [{"name": "other", "payload": {"a": 1}}]}), - encoding="utf-8", - ) - processed = _panel([1.0, 2.0, 3.0, 4.0], KEYS) - with pytest.raises(ValueError, match="data_coverage"): - reconcile_with_eval_artifact("eval_x", processed, ["000001.SZ"], tmp_path) +def test_a_bare_invocation_is_non_zero_and_explains_itself(capsys): + assert main([]) == 1 + captured = capsys.readouterr() + assert "RETIRED" in captured.err and "--verify" in captured.err # --------------------------------------------------------------------------- # -# --only selection (added for the PR-C intraday-cutoff correctness re-freeze) +# Verification: the green control, then one tampering per failure mode # --------------------------------------------------------------------------- # -def test_selection_default_is_none_so_the_full_freeze_is_untouched(): - assert resolve_selection(None) is None - - -def test_selection_keeps_the_given_order_and_accepts_book_and_minute_ids(): - assert resolve_selection(["jump_amount_corr_20", "value_ep"]) == ( - "jump_amount_corr_20", - "value_ep", +def test_an_untouched_frozen_tree_verifies_green(tmp_path: Path): + doc = build_frozen_tree(tmp_path) + result = verify_frozen_panels(tmp_path, doc) + assert result.ok and result.n_ok == 2 and not result.problems + + +def test_a_changed_cell_is_convicted_and_attributed_to_its_factor(tmp_path: Path): + doc = build_frozen_tree(tmp_path) + assert verify_frozen_panels(tmp_path, doc).ok # green control + + panel = make_panel("alpha_20") + panel.iloc[0] = float(panel.iloc[0]) + 1e-9 + rewrite_panel(tmp_path, "alpha_20", panel) + + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + bad = {panel.factor_id for panel in result.panels if not panel.ok} + assert bad == {"alpha_20"} + assert any("canonical content hash" in p for p in _problems(result, "alpha_20")) + + +def test_moving_the_machine_manifest_too_does_not_buy_a_pass(tmp_path: Path): + """The failure mode the git/gitignore split exists for: a tree regenerated + together with its own manifest. Both local witnesses now agree with each + other; the git-tracked document must still convict.""" + doc = build_frozen_tree(tmp_path) + panel = make_panel("alpha_20") + panel.iloc[0] = float(panel.iloc[0]) + 1e-9 + rewrite_panel(tmp_path, "alpha_20", panel) + new_hash = canonical_content_hash(panel) + new_file_hash = file_sha256(tmp_path / "panels" / "alpha_20.parquet") + + def _move(document): + for row in document["rows"]: + if row["factor_id"] == "alpha_20": + row.update( + canonical_sha256=new_hash, + file_sha256=new_file_hash, + mean=float(panel.mean()), + std=float(panel.std()), + ) + + patch_manifest(tmp_path, "manifest.json", _move) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("git-tracked" in p for p in _problems(result, "alpha_20")) + + +def test_a_missing_panel_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path) + (tmp_path / "panels" / "book_x.parquet").unlink() + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("missing from disk" in p for p in _problems(result, "book_x")) + + +def test_an_extra_panel_is_convicted_rather_than_ignored(tmp_path: Path): + """An unregistered panel is a problem, not a bonus: the frozen inventory is + defined by the document, so an extra file means someone wrote into the + frozen tree.""" + doc = build_frozen_tree(tmp_path) + atomic_write_parquet(make_panel("intruder_20"), tmp_path / "panels" / "intruder_20.parquet") + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("unregistered panel" in p for p in result.problems) + + +def test_a_missing_machine_manifest_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path) + (tmp_path / "manifest.json").unlink() + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("machine manifest missing" in p for p in result.problems) + + +def test_a_producing_sha_that_disagrees_with_the_document_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path, producing_sha="a" * 40) + patch_manifest( + tmp_path, "manifest.json", + lambda d: d["header"].update(producing_git_sha="b" * 40), ) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("producing SHA mismatch" in p for p in result.problems) -def test_selection_rejects_an_unknown_factor_id(): - """A typo must cost a readable error, not an empty output root reported as - success — "froze 0 panels" is how a correctness re-freeze silently produces - nothing at all.""" - with pytest.raises(ValueError, match="does not produce"): - resolve_selection(["jump_amount_corr"]) # missing the _20 window suffix +def test_a_reconciliation_recorded_as_not_ok_is_convicted(tmp_path: Path): + """A divergent panel must never have been frozen; a manifest edited to say + otherwise is exactly what this check is for.""" + doc = build_frozen_tree(tmp_path) + def _fail_one(document): + document["reconciliation"]["alpha_20"]["checks"]["panel_rows"]["ok"] = False -def test_selection_rejects_an_empty_selection(): - with pytest.raises(ValueError, match="empty selection"): - resolve_selection([]) + patch_manifest(tmp_path, "manifest.json", _fail_one) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("recorded as NOT ok" in p for p in result.problems) -def test_freezable_ids_are_the_14_the_freeze_writes(): - ids = freezable_factor_ids() - assert len(ids) == 14 - assert {"value_ep", "value_bp", "volatility_20"} <= ids - # Every determinism subject must be freezable, or --only could silently drop - # the double-run subject the manifest still claims to have checked. - assert set(DETERMINISM_FACTORS) <= ids +def test_a_dropped_reconciliation_check_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path) + patch_manifest( + tmp_path, "manifest.json", + lambda d: d["reconciliation"]["alpha_20"]["checks"].pop("factor_nan_rate"), + ) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("missing the 'factor_nan_rate' check" in p for p in result.problems) + + +def test_a_rewritten_but_content_identical_file_is_still_reported(tmp_path: Path): + """Same values, new bytes. A frozen-forever file has no legitimate reason to + be rewritten, so this is reported — and the message says which of the two it + is, so nobody reads it as a value change.""" + doc = build_frozen_tree(tmp_path) + target = tmp_path / "panels" / "alpha_20.parquet" + before = file_sha256(target) + frame = pd.read_parquet(target) + frame.to_parquet(target, engine="pyarrow", index=False, compression="gzip") + assert file_sha256(target) != before # the mutation landed + + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + problems = _problems(result, "alpha_20") + assert any("file sha256" in p and "CONTENT still matches" in p for p in problems) + assert not any("canonical content hash" in p for p in problems) + + +def test_verify_writes_nothing_into_the_tree_it_checks(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + before = { + path: (path.stat().st_mtime_ns, path.stat().st_size) + for path in sorted(tmp_path.rglob("*")) if path.is_file() + } + assert verify_frozen_panels(tmp_path, doc, show_manifest=True).ok + after = { + path: (path.stat().st_mtime_ns, path.stat().st_size) + for path in sorted(tmp_path.rglob("*")) if path.is_file() + } + assert after == before + + +def test_show_manifest_renders_the_table_from_the_panels_on_disk(tmp_path: Path): + doc = build_frozen_tree(tmp_path) + result = verify_frozen_panels(tmp_path, doc, show_manifest=True) + assert result.rendered_manifest is not None + assert "alpha_20" in result.rendered_manifest + assert verify_frozen_panels(tmp_path, doc).rendered_manifest is None + + +def _problems(result, factor_id: str) -> list[str]: + return [ + problem + for panel in result.panels + if panel.factor_id == factor_id + for problem in panel.problems + ] + + +def test_a_machine_manifest_row_that_disagrees_with_its_panel_is_convicted(tmp_path: Path): + """The document carries the hashes; the machine manifest carries the fuller + statistical row. This case is the one only the SECOND witness can see: the + document still agrees with the panel, and only the recorded row does not.""" + doc = build_frozen_tree(tmp_path) + assert verify_frozen_panels(tmp_path, doc).ok # green control + + def _lie(document): + for row in document["rows"]: + if row["factor_id"] == "alpha_20": + row["n_nan"] = row["n_nan"] + 7 + + patch_manifest(tmp_path, "manifest.json", _lie) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + problems = _problems(result, "alpha_20") + assert any("manifest.json n_nan" in p for p in problems) + # nothing else fired: the git-tracked side is untouched and still agrees + assert not any("git-tracked" in p for p in problems) + + +def test_a_deleted_reconciliation_entry_is_convicted(tmp_path: Path): + """The gap this closes: iterating only the entries PRESENT cannot see one + that was removed. The next PR is a deletion PR, so a verifier blind to + removals would be blind exactly when it matters. + + TWO minute factors on purpose. With one, deleting its entry empties the + block and the pre-existing "no reconciliation block" message fires — which + would let this test pass without the closed-set check existing at all. The + real case is 11 entries becoming 10, and that is what is built here. + """ + doc = build_frozen_tree( + tmp_path, factors={"alpha_20": "minute", "beta_20": "minute", "book_x": "book"} + ) + assert verify_frozen_panels(tmp_path, doc).ok # green control + + def _drop_one(document): + removed = document["reconciliation"].pop("alpha_20") + assert removed, "fixture no longer has the entry this test removes" + + patch_manifest(tmp_path, "manifest.json", _drop_one) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any( + "reconciliation has no entry for alpha_20" in p for p in result.problems + ), result.problems + # The block is NOT empty, so the pre-existing "no reconciliation block" + # message cannot be what convicted here. + assert not any("carries no reconciliation block" in p for p in result.problems) + + +def test_an_entry_for_a_non_minute_panel_is_convicted(tmp_path: Path): + """The other direction of the closed set: an invented entry.""" + doc = build_frozen_tree(tmp_path) + + def _invent(document): + document["reconciliation"]["book_x"] = { + "artifact": "eval_book_x_no_book.json", + "checks": {}, + } + + patch_manifest(tmp_path, "manifest.json", _invent) + result = verify_frozen_panels(tmp_path, doc) + assert not result.ok + assert any("carries an entry for book_x" in p for p in result.problems) + + +def test_the_real_manifests_reconcile_exactly_their_minute_panels(tmp_path: Path): + """Coupling check: the closed set is only meaningful if the real frozen trees + actually satisfy it. Both do — D1 has 11 minute panels and 11 entries, the + PR-C tree 1 and 1 — so this check is a live constraint, not a rule the real + data was already exempt from.""" + import json + + root = Path(__file__).resolve().parents[1] / "artifacts" / "refactor_baseline" + if not (root / "manifest.json").exists(): # gitignored bulk tree + pytest.skip("frozen baseline not present in this checkout") + for relative in ("manifest.json", "pr_c_cutoff_fix/manifest.json"): + document = json.loads((root / relative).read_text(encoding="utf-8")) + minute = { + row["factor_id"] for row in document["rows"] if row["kind"] == "minute" + } + assert minute == set(document["reconciliation"]), relative + assert minute, relative diff --git a/tests/test_panel_reconcile_d2.py b/tests/test_panel_reconcile_d2.py index 8b6d65f..428c49e 100644 --- a/tests/test_panel_reconcile_d2.py +++ b/tests/test_panel_reconcile_d2.py @@ -1,18 +1,39 @@ -"""Teeth tests for the D2 cell-by-cell panel comparator (network-free). - -A reconciliation whose comparator cannot fail is the ``compare_postmerge.py`` -failure mode; these tests feed the comparator engineered defects and assert it -CONVICTS each one — and that the one legitimate difference class (float -reordering within 1e-12) passes without being confused with hash equality. +"""Teeth tests for the D2 panel reconciliation (network-free, VERIFY-ONLY era). + +Two halves: + +* the cell-by-cell comparator — a reconciliation whose comparator cannot fail is + the ``compare_postmerge.py`` failure mode, so it is fed engineered defects and + must CONVICT each one, while the one legitimate difference class (float + reordering within 1e-12) passes without being confused with hash equality; +* the verify-only entry point, which re-derives the frozen D2 verdict from the + two frozen panel sets. Its tests always establish a GREEN tree first, so a red + result is attributable to the tampering rather than to a comparator that is + red on everything. """ from __future__ import annotations +from pathlib import Path + import numpy as np import pandas as pd import pytest -from qt.panel_reconcile import RELATIVE_TOLERANCE, compare_panels +from qt.panel_freeze import RegenerationRetiredError, canonical_content_hash +from qt.panel_reconcile import ( + RELATIVE_TOLERANCE, + compare_panels, + main, + run_panel_reconcile, + verify_d2_reconciliation, +) +from tests.fixtures.frozen_baseline import ( + build_frozen_tree, + make_panel, + patch_manifest, + rewrite_panel, +) def _series(values, dates=None, symbols=("AAA", "BBB")): @@ -78,3 +99,146 @@ def test_sign_flip_at_zero_magnitude_is_within_denominator_rule(): b2.iloc[0] = 1e-30 comp2 = compare_panels(a, b2, "f") assert not comp2.ok and comp2.max_rel_diff == pytest.approx(1.0) + + +# --------------------------------------------------------------------------- # +# Retired regeneration entry point +# --------------------------------------------------------------------------- # +def test_the_rebuild_raises_instead_of_quietly_doing_nothing(): + with pytest.raises(RegenerationRetiredError, match="RETIRED"): + run_panel_reconcile() + + +def test_the_retirement_message_says_the_comparison_survives(): + """The three retired tools do NOT verify the same kind of thing, so each + states its own. Claiming a check it does not perform is the defect this + repository keeps catching.""" + with pytest.raises(RegenerationRetiredError) as caught: + run_panel_reconcile() + text = str(caught.value) + assert "python -m qt.panel_reconcile --verify" in text + assert "COMPARISON never needed those loaders" in text + + +def test_the_old_resume_command_still_parses_so_it_gets_the_explanation(capsys): + assert main(["--resume"]) == 1 + assert "RETIRED" in capsys.readouterr().err + + +# --------------------------------------------------------------------------- # +# Verification: re-deriving the frozen D2 verdict +# --------------------------------------------------------------------------- # +def test_two_intact_frozen_sides_re_derive_the_passing_verdict(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + result = verify_d2_reconciliation(tmp_path, doc) + assert result.ok and result.d1_ok and result.recorded_all_ok is True + assert len(result.comparisons) == 2 + assert all(c.max_rel_diff == 0.0 and c.hashes_equal for c in result.comparisons) + + +def test_a_drifted_d2_panel_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + assert verify_d2_reconciliation(tmp_path, doc).ok # green control + + panel = make_panel("alpha_20") + panel.iloc[0] = float(panel.iloc[0]) + 1.0 + rewrite_panel(tmp_path, "alpha_20", panel, d2=True) + + result = verify_d2_reconciliation(tmp_path, doc) + assert not result.ok + convicted = [c for c in result.comparisons if not c.ok] + assert [c.factor_id for c in convicted] == ["alpha_20"] + assert any("D2 panel canonical hash" in p for p in result.problems) + + +def test_a_drifted_d1_side_is_convicted_before_the_cells_are_believed(tmp_path: Path): + """Comparing two panel sets while neither has been authenticated would report + agreement between two unknowns — so a D1 side that no longer matches git + fails the run even though the two sides still agree with EACH OTHER.""" + doc = build_frozen_tree(tmp_path, with_d2=True) + panel = make_panel("alpha_20") + panel.iloc[0] = float(panel.iloc[0]) + 1.0 + rewrite_panel(tmp_path, "alpha_20", panel) + rewrite_panel(tmp_path, "alpha_20", panel, d2=True) + + result = verify_d2_reconciliation(tmp_path, doc) + assert all(c.ok for c in result.comparisons) # the two sides DO agree + assert not result.d1_ok and not result.ok + assert any(p.startswith("D1 side") for p in result.problems) + + +def test_a_nan_set_change_in_the_d2_side_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + panel = make_panel("alpha_20") + panel.iloc[1] = np.nan + rewrite_panel(tmp_path, "alpha_20", panel, d2=True) + result = verify_d2_reconciliation(tmp_path, doc) + convicted = [c for c in result.comparisons if not c.ok] + assert convicted and convicted[0].nan_only_in_new == 1 + assert not result.ok + + +def test_a_missing_d2_manifest_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + (tmp_path / "manifest_d2.json").unlink() + result = verify_d2_reconciliation(tmp_path, doc) + assert not result.ok + assert any("D2 manifest missing" in p for p in result.problems) + + +def test_a_recorded_failure_is_never_treated_as_a_pass(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + patch_manifest(tmp_path, "manifest_d2.json", lambda d: d["header"].update(all_ok=False)) + result = verify_d2_reconciliation(tmp_path, doc) + assert not result.ok + assert any("all_ok=False" in p for p in result.problems) + + +def test_a_d2_manifest_row_that_disagrees_with_its_panel_is_convicted(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + + def _lie(document): + for row in document["rows"]: + if row["factor_id"] == "alpha_20": + row["canonical_sha256"] = "f" * 64 + + patch_manifest(tmp_path, "manifest_d2.json", _lie) + result = verify_d2_reconciliation(tmp_path, doc) + assert not result.ok + assert any("manifest_d2.json records canonical" in p for p in result.problems) + + +def test_an_absent_d2_directory_cannot_pass_by_having_nothing_to_compare(tmp_path: Path): + """Zero comparisons is not a passing reconciliation.""" + doc = build_frozen_tree(tmp_path, with_d2=False) + result = verify_d2_reconciliation(tmp_path, doc) + assert not result.ok and result.comparisons == () + + +def test_verify_writes_nothing_into_the_frozen_trees(tmp_path: Path): + doc = build_frozen_tree(tmp_path, with_d2=True) + before = { + path: (path.stat().st_mtime_ns, path.stat().st_size) + for path in sorted(tmp_path.rglob("*")) if path.is_file() + } + assert verify_d2_reconciliation(tmp_path, doc).ok + after = { + path: (path.stat().st_mtime_ns, path.stat().st_size) + for path in sorted(tmp_path.rglob("*")) if path.is_file() + } + assert after == before + + +def test_the_two_sides_are_read_from_disk_independently(tmp_path: Path): + """Both sides come from their own file read; no shared in-memory object can + make the equality vacuous. Shown by making the two files differ: if one read + were reused for both, this could not be detected.""" + doc = build_frozen_tree(tmp_path, with_d2=True) + panel = make_panel("book_x", offset=1.0) + panel.iloc[2] = float(panel.iloc[2]) + 7.0 + rewrite_panel(tmp_path, "book_x", panel, d2=True) + assert canonical_content_hash(panel) != canonical_content_hash(make_panel("book_x", offset=1.0)) + + result = verify_d2_reconciliation(tmp_path, doc) + convicted = [c for c in result.comparisons if not c.ok] + assert [c.factor_id for c in convicted] == ["book_x"] diff --git a/tests/test_property_test_migration_map.py b/tests/test_property_test_migration_map.py new file mode 100644 index 0000000..80ee310 --- /dev/null +++ b/tests/test_property_test_migration_map.py @@ -0,0 +1,146 @@ +"""R16: the property-test migration map must stay true to the suite. + +``docs/factors/d5_property_test_migration_map.md`` is the admission ticket for +deleting the eleven legacy eval runners and their tests: it claims that each of +the 59 legacy runner tests has a mapping into the new suite, so coverage does +not fall. A claim like that is worth exactly as much as its citations, and a +document cannot notice when a test it names is renamed away. + +So these tests read the map and check it against the tree: + +* every new-suite test the map cites exists, resolves to ONE file, and family + wildcards have the size the document claims; +* while the legacy files still exist, none of their tests is absent from the + map. + +Guard range, stated here because a guard that does not say what it cannot see +invites being trusted further than it should be: this checks that citations +RESOLVE, not that the cited test still asserts the property the map claims for +it. A test renamed away is caught; a test gutted in place is not. + +The second check goes vacuous the day the legacy files are deleted — by design. +Its job is to make an UNMAPPED legacy test impossible while they exist, not to +keep them alive; a guard that failed on their deletion would be a booby trap for +the very PR this map exists to authorise. +""" + +from __future__ import annotations + +import re +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] +MAP = REPO / "docs/factors/d5_property_test_migration_map.md" +TESTS = REPO / "tests" + +QUALIFIED = re.compile( + r"tests/test_[A-Za-z0-9_]+\.py::(test_[A-Za-z0-9_]+)(?![A-Za-z0-9_*])" +) +# ``(?![A-Za-z0-9_*])`` on both patterns: a bare ``(?!\*)`` is defeated by +# backtracking (the name simply gives back its trailing underscore and matches a +# shorter, still-wrong name), which reports three phantom missing tests. +BARE = re.compile(r"(? dict[str, set[str]]: + """{test function name: {files defining it}} across the whole suite.""" + names: dict[str, set[str]] = {} + for path in sorted(TESTS.rglob("test_*.py")): + for line in path.read_text(encoding="utf-8").splitlines(): + if line.startswith("def test"): + names.setdefault(line[4:].split("(")[0], set()).add(path.name) + return names + + +def _map_text() -> str: + return MAP.read_text(encoding="utf-8") + + +def _legacy_runner_tests() -> dict[str, list[str]]: + return { + path.name: [ + line[4:].split("(")[0] + for line in path.read_text(encoding="utf-8").splitlines() + if line.startswith("def test") + ] + for path in sorted(TESTS.glob("test_eval_*_runner.py")) + } + + +def test_the_map_document_exists_and_carries_citations(): + """Anti-vacuity: every check below is a loop over parsed citations, and an + empty parse would make all of them pass.""" + text = _map_text() + cited = set(QUALIFIED.findall(text)) | set(BARE.findall(text)) + assert len(cited) >= 40, f"only {len(cited)} citations parsed — parser or map broke" + + +@pytest.mark.parametrize("pattern", [QUALIFIED, BARE], ids=["qualified", "bare"]) +def test_every_cited_new_suite_test_exists_and_is_unambiguous(pattern: re.Pattern): + suite = _suite_test_names() + missing, ambiguous = [], [] + for name in sorted(set(pattern.findall(_map_text()))): + files = suite.get(name) + if not files: + missing.append(name) + elif len(files) > 1: + ambiguous.append(f"{name} -> {sorted(files)}") + assert not missing, f"cited by the migration map but not defined anywhere: {missing}" + # A duplicated test-function name silently drops one of the two from a full + # run in this repo (no __init__.py in tests/), so an ambiguous citation is a + # finding in its own right. + assert not ambiguous, f"cited name defined in more than one file: {ambiguous}" + + +def test_every_family_wildcard_has_the_size_the_document_claims(): + suite = _suite_test_names() + families = FAMILY.findall(_map_text()) + assert families, "no family wildcards parsed — the pattern or the map changed" + for stem, claimed in families: + prefix = f"{stem}_" + found = sorted(name for name in suite if name.startswith(prefix)) + assert len(found) == int(claimed), ( + f"{prefix}*: document claims {claimed}, suite has {len(found)}: {found}" + ) + + +def test_no_legacy_runner_test_is_missing_from_the_map(): + """Vacuous once the legacy files are deleted — see the module docstring.""" + text = _map_text() + legacy = _legacy_runner_tests() + unmapped = { + file_name: [name for name in names if f"`{name}`" not in text] + for file_name, names in legacy.items() + } + unmapped = {k: v for k, v in unmapped.items() if v} + assert not unmapped, f"legacy tests with no row in the migration map: {unmapped}" + + +def test_the_counting_table_is_internally_consistent(): + """7 migrated + 50 covered + 2 carrier-retired must equal the 59 claimed.""" + text = _map_text() + numbers = { + label: int(re.search(pattern, text).group(1)) + for label, pattern in ( + ("total", r"旧套件总条数（10 文件） \| \*\*(\d+)\*\*"), + ("migrated", r"已搬迁（逐字移入新文件） \| (\d+)"), + ("covered", r"已覆盖（性质在新套件有钉） \| (\d+)"), + ("retired", r"载体退役（[^|]*） \| (\d+)"), + ) + } + assert ( + numbers["migrated"] + numbers["covered"] + numbers["retired"] == numbers["total"] + ), numbers + assert "59 / 59（100%）" in text + assert "完全无映射、无理由退役 | **0**" in text + + +def test_the_claimed_total_matches_the_legacy_files_while_they_exist(): + legacy = _legacy_runner_tests() + if not legacy: # deleted: the claim is history, not a live count + pytest.skip("legacy runner test files are gone; the count is historical") + assert len(legacy) == 10 + assert sum(len(names) for names in legacy.values()) == 59