diff --git a/AGENTS.md b/AGENTS.md index afcc8f5..15f8d68 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -107,6 +107,15 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 - **真实结果(关键发现,~16min,77 成分/2 年,持有窗口切片)**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 -- ✅ 质量门:`pytest` **285 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16);`ruff` clean;`validate-config`(demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`)+ `run-phase0`(demo)均 OK。 -- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据、因子 IC 小样本不稳定(P3-3 实证)。 -- 路线图下一步:更长历史/更宽 universe 的稳定性复检,或分钟级(architecture.html §11)。 +- 🔧 **Phase 3-4 robustness matrix**(`p3-robustness-matrix` 分支,代劳待验收):把 P3-3 OOS 检验批量跑到 **universe × window 矩阵**上,回答 P3-2/P3-3 结论是否 SSE50 小样本偶然。不加因子/alpha、不改 portfolio/execution/factor math。 + - 新 run mode `run-phase3-robustness`(`qt/robustness.py`)+ `config/phase3_real_robustness_matrix.yaml`(SSE50+CSI300 × 2020-2022/2022-2024 两 fold;`skip_cells` 显式跳过 CSI300×2020-2022——runtime 预算,报告披露,绝不静默缩覆盖)。 + - **每个 cell 逐字复用 P3-3 cell 核心**(`_run_oos_cell` 重构抽出,单 run 行为不变;持有窗口切片/实现日 IC 切片/walk-forward/ic_weighted guard 全继承);cell config 仅替换 universe/窗口/split/output_name(重过全套 pydantic 验证,parquet 不互覆盖);跨 cell 汇总严格按 cell 归属(测试锁定不串数)。 + - **真实矩阵结果**(3 cells,总 2h:960s/934s/5301s,CSI300 399 名): + - **P3-3 不漂移 smoke ✓**:SSE50|2022-2024 cell 逐数复现 P3-3 报告(−11.92%/−5.27%/−8.31%,boundary 2023-06-30)。 + - 原始因子 train→test sign consistency:momentum 1/3 / roe 0/3 / np_yoy 0/3——**IC 翻号跨 cell 普遍,P3-3 结论非偶然**。 + - `combo_ic_weighted` 是唯一跨 cell 稳定序列(test IC 3/3 正 + 3/3 sign consistent)但量级 ≈0.004~0.008 极小。 + - **关键负面发现**:ic_weighted 组合绩效不稳健——train→test 翻车形态出现在 **3 cells 中的 2 个**:SSE50|2020-2022 train **+7.09%→test −7.85%**、CSI300|2022-2024 train **+11.07%→test −17.74%**(sharpe +0.59→−1.01;CSI300 换手 1.2~1.6 成本拖累更重)。**凡 ic_weighted 样本内显著为正,样本外即翻负——典型过拟合签名;P3-2 的"跑赢"不能外推;微弱正 IC ≠ 净值赢。非收益声明。** + - secret scan 报告 0 处;demo 0.96/0.84 不变。 +- ✅ 质量门:`pytest` **299 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16 / P3-4=14);`ruff` clean;`validate-config`(旧 6 配置 + `phase3_real_robustness_matrix.yaml`)+ `run-phase0`(demo)均 OK。 +- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据、三因子组 IC 跨期翻号且 ic_weighted 宽 universe 样本外崩(P3-3/P3-4 实证——当前因子组不具备可交易信号强度)。 +- 路线图下一步:换/扩因子集(在既有验证框架下迭代),或分钟级(architecture.html §11)。 diff --git a/CLAUDE.md b/CLAUDE.md index afcc8f5..15f8d68 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -107,6 +107,15 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 - **真实结果(关键发现,~16min,77 成分/2 年,持有窗口切片)**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 -- ✅ 质量门:`pytest` **285 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16);`ruff` clean;`validate-config`(demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`)+ `run-phase0`(demo)均 OK。 -- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据、因子 IC 小样本不稳定(P3-3 实证)。 -- 路线图下一步:更长历史/更宽 universe 的稳定性复检,或分钟级(architecture.html §11)。 +- 🔧 **Phase 3-4 robustness matrix**(`p3-robustness-matrix` 分支,代劳待验收):把 P3-3 OOS 检验批量跑到 **universe × window 矩阵**上,回答 P3-2/P3-3 结论是否 SSE50 小样本偶然。不加因子/alpha、不改 portfolio/execution/factor math。 + - 新 run mode `run-phase3-robustness`(`qt/robustness.py`)+ `config/phase3_real_robustness_matrix.yaml`(SSE50+CSI300 × 2020-2022/2022-2024 两 fold;`skip_cells` 显式跳过 CSI300×2020-2022——runtime 预算,报告披露,绝不静默缩覆盖)。 + - **每个 cell 逐字复用 P3-3 cell 核心**(`_run_oos_cell` 重构抽出,单 run 行为不变;持有窗口切片/实现日 IC 切片/walk-forward/ic_weighted guard 全继承);cell config 仅替换 universe/窗口/split/output_name(重过全套 pydantic 验证,parquet 不互覆盖);跨 cell 汇总严格按 cell 归属(测试锁定不串数)。 + - **真实矩阵结果**(3 cells,总 2h:960s/934s/5301s,CSI300 399 名): + - **P3-3 不漂移 smoke ✓**:SSE50|2022-2024 cell 逐数复现 P3-3 报告(−11.92%/−5.27%/−8.31%,boundary 2023-06-30)。 + - 原始因子 train→test sign consistency:momentum 1/3 / roe 0/3 / np_yoy 0/3——**IC 翻号跨 cell 普遍,P3-3 结论非偶然**。 + - `combo_ic_weighted` 是唯一跨 cell 稳定序列(test IC 3/3 正 + 3/3 sign consistent)但量级 ≈0.004~0.008 极小。 + - **关键负面发现**:ic_weighted 组合绩效不稳健——train→test 翻车形态出现在 **3 cells 中的 2 个**:SSE50|2020-2022 train **+7.09%→test −7.85%**、CSI300|2022-2024 train **+11.07%→test −17.74%**(sharpe +0.59→−1.01;CSI300 换手 1.2~1.6 成本拖累更重)。**凡 ic_weighted 样本内显著为正,样本外即翻负——典型过拟合签名;P3-2 的"跑赢"不能外推;微弱正 IC ≠ 净值赢。非收益声明。** + - secret scan 报告 0 处;demo 0.96/0.84 不变。 +- ✅ 质量门:`pytest` **299 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16 / P3-4=14);`ruff` clean;`validate-config`(旧 6 配置 + `phase3_real_robustness_matrix.yaml`)+ `run-phase0`(demo)均 OK。 +- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据、三因子组 IC 跨期翻号且 ic_weighted 宽 universe 样本外崩(P3-3/P3-4 实证——当前因子组不具备可交易信号强度)。 +- 路线图下一步:换/扩因子集(在既有验证框架下迭代),或分钟级(architecture.html §11)。 diff --git a/RUNBOOK.md b/RUNBOOK.md index f269429..383dd35 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -308,6 +308,46 @@ ic_weighted weight stability (per-rebalance weights with train/test labels, sign-flip counts on trained rows, fallback count + reasons); and the explicit caveat that this is a small-sample stability check, NOT a return claim. +## Phase 3-4 — robustness matrix (longer history + wider universe) + +Re-runs the P3-3 OOS stability check on every (universe × window) cell to test +whether the single-sample conclusions generalize. No new factor, no new alpha, +no portfolio / execution / factor-math change. Documented by +`config/phase3_real_robustness_matrix.yaml` (SSE50 + CSI300 × two 2-year +folds 2020-2022 / 2022-2024; the CSI300×2020-2022 cell is explicitly +`skip_cells`-listed for runtime budget and DISCLOSED in the report). + +```bash +# validate (no network) +... -m qt.cli validate-config --config config/phase3_real_robustness_matrix.yaml +# run the matrix (network + token; HEAVY — a CSI300 2y cell alone is ~60-90 min) +... -m qt.cli run-phase3-robustness --config config/phase3_real_robustness_matrix.yaml +``` + +Mechanics (locked by tests): + +- Every cell reuses the P3-3 cell core VERBATIM (`qt.oos_stability._run_oos_cell`): + holding-window performance slicing, realization-date IC slicing, walk-forward + weights, equal-weight fallback disclosure, and the shared guards (real source, + `oos` section, the ic_weighted fake-comparison guard). The single-run + `run-phase3-oos` behaviour is unchanged. +- Per-cell configs are derived from the base with ONLY the cell identity + swapped (universe.index_code, data window, split, and a per-cell + `output_name` so panels never overwrite each other); the derived config + re-runs full pydantic validation. +- `robustness.skip_cells` reduces coverage EXPLICITLY (validated against the + declared universes/windows; at least one cell must remain) and every skip is + disclosed in the report — coverage is never silently reduced. + +Report (`artifacts/reports/phase3_robustness_matrix.md`): the cell table +(window / split / per-cell runtime) + skipped cells; a CROSS-CELL stability +summary (per series: #cells with positive test IC, #cells train→test +sign-consistent, test IC by cell; #cells where ic_weighted beats equal_weight +on test annual); full per-cell diagnostics (subperiod performance, IC +stability, weight sign flips, fallback counts, boundary rebalances); and the +explicit caveat that this is a stability check — NOT a return claim, not a +tuned result. + ## Quality gate ```bash diff --git a/TEST_REPORT.md b/TEST_REPORT.md index 4e1874a..ec8e468 100644 --- a/TEST_REPORT.md +++ b/TEST_REPORT.md @@ -1,4 +1,4 @@ -# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha → OOS stability) +# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha → OOS stability → robustness matrix) ## Commands @@ -13,6 +13,7 @@ Run from the repo root with the project python (env `quant_mf`): /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_multifactor.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_ic_weighted.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_oos_stability.yaml +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_robustness_matrix.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli run-phase0 --config config/example.yaml ``` @@ -20,12 +21,12 @@ Run from the repo root with the project python (env `quant_mf`): | Gate | Command | Result | |---|---|---| -| Unit + integration | `pytest -q` | **285 passed, 0 failed** | +| Unit + integration | `pytest -q` | **299 passed, 0 failed** | | Lint | `ruff check .` | **All checks passed** | -| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`) | exit `0`, prints `OK` | +| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml` + `phase3_real_robustness_matrix.yaml`) | exit `0`, prints `OK` | | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | -Counts below are the actual per-file `pytest` numbers (sum = 285). +Counts below are the actual per-file `pytest` numbers (sum = 299). ## Per-file breakdown — Phase 0 core (97) @@ -116,7 +117,13 @@ Counts below are the actual per-file `pytest` numbers (sum = 285). | Test file | Tests | Red-line / feature | |---|---|---| | `test_oos_stability.py` | 16 | **split-boundary no-leakage**: perturbing every post-split forward return leaves all train-period weights bit-identical; **holding-window slicing** (`split_nav_by_holding`: train = holding end ≤ split, test = start ≥ split, straddlers + unknown-end rows excluded and disclosed); `subperiod_perf` rebased nav + empty-slice NaN; `ic_period_stats` sliced by realization date (t+h) with mean/IR/hit-rate/n; `sign_consistent` nonzero-same-sign; `weight_sign_flips` on trained rows only (fallback rows excluded); fallback-reason aggregation; OOS config validates + split-inside-window ConfigError; runner rejects demo source / missing `oos` section / **non-ic_weighted alpha (fake-comparison guard)**; report renders boundaries / straddler disclosure / OOS metrics / weight stability / caveat / no secret | -| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3)** | **285** | | + +## Per-file breakdown — Phase 3-4 robustness matrix (14) + +| Test file | Tests | Red-line / feature | +|---|---|---| +| `test_robustness_matrix.py` | 14 | matrix config validation (universes/windows/unique labels, split-inside-window, unknown-skip + all-skipped ConfigErrors); cell enumeration excludes skips in config order; per-cell config derivation swaps ONLY the cell identity (incl. unique per-cell output_name; derived cell passes the shared OOS preconditions); cross-cell aggregation attributes strictly per cell (no mixing, no dilution); runner guards (demo source / missing robustness section / non-ic_weighted alpha); report renders cell labels + skipped-cell disclosure + boundary/fallback/sign-consistency + cross-cell summary + caveat + no secret | +| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3 + P3-4)** | **299** | | ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) @@ -200,6 +207,11 @@ Counts below are the actual per-file `pytest` numbers (sum = 285). proves post-split forward returns cannot move any train-period weight. The runner refuses a non-ic_weighted `alpha.model` (a fake comparison guard). Portfolio / execution / factor math are untouched. +- **P3-4 robustness matrix (locked by tests):** `run-phase3-robustness` runs the + UNCHANGED P3-3 cell core on every (universe × window) cell; per-cell configs + derive from the base with only the cell identity swapped; skipped cells are + config-validated and disclosed (never silent); cross-cell aggregation is + strictly per cell. Single-run `run-phase3-oos` behaviour is unchanged. - A duplicate test-function name across two files was found and renamed during P2-2 (it had been silently shadowing one test in the full-suite run). A second, harmless duplicate (`test_enrich_does_not_mutate_input` in two files) was verified NOT to drop diff --git a/config/phase3_real_robustness_matrix.yaml b/config/phase3_real_robustness_matrix.yaml new file mode 100644 index 0000000..1b5fcf9 --- /dev/null +++ b/config/phase3_real_robustness_matrix.yaml @@ -0,0 +1,138 @@ +# Phase 3-4 — REAL robustness matrix (longer history + wider universe). +# +# Validation layer over the P3-3 OOS framework: the SAME three factors and the +# SAME equal_weight vs ic_weighted comparison are re-checked on every +# universe x window cell below, to answer whether the P3-2/P3-3 conclusions +# were an SSE50 small-sample accident. No new factor, no new alpha, no change +# to portfolio / execution / factor math. Every cell reuses the P3-3 cell core +# verbatim (holding-window performance slicing, realization-date IC slicing, +# walk-forward weights, ic_weighted guard). +# +# The data/universe sections below are the BASE; the matrix runner derives one +# config per cell (universe.index_code + data.start/end + oos.split_date + +# output_name swapped in; everything else identical). The top-level data window +# only validates the base config -- per-cell windows come from robustness.windows. +# +# Coverage note (runtime disclosure): CSI300 cells pull ~300+ historical names +# through the rate-limited tushare SDK (~60-90 min per 2y cell), so this config +# limits CSI300 to the recent fold; add {label: csi300-2020-2022-like} windows +# for a fuller matrix when the runtime budget allows. The report discloses the +# exact cells run + per-cell runtime. +# +# Run: python -m qt.cli run-phase3-robustness --config config/phase3_real_robustness_matrix.yaml +# (report: artifacts/reports/phase3_robustness_matrix.md; needs the tushare +# token in the external .config.json; hits the network; heavy ~1.5-2 h total.) + +project: + name: quantitative_trading_phase3_robustness_matrix + timezone: Asia/Shanghai + +data: + source: tushare + freq: D + start: "2022-07-01" + end: "2024-06-30" + external_secret_file: "/home/shaofl/Projects/financial_projects/.config.json" + tushare_token_key: "tushare.token" + output_name: phase3rm_daily + +universe: + type: index + index_code: "000016.SH" + symbols: [] + min_listing_days: 60 + filters: + missing_close: true + suspended: true + st: true + limit_up_down: true + +factors: + - name: momentum_20 + enabled: true + params: + window: 20 + price_col: close + - name: roe + enabled: true + params: {} + - name: netprofit_yoy + enabled: true + params: {} + +processing: + drop_missing: true + standardize: + enabled: true + method: zscore + winsorize: + enabled: false + method: mad + n: 3.0 + neutralize: + enabled: true + industry_col: industry + size_col: market_cap + industry_level: L1 # PIT SW level (L1/L2/L3); L1 = 31 broad sectors (standard, DOF-safe) + +alpha: + model: ic_weighted # P3-2 walk-forward rolling-IC weights + params: + window: 60 # trailing trading days of REALIZED ICs (conservative rolling) + min_periods: 20 # < this many valid realized ICs -> equal-weight fallback (disclosed) + mode: rolling # rolling (default) | expanding + +portfolio: + constructor: topn_equal_weight + top_n: 20 + long_only: true + max_weight: null + turnover_cap: null + +backtest: + initial_nav: 1.0 + rebalance: monthly + event_order: close_to_next_period + cash_return: 0.0 + +cost: + fee_rate: 0.001 + slippage_rate: 0.0 + turnover_formula: l1 + +analytics: + forward_return_periods: + - 1 + - 5 + - 20 + quantiles: 5 + benchmark: null + +output: + root_dir: artifacts + data_dir: artifacts/data + factor_dir: artifacts/factors + report_dir: artifacts/reports + log_dir: artifacts/logs + overwrite: true + +oos: + split_date: "2023-07-01" # base-config validation only; per-cell splits below + +robustness: + universes: + - "000016.SH" # SSE50 (the P3-3 sample) + - "000300.SH" # CSI300 (wider universe) + windows: + - label: "2020-2022" + start: "2020-07-01" + end: "2022-06-30" + split: "2021-07-01" + - label: "2022-2024" + start: "2022-07-01" + end: "2024-06-30" + split: "2023-07-01" + # cells = universes x windows MINUS the skips below (runtime budget; disclosed). + skip_cells: + - universe: "000300.SH" + window: "2020-2022" diff --git a/qt/cli.py b/qt/cli.py index 18c9d79..8c3b94c 100644 --- a/qt/cli.py +++ b/qt/cli.py @@ -111,6 +111,25 @@ def _cmd_run_phase3_oos(args: argparse.Namespace) -> int: return 0 +def _cmd_run_phase3_robustness(args: argparse.Namespace) -> int: + """Run the robustness matrix (universes x windows OOS cells) + report.""" + from qt.robustness import run_phase3_robustness + + try: + result = run_phase3_robustness(args.config) + except (ConfigError, ValueError, FileNotFoundError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + print( + f"OK run-phase3-robustness: cells={result.summary['n_cells']}, " + f"skipped={len(result.skipped_cells)}, " + f"ic_beats_eq_test={result.summary['ic_beats_eq_test']}/" + f"{result.summary['n_cells']} ({result.elapsed_seconds:.1f}s)\n" + f"report: {result.report_path}" + ) + return 0 + + def _cmd_fetch_data(args: argparse.Namespace) -> int: """Stage helper: run the spine and report the data-fetch stage.""" return _run_pipeline_cmd(args.config, "fetch-data") @@ -156,6 +175,13 @@ def build_parser() -> argparse.ArgumentParser: p_oos.add_argument("--config", required=True, help="Path to the YAML config.") p_oos.set_defaults(func=_cmd_run_phase3_oos) + p_rob = sub.add_parser( + "run-phase3-robustness", + help="Run the REAL-data robustness matrix (universes x windows OOS cells).", + ) + p_rob.add_argument("--config", required=True, help="Path to the YAML config.") + p_rob.set_defaults(func=_cmd_run_phase3_robustness) + for name, func, help_text in ( ("fetch-data", _cmd_fetch_data, "Run the spine, report data fetch."), ("compute-factors", _cmd_compute_factors, "Run the spine, report factor compute."), diff --git a/qt/config.py b/qt/config.py index 60e97c5..fc97c7d 100644 --- a/qt/config.py +++ b/qt/config.py @@ -200,6 +200,92 @@ def _coerce_date_to_str(cls, v: Any) -> Any: return v +class RobustnessWindowCfg(_Strict): + """One time fold of the P3-4 robustness matrix: [start, end] split at split.""" + + label: str + start: str + end: str + split: str + + @field_validator("start", "end", "split", mode="before") + @classmethod + def _coerce_date_to_str(cls, v: Any) -> Any: + if isinstance(v, (_date, datetime)): + return v.strftime("%Y-%m-%d") + return v + + @model_validator(mode="after") + def _check_window(self) -> "RobustnessWindowCfg": + try: + start = datetime.strptime(self.start, "%Y-%m-%d") + end = datetime.strptime(self.end, "%Y-%m-%d") + split = datetime.strptime(self.split, "%Y-%m-%d") + except ValueError as exc: + raise ValueError( + f"robustness window {self.label!r}: start/end/split must be " + f"'YYYY-MM-DD' dates ({exc})." + ) from exc + if not (start < split < end): + raise ValueError( + f"robustness window {self.label!r}: split ({self.split}) must lie " + f"STRICTLY inside [{self.start}, {self.end}] so both subperiods " + "are non-empty." + ) + return self + + +class RobustnessSkipCfg(_Strict): + """One EXPLICITLY skipped matrix cell (runtime budget; disclosed in report).""" + + universe: str + window: str # a windows[].label + + +class RobustnessCfg(_Strict): + """P3-4 robustness matrix: every universe × window pair is one OOS cell. + + ``skip_cells`` removes named cells from the run (e.g. a wide-universe long + fold whose rate-limited pull would blow the runtime budget); skipped cells + are DISCLOSED in the report — coverage is never silently reduced. + """ + + universes: list[str] + windows: list[RobustnessWindowCfg] + skip_cells: list[RobustnessSkipCfg] = Field(default_factory=list) + + @model_validator(mode="after") + def _check_non_empty(self) -> "RobustnessCfg": + if not self.universes: + raise ValueError("robustness.universes must list at least one index code.") + if not self.windows: + raise ValueError("robustness.windows must list at least one window.") + labels = [w.label for w in self.windows] + dupes = sorted({x for x in labels if labels.count(x) > 1}) + if dupes: + raise ValueError( + f"robustness.windows labels must be unique; duplicate(s): {dupes}." + ) + for skip in self.skip_cells: + if skip.universe not in self.universes: + raise ValueError( + f"robustness.skip_cells references unknown universe " + f"{skip.universe!r} (declared: {self.universes})." + ) + if skip.window not in labels: + raise ValueError( + f"robustness.skip_cells references unknown window label " + f"{skip.window!r} (declared: {labels})." + ) + run_cells = len(self.universes) * len(self.windows) - len(self.skip_cells) + if run_cells < 1: + raise ValueError( + "robustness.skip_cells removes every cell; at least one cell " + "must remain to run." + ) + return self + + class RootConfig(_Strict): """Top-level config composing every section. @@ -221,6 +307,8 @@ class RootConfig(_Strict): analytics: AnalyticsCfg = Field(default_factory=AnalyticsCfg) output: OutputCfg oos: OOSCfg | None = None + # P3-4 robustness matrix (consumed only by run-phase3-robustness). + robustness: RobustnessCfg | None = None @model_validator(mode="after") def _check_oos_split_inside_window(self) -> "RootConfig": diff --git a/qt/oos_stability.py b/qt/oos_stability.py index 70a7489..c30a294 100644 --- a/qt/oos_stability.py +++ b/qt/oos_stability.py @@ -261,10 +261,11 @@ def _oos_downgrades(cfg: RootConfig) -> tuple[str, ...]: "on/before the split, test rows start on/after it; a straddling rebalance " "is excluded from both and disclosed) and IC stats by the realization " "date — never by the row's signal date alone.", - "This is a SMALL-SAMPLE stability check (one index, two years, ~22 " - "rebalances), NOT a return claim and NOT a tuned result: subperiod " - "metrics carry wide uncertainty and must not be read as expected " - "performance.", + "This is a SMALL-SAMPLE stability check — each run/cell covers ONE index " + "universe over ONE window (~22 rebalances per train+test pair) — NOT a " + "return claim and NOT a tuned result: subperiod metrics carry wide " + "uncertainty and must not be read as expected performance. (A robustness " + "matrix spans multiple universes × windows; see its MATRIX SCOPE line.)", ) return _collect_downgrades(cfg) + extra @@ -288,38 +289,69 @@ def _run_backtest_for(cfg: RootConfig, panel, universe, score_panel) -> pd.DataF return driver.run() -def run_phase3_oos(config_path: str) -> OOSResult: - """Run the OOS stability validation and write the phase3 OOS report. +def check_oos_preconditions(cfg: RootConfig, runner: str = "run-phase3-oos") -> None: + """Shared guards for the OOS cell computation (single run AND matrix, P3-4). - Requires ``data.source='tushare'`` (a stability check on offline demo data - would be a category error) and an ``oos`` config section (the split date). - All writes land under the configured output dirs; no secret is echoed. + Real tushare source only; an ``oos`` section (the split date); and the + alpha section MUST be ic_weighted — anything else would silently run + equal_weight twice and label one leg ic_weighted (a fake comparison). """ - cfg = load_config(config_path) if cfg.data.source != "tushare": raise ValueError( - "run-phase3-oos is a REAL-data validation and requires " + f"{runner} is a REAL-data validation and requires " "data.source='tushare'; the demo path carries no PIT/ann_date meaning. " f"Got data.source={cfg.data.source!r}." ) if cfg.oos is None: raise ValueError( - "run-phase3-oos requires an 'oos' config section with split_date " + f"{runner} requires an 'oos' config section with split_date " "(train = [data.start, split), test = [split, data.end])." ) if cfg.alpha.model != "ic_weighted": raise ValueError( - "run-phase3-oos compares equal_weight vs ic_weighted: the config's " + f"{runner} compares equal_weight vs ic_weighted: the config's " "alpha section must set model='ic_weighted' (it carries the " "ic-weighted leg's params; the equal-weight control is built " f"internally). Got alpha.model={cfg.alpha.model!r} — running that " "would silently label an equal_weight leg as ic_weighted." ) - split = pd.Timestamp(cfg.oos.split_date) - t0 = time.perf_counter() + +def run_phase3_oos(config_path: str) -> OOSResult: + """Run the OOS stability validation and write the phase3 OOS report. + + Requires ``data.source='tushare'`` (a stability check on offline demo data + would be a category error) and an ``oos`` config section (the split date). + All writes land under the configured output dirs; no secret is echoed. + """ + cfg = load_config(config_path) + check_oos_preconditions(cfg, runner="run-phase3-oos") log_path = Path(cfg.output.log_dir) / "run_phase3_oos.log" logger = _make_logger(log_path, name=_LOGGER_NAME) + result = _run_oos_cell(cfg, logger, log_path) + write_oos_stability_summary(result) + logger.info( + "phase3 oos done: eq test annual=%.4f ic test annual=%.4f report=%s (%.1fs)", + result.performance["equal_weight"]["test"].get("annual_return", float("nan")), + result.performance["ic_weighted"]["test"].get("annual_return", float("nan")), + result.report_path, result.elapsed_seconds, + ) + return result + + +def _run_oos_cell( + cfg: RootConfig, logger, log_path: Path +) -> OOSResult: + """Compute ONE OOS cell (the P3-3 core; no report write). + + Shared verbatim by the single run (:func:`run_phase3_oos`) and the P3-4 + robustness matrix, so every matrix cell gets the same holding-window + slicing, realization-date IC slicing and walk-forward semantics. The + caller is responsible for guards (:func:`check_oos_preconditions`) and for + writing whatever report it owns. + """ + split = pd.Timestamp(cfg.oos.split_date) + t0 = time.perf_counter() logger.info("phase3 oos start: project=%s split=%s", cfg.project.name, split.date()) # --- ONE shared data load / factor panel (identical inputs for both) ----- # @@ -418,11 +450,10 @@ def run_phase3_oos(config_path: str) -> OOSResult: report_path=Path(cfg.output.report_dir) / "phase3_oos_stability.md", log_path=log_path, ) - write_oos_stability_summary(result) logger.info( - "phase3 oos done: eq test annual=%.4f ic test annual=%.4f report=%s (%.1fs)", + "oos cell done: eq test annual=%.4f ic test annual=%.4f (%.1fs)", performance["equal_weight"]["test"].get("annual_return", float("nan")), performance["ic_weighted"]["test"].get("annual_return", float("nan")), - result.report_path, result.elapsed_seconds, + result.elapsed_seconds, ) return result diff --git a/qt/reports.py b/qt/reports.py index 36c9b4e..95f581a 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -790,6 +790,137 @@ def write_oos_stability_summary(result) -> Path: return target +# --------------------------------------------------------------------------- # +# Phase 3-4 robustness matrix report (qt.robustness). +# --------------------------------------------------------------------------- # +def _matrix_summary_block(summary: dict) -> str: + """Cross-cell stability matrix: which findings hold across cells.""" + n = int(summary.get("n_cells", 0)) + lines = [ + f"- cells aggregated: **{n}**\n", + f"- ic_weighted beats equal_weight on TEST annual return in " + f"**{summary.get('ic_beats_eq_test', 0)}/{n}** cells\n\n", + "| Series | test IC > 0 | train→test sign consistent | test IC by cell |\n" + "|---|---|---|---|\n", + ] + for name, s in (summary.get("series") or {}).items(): + by_cell = ", ".join( + f"{label}: {_fmt(v)}" for label, v in (s.get("test_ic_by_cell") or {}).items() + ) + lines.append( + f"| `{name}` | {s.get('test_ic_positive', 0)}/{s.get('n_cells', 0)} | " + f"{s.get('sign_consistent', 0)}/{s.get('n_cells', 0)} | {by_cell} |\n" + ) + return "".join(lines) + + +def render_robustness_matrix(result) -> str: + """Build the phase3 robustness-matrix markdown (pure; no I/O, no secrets).""" + cfg = result.config + lines: list[str] = [] + lines.append("# Phase 3-4 — Robustness Matrix " + "(universes × windows, equal_weight vs ic_weighted)\n") + lines.append( + f"Project: **{cfg.project.name}** · source: **{cfg.data.source}** · " + f"ran in **{result.elapsed_seconds:.1f}s**\n" + ) + lines.append( + "\n> **This is NOT a return claim and not a tuned result.** It re-runs the " + "P3-3 OOS stability check (same three factors, same rules, walk-forward " + "weights, holding-window subperiod slicing) on every universe × window " + "cell, to test whether the single-sample conclusions generalize. " + "Small-sample caveats apply to every cell.\n" + ) + + lines.append("\n## Cells\n") + lines.append( + "| Cell (universe \\| window) | window | split | runtime |\n|---|---|---|---|\n" + ) + for label, cell in result.cells.items(): + runtime = result.cell_runtimes.get(label, float("nan")) + lines.append( + f"| `{label}` | {_date_str(cell.train_start)} → " + f"{_date_str(cell.test_end)} | {_date_str(cell.split_date)} | " + f"{runtime:.0f}s |\n" + ) + if result.skipped_cells: + sk = ", ".join(f"`{s}`" for s in result.skipped_cells) + lines.append( + f"\n- **skipped cells (disclosed, runtime budget — coverage is " + f"reduced, not hidden):** {sk}\n" + ) + + lines.append("\n## Cross-cell stability summary\n") + lines.append(_matrix_summary_block(result.summary)) + + for label, cell in result.cells.items(): + lines.append(f"\n## Cell `{label}`\n") + lines.append( + f"- train: {_date_str(cell.train_start)} → {_date_str(cell.train_end)} " + f"({cell.n_train_days}d) · test: {_date_str(cell.test_start)} → " + f"{_date_str(cell.test_end)} ({cell.n_test_days}d) · split " + f"{_date_str(cell.split_date)}\n" + ) + if cell.boundary_dates: + bd = ", ".join(_date_str(d) for d in cell.boundary_dates) + lines.append( + f"- boundary rebalance(s) excluded from both subperiods " + f"(holding window straddles the split): {bd}\n" + ) + lines.append("\n### Subperiod performance\n") + lines.append(_oos_perf_table(cell.performance)) + lines.append("\n### IC stability\n") + lines.append(_oos_ic_table(cell.ic_stats, cell.sign_consistency)) + flips = ", ".join(f"`{k}` {v}" for k, v in (cell.sign_flips or {}).items()) + lines.append( + f"\n### Weight stability (ic_weighted)\n" + f"- scored dates: **{cell.n_scored}** · equal-weight fallbacks: " + f"**{cell.n_fallback}**\n" + f"- sign flips between consecutive trained rebalance weights: " + f"{flips or '_n/a_'}\n" + ) + + lines.append("\n## DOWNGRADES / caveats (INV-007 — must be disclosed)\n") + # matrix-level scope FIRST: the disclosures below must never read as a + # single-universe run (the matrix spans several universes × windows). + run_labels = ", ".join(f"`{label}`" for label in result.cells) + universes = sorted({label.split("|", 1)[0] for label in result.cells}) + windows = sorted({label.split("|", 1)[1] for label in result.cells}) + skipped = ", ".join(f"`{s}`" for s in result.skipped_cells) or "none" + lines.append( + f"- MATRIX SCOPE: run cells: {run_labels}; skipped cells: {skipped}; " + f"universes covered: {universes}; windows covered: {windows}. " + "Universe-specific disclosures below are the UNION over all run cells " + "(each universe's PIT-membership line appears once — never only the " + "first cell's; see the Cells section for per-cell identity).\n" + ) + seen: set[str] = set() + for cell in result.cells.values(): + for item in cell.downgrades: + if item not in seen: + seen.add(item) + lines.append(f"- {item}\n") + lines.append( + "- MATRIX CAVEAT: per-cell metrics remain SMALL-SAMPLE (each cell = one " + "index over one window, ~22 rebalances); the matrix spans multiple " + "universes × windows to test REPEATABILITY — the cross-cell summary " + "shows which findings repeat, not that any of them is tradable. NOT a " + "return claim.\n" + ) + + lines.append("\n## Artifacts\n") + lines.append(f"- report: `{result.report_path}`\n- log: `{result.log_path}`\n") + return "".join(lines) + + +def write_robustness_matrix_summary(result) -> Path: + """Render and write the robustness-matrix report; return the path (SEC-003).""" + target = result.report_path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(render_robustness_matrix(result), encoding="utf-8") + return target + + # --------------------------------------------------------------------------- # # Repo-root delivery docs (framework spec §10). These describe the run; they # carry no secrets and are regenerable. diff --git a/qt/robustness.py b/qt/robustness.py new file mode 100644 index 0000000..aa48d82 --- /dev/null +++ b/qt/robustness.py @@ -0,0 +1,243 @@ +"""Phase 3-4: robustness matrix — the P3-3 OOS check over universes × windows. + +A REPORT-ONLY validation layer on top of :mod:`qt.oos_stability`: the SAME +three factors and the SAME equal_weight-vs-ic_weighted comparison are re-run +on every (universe, window) cell declared in the ``robustness`` config section, +to answer whether the P3-2/P3-3 conclusions were an SSE50 small-sample +accident. No new factor, no new alpha, no change to portfolio / execution / +factor math. + +Every cell reuses the P3-3 cell core VERBATIM (:func:`qt.oos_stability._run_oos_cell`): +holding-window performance slicing, realization-date IC slicing, walk-forward +weights, and the shared guards (:func:`qt.oos_stability.check_oos_preconditions`, +incl. the ic_weighted fake-comparison guard). Per-cell configs are derived from +the base config with ONLY the cell identity swapped in (universe.index_code, +data.start/end, oos.split_date, data.output_name — the last so cells never +overwrite each other's panel parquet). + +``robustness.skip_cells`` may drop named cells (runtime budget — a wide +universe × long fold can take hours through the rate-limited SDK); skipped +cells are DISCLOSED in the report, never silently absent. The report carries +per-cell diagnostics plus a cross-cell summary matrix (which findings hold +across cells and which do not) and the explicit not-a-return-claim caveat. +""" + +from __future__ import annotations + +import math +import time +from collections.abc import Iterator +from dataclasses import dataclass +from pathlib import Path + + +from qt.config import RobustnessWindowCfg, RootConfig, load_config +from qt.oos_stability import OOSResult, _run_oos_cell, check_oos_preconditions +from qt.pipeline import _make_logger +from qt.reports import render_robustness_matrix, write_robustness_matrix_summary + +_LOGGER_NAME = "qt.run_phase3_robustness" + +__all__ = ["RobustnessResult", "run_phase3_robustness", "render_robustness_matrix", + "cell_label", "iter_cells", "derive_cell_config", "summarize_matrix"] + + +@dataclass(frozen=True) +class RobustnessResult: + """Immutable summary of one robustness-matrix run (what the report consumes).""" + + config: RootConfig + elapsed_seconds: float + # cell label ("|") -> the cell's OOSResult + cells: dict[str, OOSResult] + cell_runtimes: dict[str, float] + skipped_cells: tuple[str, ...] + # cross-cell aggregation (see summarize_matrix) + summary: dict + report_path: Path + log_path: Path + + +# --------------------------------------------------------------------------- # +# Cell enumeration / derivation (pure; unit-tested) +# --------------------------------------------------------------------------- # +def cell_label(universe: str, window: RobustnessWindowCfg) -> str: + """Canonical cell id: '|'.""" + return f"{universe}|{window.label}" + + +def iter_cells(cfg: RootConfig) -> Iterator[tuple[str, RobustnessWindowCfg]]: + """Yield the (universe, window) cells to RUN, config order, skips removed.""" + rob = cfg.robustness + if rob is None: + return + skipped = {(s.universe, s.window) for s in rob.skip_cells} + for universe in rob.universes: + for window in rob.windows: + if (universe, window.label) in skipped: + continue + yield universe, window + + +def skipped_cell_labels(cfg: RootConfig) -> tuple[str, ...]: + """Labels of the explicitly skipped cells (disclosed in the report).""" + rob = cfg.robustness + if rob is None: + return () + by_label = {w.label: w for w in rob.windows} + return tuple( + cell_label(s.universe, by_label[s.window]) for s in rob.skip_cells + ) + + +def derive_cell_config( + cfg: RootConfig, universe: str, window: RobustnessWindowCfg +) -> RootConfig: + """One cell's config: the base verbatim with ONLY the cell identity swapped. + + Rebuilt through ``RootConfig(**dict)`` so every pydantic validation + (date order, split-inside-window, ...) re-runs on the derived values. + ``data.output_name`` gets a per-cell suffix so cells never overwrite each + other's panel parquet. + """ + raw = cfg.model_dump() + raw["universe"]["index_code"] = universe + raw["data"]["start"] = window.start + raw["data"]["end"] = window.end + raw["oos"] = {"split_date": window.split} + safe_universe = universe.replace(".", "_").lower() + raw["data"]["output_name"] = ( + f"{cfg.data.output_name}_{safe_universe}_{window.label.replace('-', '_')}" + ) + return RootConfig(**raw) + + +# --------------------------------------------------------------------------- # +# Cross-cell aggregation (pure; unit-tested) +# --------------------------------------------------------------------------- # +def summarize_matrix(cells: dict[str, dict]) -> dict: + """Aggregate per-cell findings into the cross-cell stability summary. + + ``cells`` maps cell label -> a dict (or OOSResult-like mapping) carrying + ``performance`` / ``ic_stats`` / ``sign_consistency`` / ``sign_flips`` / + ``n_scored`` / ``n_fallback``. Counts are attributed strictly per cell + (never pooled across cells — pooling would let a big cell mask a small + one). Returns:: + + {"n_cells", "ic_beats_eq_test": <#cells where ic test annual > eq>, + "series": {name: {"n_cells", "test_ic_positive", "sign_consistent", + "test_ic_by_cell": {label: mean}}}} + """ + series_names: list[str] = [] + for cell in cells.values(): + for name in cell["ic_stats"]: + if name not in series_names: + series_names.append(name) + + series_summary: dict[str, dict] = {} + for name in series_names: + present = { + label: cell for label, cell in cells.items() if name in cell["ic_stats"] + } + test_by_cell = { + label: float(cell["ic_stats"][name]["test"].get("ic_mean", float("nan"))) + for label, cell in present.items() + } + series_summary[name] = { + "n_cells": len(present), + "test_ic_positive": sum( + 1 for v in test_by_cell.values() if math.isfinite(v) and v > 0 + ), + "sign_consistent": sum( + 1 for cell in present.values() + if bool(cell["sign_consistency"].get(name)) + ), + "test_ic_by_cell": test_by_cell, + } + + ic_beats = 0 + for cell in cells.values(): + eq = float( + cell["performance"]["equal_weight"]["test"].get("annual_return", float("nan")) + ) + ic = float( + cell["performance"]["ic_weighted"]["test"].get("annual_return", float("nan")) + ) + if math.isfinite(eq) and math.isfinite(ic) and ic > eq: + ic_beats += 1 + return { + "n_cells": len(cells), + "ic_beats_eq_test": ic_beats, + "series": series_summary, + } + + +# --------------------------------------------------------------------------- # +# Orchestration +# --------------------------------------------------------------------------- # +def run_phase3_robustness(config_path: str) -> RobustnessResult: + """Run every matrix cell through the P3-3 core and write the matrix report. + + Guards are the single-run preconditions (tushare source, ic_weighted alpha; + the base ``oos`` section validates the base config) plus a ``robustness`` + section. Cells run sequentially (the tushare SDK is rate-limited anyway); + each cell's runtime is recorded and disclosed. + """ + cfg = load_config(config_path) + check_oos_preconditions(cfg, runner="run-phase3-robustness") + if cfg.robustness is None: + raise ValueError( + "run-phase3-robustness requires a 'robustness' config section " + "(universes + windows; optional skip_cells)." + ) + + t0 = time.perf_counter() + log_path = Path(cfg.output.log_dir) / "run_phase3_robustness.log" + logger = _make_logger(log_path, name=_LOGGER_NAME) + cells_to_run = list(iter_cells(cfg)) + skipped = skipped_cell_labels(cfg) + logger.info( + "phase3 robustness start: %d cells to run, %d skipped (%s)", + len(cells_to_run), len(skipped), ", ".join(skipped) or "none", + ) + + cells: dict[str, OOSResult] = {} + runtimes: dict[str, float] = {} + for universe, window in cells_to_run: + label = cell_label(universe, window) + cell_cfg = derive_cell_config(cfg, universe, window) + logger.info("cell %s: start (window %s -> %s, split %s)", + label, window.start, window.end, window.split) + cell_t0 = time.perf_counter() + cells[label] = _run_oos_cell(cell_cfg, logger, log_path) + runtimes[label] = time.perf_counter() - cell_t0 + logger.info("cell %s: done in %.1fs", label, runtimes[label]) + + plain = { + label: { + "performance": c.performance, + "ic_stats": c.ic_stats, + "sign_consistency": c.sign_consistency, + "sign_flips": c.sign_flips, + "n_scored": c.n_scored, + "n_fallback": c.n_fallback, + } + for label, c in cells.items() + } + result = RobustnessResult( + config=cfg, + elapsed_seconds=time.perf_counter() - t0, + cells=cells, + cell_runtimes=runtimes, + skipped_cells=skipped, + summary=summarize_matrix(plain), + report_path=Path(cfg.output.report_dir) / "phase3_robustness_matrix.md", + log_path=log_path, + ) + write_robustness_matrix_summary(result) + logger.info( + "phase3 robustness done: %d cells, ic beats eq in %d, report=%s (%.1fs)", + result.summary["n_cells"], result.summary["ic_beats_eq_test"], + result.report_path, result.elapsed_seconds, + ) + return result diff --git a/tests/test_robustness_matrix.py b/tests/test_robustness_matrix.py new file mode 100644 index 0000000..52440ae --- /dev/null +++ b/tests/test_robustness_matrix.py @@ -0,0 +1,298 @@ +"""P3-4: robustness matrix over the P3-3 OOS cell core (network-free). + +Locks the matrix contract: + * the matrix config validates (universes × windows, unique labels, skips must + reference declared cells, at least one cell must remain); + * per-cell configs are derived from the base verbatim except universe / + window / split / output_name — every cell reuses the SAME P3-3 cell core + (holding-window slicing etc.) and the matrix runner shares the single-run + guards (incl. the ic_weighted fake-comparison guard); + * multi-cell aggregation never mixes cells' numbers; + * the report carries universe/window/fold labels, skipped-cell disclosure, + boundary/fallback/sign-consistency columns, the caveat, and no secret; + * the single-run P3-3 behaviour is unchanged (its own test file still passes). +""" + +from __future__ import annotations + +import dataclasses +from pathlib import Path + +import pytest +import yaml + +from qt.config import ConfigError, load_config +from qt.robustness import ( + cell_label, + derive_cell_config, + iter_cells, + run_phase3_robustness, + summarize_matrix, +) + +_MATRIX_CONFIG = str( + Path(__file__).resolve().parents[1] + / "config" / "phase3_real_robustness_matrix.yaml" +) + + +# --------------------------------------------------------------------------- # +# config validation +# --------------------------------------------------------------------------- # +def test_matrix_config_validates(): + cfg = load_config(_MATRIX_CONFIG) + assert cfg.robustness is not None + assert "000016.SH" in cfg.robustness.universes + assert "000300.SH" in cfg.robustness.universes + labels = [w.label for w in cfg.robustness.windows] + assert len(labels) == len(set(labels)) + assert cfg.alpha.model == "ic_weighted" + + +def _mutate(tmp_path, **patch): + raw = yaml.safe_load(Path(_MATRIX_CONFIG).read_text(encoding="utf-8")) + for key, value in patch.items(): + raw["robustness"][key] = value + p = tmp_path / "m.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + return str(p) + + +def test_matrix_config_rejects_split_outside_window(tmp_path): + bad = [{"label": "w", "start": "2022-07-01", "end": "2024-06-30", + "split": "2024-07-01"}] + with pytest.raises(ConfigError, match="STRICTLY inside"): + load_config(_mutate(tmp_path, windows=bad)) + + +def test_matrix_config_rejects_unknown_skip(tmp_path): + with pytest.raises(ConfigError, match="unknown"): + load_config(_mutate( + tmp_path, skip_cells=[{"universe": "999999.SH", "window": "2022-2024"}] + )) + + +def test_matrix_config_rejects_all_cells_skipped(tmp_path): + raw = yaml.safe_load(Path(_MATRIX_CONFIG).read_text(encoding="utf-8")) + skips = [ + {"universe": u, "window": w["label"]} + for u in raw["robustness"]["universes"] + for w in raw["robustness"]["windows"] + ] + with pytest.raises(ConfigError, match="at least one cell"): + load_config(_mutate(tmp_path, skip_cells=skips)) + + +# --------------------------------------------------------------------------- # +# cell derivation + enumeration +# --------------------------------------------------------------------------- # +def test_iter_cells_excludes_skips_and_keeps_order(): + cfg = load_config(_MATRIX_CONFIG) + cells = list(iter_cells(cfg)) + labels = [cell_label(u, w) for u, w in cells] + # config: 2 universes x 2 windows - 1 skip (CSI300/2020-2022) = 3 cells + assert labels == [ + "000016.SH|2020-2022", "000016.SH|2022-2024", "000300.SH|2022-2024", + ] + + +def test_derive_cell_config_swaps_only_cell_fields(): + cfg = load_config(_MATRIX_CONFIG) + window = cfg.robustness.windows[0] # 2020-2022 + cell = derive_cell_config(cfg, "000300.SH", window) + # swapped-in cell identity + assert cell.universe.index_code == "000300.SH" + assert cell.data.start == window.start and cell.data.end == window.end + assert cell.oos.split_date == window.split + assert cell.data.output_name != cfg.data.output_name # no parquet collision + # everything else identical to the base (no drift into the cells) + assert cell.factors == cfg.factors + assert cell.alpha == cfg.alpha + assert cell.processing == cfg.processing + assert cell.portfolio == cfg.portfolio + assert cell.cost == cfg.cost + # derived cell still passes the OOS preconditions (ic_weighted guard etc.) + from qt.oos_stability import check_oos_preconditions + + check_oos_preconditions(cell, runner="run-phase3-robustness") + + +def test_derive_cell_config_output_names_unique_per_cell(): + cfg = load_config(_MATRIX_CONFIG) + names = { + derive_cell_config(cfg, u, w).data.output_name + for u in cfg.robustness.universes + for w in cfg.robustness.windows + } + assert len(names) == 4 # every (universe, window) gets its own panel file + + +# --------------------------------------------------------------------------- # +# aggregation — no cross-cell mixing +# --------------------------------------------------------------------------- # +def _fake_cell(ic_test_mean: float, consistent: bool, ic_beats_eq: bool): + perf = lambda a: {"annual_return": a, "volatility": 0.15, "sharpe": -0.2, # noqa: E731 + "max_drawdown": -0.1, "avg_turnover": 0.8, "n_rebalances": 11} + eq_test, ic_test = (-0.05, -0.02) if ic_beats_eq else (-0.02, -0.05) + stats = { + "train": {"ic_mean": 0.01 if consistent else -0.01, + "ic_ir": 0.1, "hit_rate": 0.5, "n": 100}, + "test": {"ic_mean": ic_test_mean, "ic_ir": 0.1, "hit_rate": 0.5, "n": 100}, + } + return { + "performance": { + "equal_weight": {"train": perf(-0.06), "test": perf(eq_test)}, + "ic_weighted": {"train": perf(-0.03), "test": perf(ic_test)}, + }, + "ic_stats": {"momentum_20": stats, + "combo_ic_weighted": stats}, + "sign_consistency": {"momentum_20": consistent, + "combo_ic_weighted": consistent}, + "sign_flips": {"momentum_20": 3}, + "n_scored": 200, "n_fallback": 20, + } + + +def test_summarize_matrix_attributes_cells_correctly(): + cells = { + "A|w1": _fake_cell(ic_test_mean=0.02, consistent=True, ic_beats_eq=True), + "B|w1": _fake_cell(ic_test_mean=-0.03, consistent=False, ic_beats_eq=False), + } + summary = summarize_matrix(cells) + mom = summary["series"]["momentum_20"] + assert mom["n_cells"] == 2 + assert mom["test_ic_positive"] == 1 # only cell A + assert mom["sign_consistent"] == 1 # only cell A + assert summary["ic_beats_eq_test"] == 1 # only cell A + assert summary["n_cells"] == 2 + + +def test_summarize_matrix_single_cell_not_diluted(): + summary = summarize_matrix( + {"A|w1": _fake_cell(ic_test_mean=0.02, consistent=True, ic_beats_eq=True)} + ) + mom = summary["series"]["momentum_20"] + assert mom["n_cells"] == 1 and mom["test_ic_positive"] == 1 + + +# --------------------------------------------------------------------------- # +# runner guards (no network) +# --------------------------------------------------------------------------- # +def test_matrix_runner_rejects_demo_source(example_config_path): + with pytest.raises(ValueError, match="tushare|REAL"): + run_phase3_robustness(example_config_path) + + +def test_matrix_runner_requires_robustness_section(tmp_path): + raw = yaml.safe_load(Path(_MATRIX_CONFIG).read_text(encoding="utf-8")) + raw.pop("robustness") + p = tmp_path / "no_matrix.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + with pytest.raises(ValueError, match="robustness"): + run_phase3_robustness(str(p)) + + +def test_matrix_runner_rejects_non_ic_weighted_alpha(tmp_path): + raw = yaml.safe_load(Path(_MATRIX_CONFIG).read_text(encoding="utf-8")) + raw["alpha"] = {"model": "equal_weight", "params": {}} + p = tmp_path / "eq.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + with pytest.raises(ValueError, match="ic_weighted"): + run_phase3_robustness(str(p)) + + +# --------------------------------------------------------------------------- # +# report rendering +# --------------------------------------------------------------------------- # +def _synthetic_matrix_result(): + from qt.robustness import RobustnessResult + from tests.test_oos_stability import _synthetic_oos_result + + base = _synthetic_oos_result() + cell = dataclasses.replace( + base, + downgrades=( + "DATA PATH = REAL tushare: PIT index membership (000016.SH); ...", + "shared-disclosure line", + ), + ) + cells = { + "000016.SH|2022-2024": cell, + "000300.SH|2022-2024": dataclasses.replace( + cell, n_fallback=33, sign_flips={"a": 5, "b": 1}, + downgrades=( + "DATA PATH = REAL tushare: PIT index membership (000300.SH); ...", + "shared-disclosure line", + ), + ), + } + plain = { + label: { + "performance": c.performance, + "ic_stats": c.ic_stats, + "sign_consistency": c.sign_consistency, + "sign_flips": c.sign_flips, + "n_scored": c.n_scored, + "n_fallback": c.n_fallback, + } + for label, c in cells.items() + } + return RobustnessResult( + config=load_config(_MATRIX_CONFIG), + elapsed_seconds=1000.0, + cells=cells, + cell_runtimes={"000016.SH|2022-2024": 900.0, "000300.SH|2022-2024": 100.0}, + skipped_cells=("000300.SH|2020-2022",), + summary=summarize_matrix(plain), + report_path=Path("artifacts/reports/phase3_robustness_matrix.md"), + log_path=Path("artifacts/logs/run_phase3_robustness.log"), + ) + + +def test_render_matrix_report_covers_cells_summary_and_caveat(): + from qt.robustness import render_robustness_matrix + + md = render_robustness_matrix(_synthetic_matrix_result()) + # universe x window cell labels + skipped-cell disclosure + assert "000016.SH|2022-2024" in md and "000300.SH|2022-2024" in md + assert "000300.SH|2020-2022" in md and "skipped" in md.lower() + # per-cell content: performance models, IC stability, weights diagnostics + assert "equal_weight" in md and "ic_weighted" in md + assert "sign consistency" in md.lower() or "sign-consistency" in md.lower() + assert "sign flip" in md.lower() and "fallback" in md.lower() + # boundary disclosure survives into the matrix report + assert "boundary" in md.lower() + # cross-cell summary matrix + assert "cells" in md.lower() + # caveat + assert "not a" in md.lower() and "claim" in md.lower() + + +def test_render_matrix_downgrades_cover_all_universes(): + """The DOWNGRADES/caveats section must disclose EVERY run universe — never + only the first cell's universe-specific lines (MEDIUM review finding).""" + from qt.robustness import render_robustness_matrix + + md = render_robustness_matrix(_synthetic_matrix_result()) + caveats = md.split("## DOWNGRADES")[1] + # matrix-level scope line with run cells / skipped cells / universes / windows + assert "MATRIX SCOPE" in caveats + assert "000016.SH|2022-2024" in caveats and "000300.SH|2022-2024" in caveats + assert "000300.SH|2020-2022" in caveats # the skipped cell, in the scope line + # BOTH universes' membership disclosures present (union, not first-cell-only) + assert "PIT index membership (000016.SH)" in caveats + assert "PIT index membership (000300.SH)" in caveats + # the shared line is deduplicated (union, not concatenation) + assert caveats.count("shared-disclosure line") == 1 + # the per-cell caveat no longer reads as a single-universe claim + assert "one index, two years" not in md + assert "multiple" in caveats.lower() + + +def test_render_matrix_report_leaks_no_secret(): + from qt.robustness import render_robustness_matrix + + result = _synthetic_matrix_result() + md = render_robustness_matrix(result) + assert result.config.data.external_secret_file not in md + assert "token" not in md.lower()