From 3a4141560a577fb4c16ed49b4112a8f3bf711672 Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Wed, 10 Jun 2026 16:54:46 +0800 Subject: [PATCH 1/5] feat(validation): P3-3 OOS stability validation (equal_weight vs ic_weighted) A REPORT-ONLY validation layer -- no new alpha complexity, no change to portfolio / execution / factor math. Validates whether the P3-1 equal-weight and P3-2 ic-weighted combinations are stable out of sample before adding any further alpha sophistication. Runner (qt/oos_stability.py, CLI run-phase3-oos): - ONE shared data load + ONE processed factor panel; the backtest runs twice (EqualWeightAlpha / RollingICWeightAlpha) under identical universe, neutralization, portfolio and execution rules. - Every diagnostic is split at oos.split_date: train = [start, split), test = [split, end]; subperiod navs are rebased so nothing bleeds across. - Split semantics are walk-forward (rolling subperiod): weights at any date use only observations realized by that date (t + h <= d). A split-boundary test proves perturbing EVERY post-split forward return leaves all train-period weights bit-identical. Freeze-at-split is deliberately NOT added (it would be a new alpha mode, out of scope). - Guards: requires data.source=tushare and an 'oos' config section; the split date must lie strictly inside the data window (config-validated). Report (phase3_oos_stability.md): split boundaries (exact realized dates + day counts), per-subperiod performance for both models (annual / vol / sharpe / maxDD / turnover / rebalances), per-series IC stability (mean / IR / hit rate / n + train-vs-test sign consistency) for every raw factor and both combo scores, ic-weighted weight stability (per-rebalance weights labelled train/test, sign flips on trained rows, fallback count + reasons), and the explicit small-sample / not-a-return-claim caveat. Real run (SSE50, 2y 2022-07~2024-06, split 2023-07-01, 77 names, ~16 min) -- the honest headline is INSTABILITY: - all three raw factors FLIP IC sign train->test (momentum_20 -0.024->+0.006, roe -0.029->+0.007, netprofit_yoy -0.010->+0.005; sign consistency NO); hit rates 46-53% (coin-flip range); - ic_weighted weights flip sign 7/3/4 times across 23 rebalances; - performance: equal_weight train -6.81% / test -5.27%; ic_weighted train -1.69% / test -2.70%. ic_weighted edges equal_weight in BOTH subperiods, but with near-zero, sign-flipping ICs this must NOT be read as skill -- the P3-2 one-year result does not extrapolate, which is exactly what this validation layer was built to surface. Regressions: phase3 equal_weight rerun -9.05% / IC 0.0083 and ic_weighted rerun -3.57% both unchanged; demo unchanged (ic 0.96 / annual 0.84); the report contains no token / secret-file content (scanned). Tests: 282 passed (was 269): +13 OOS (split no-leakage, subperiod slicing, IC period stats, sign consistency, weight sign flips excl. fallback rows, fallback-reason aggregation, config + guard + render contracts). --- AGENTS.md | 10 +- CLAUDE.md | 12 +- RUNBOOK.md | 36 +++ TEST_REPORT.md | 24 +- config/phase3_real_oos_stability.yaml | 112 ++++++++ qt/cli.py | 27 ++ qt/config.py | 39 +++ qt/oos_stability.py | 359 ++++++++++++++++++++++++++ qt/reports.py | 141 ++++++++++ tests/test_oos_stability.py | 276 ++++++++++++++++++++ 10 files changed, 1025 insertions(+), 11 deletions(-) create mode 100644 config/phase3_real_oos_stability.yaml create mode 100644 qt/oos_stability.py create mode 100644 tests/test_oos_stability.py diff --git a/AGENTS.md b/AGENTS.md index d7d2ce8..fb1f165 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -97,6 +97,10 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - rolling(默认 60d/min20)/expanding;历史不足→该日退回等权(计数披露);权重 L1 归一化、保留符号。报告新增 **Alpha model** 必含小节(模型/超参/覆盖率/每期权重表/fallback/非调参声明)。 - `config/phase3_real_ic_weighted.yaml` 与 phase3_real_multifactor 唯一差异 alpha.model(直接可比)。 - 真实结果:annual **−3.57%**(等权 −9.05%),训练覆盖 201/221(20 fallback 全在窗口攒满前)。⚠️ 优于等权非业绩声明——单年窗口+权重逐期翻号即小样本不稳定,照实披露。回归不破:等权 rerun −9.05%/0.0083 不变,demo 0.96/0.84 不变。 -- ✅ 当前质量门：`pytest -p no:cacheprovider` **269 passed**；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`）OK；`run-phase0`（demo）OK。 -- ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据。 -- 路线图下一步：样本外/IC 稳定性检验 / 分钟级（architecture.html §11）。 +- 🔧 **Phase 3-3 OOS 稳定性验证**（`p3-oos-stability-validation` 分支,代劳待验收）：报告型验证层（`run-phase3-oos` + `qt/oos_stability.py` + `config/phase3_real_oos_stability.yaml`,SSE50 2 年,split 2023-07-01）;不加 alpha 复杂度、不改 portfolio/execution/factor math。 + - 一次数据加载、两次回测(equal_weight vs ic_weighted),诊断按 split 切段(子段 nav 重归一)。walk-forward 边界测试锁定:扰动 split 后全部 fwd,train 期权重逐 bit 不变。 + - 报告:split 边界/分期绩效/逐序列 IC(mean/IR/hit/sign consistency)/权重稳定性(sign flips、fallback+原因)/小样本 caveat。 + - **真实关键发现**:三因子 train→test IC **全部翻号**(sign consistency 全 NO),hit 46~53%;权重 sign flips 7/3/4;eq −6.81%/−5.27%,ic −1.69%/−2.70%(train/test)。**P3-2 单年跑赢不可外推——这正是验证层要拿到的证据;非收益声明。**回归不破:eq −9.05%/ic −3.57%/demo 0.96/0.84 全不变;报告 secret scan 干净。 +- ✅ 当前质量门：`pytest -p no:cacheprovider` **282 passed**；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）OK；`run-phase0`（demo）OK。 +- ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、因子 IC 小样本不稳定（P3-3 实证）。 +- 路线图下一步：更长历史/更宽 universe 的稳定性复检,或分钟级（architecture.html §11）。 diff --git a/CLAUDE.md b/CLAUDE.md index 0987dcd..8110bbb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -101,6 +101,12 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - `config/phase3_real_ic_weighted.yaml`：与 phase3_real_multifactor **唯一差异是 alpha.model**（universe/window/因子/中性化全同,直接可比）。 - **真实结果**（SSE50 2023-07~2024-06,~14min）:annual **−3.57%**（等权 −9.05% / 单因子 −10.19%）,maxDD −12.93%,训练覆盖 **201/221**（20 个 fallback 全在窗口攒满前,90.95%）。⚠️ 优于等权**不是**业绩声明——单年窗口 + 权重逐期翻号（如 momentum_20 从 −0.58 到 +0.36）正是小样本不稳定的体现,照实披露。 - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变（测试锁定）。 -- ✅ 质量门：`pytest` **269 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`）+ `run-phase0`（demo）均 OK。 -- ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据。 -- 路线图下一步：样本外/IC 稳定性检验 / 分钟级（architecture.html §11）。 +- 🔧 **Phase 3-3 OOS 稳定性验证**（`p3-oos-stability-validation` 分支,代劳待验收）：**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`（`qt/oos_stability.py`）+ `config/phase3_real_oos_stability.yaml`（SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照）。 + - **一次数据加载、同一 processed 因子面板、两次回测**（equal_weight vs ic_weighted）;所有诊断按 split 切段（子段 nav 重新归一,绝不跨段串味）。 + - **边界语义（测试锁定）**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。 + - 报告 `phase3_oos_stability.md`:split 边界/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 + - **真实结果（关键发现,~16min,77 成分/2 年）**:三个原始因子 train→test **IC 全部翻号**(momentum −0.024→+0.006 / roe −0.029→+0.007 / np_yoy −0.010→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −6.81%/test −5.27%,ic train −1.69%/test −2.70%。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** + - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 +- ✅ 质量门：`pytest` **282 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=13）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）+ `run-phase0`（demo）均 OK。 +- ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、因子 IC 小样本不稳定（P3-3 实证）。 +- 路线图下一步：更长历史/更宽 universe 的稳定性复检,或分钟级（architecture.html §11）。 diff --git a/RUNBOOK.md b/RUNBOOK.md index e4f0684..9a87c5f 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -262,6 +262,42 @@ The baseline report gains an **Alpha model** section: active model, hyper-params, training coverage (fallback count), and the effective weights at every settled rebalance date (fallback rows flagged). +## Phase 3-3 — OOS stability validation (equal_weight vs ic_weighted) + +A REPORT-ONLY validation layer (no portfolio / execution / factor-math change): +one shared data load, the SAME processed factor panel, two backtests — one per +alpha — and every diagnostic split at `oos.split_date`. Documented by +`config/phase3_real_oos_stability.yaml` (SSE50, two years 2022-07 ~ 2024-06, +split 2023-07-01 → train 1y / test 1y; the test year equals the phase3 +baselines' window, so numbers line up). + +```bash +# validate (no network) +... -m qt.cli validate-config --config config/phase3_real_oos_stability.yaml +# run the OOS validation (network + token; heavy, ~15-30 min) +... -m qt.cli run-phase3-oos --config config/phase3_real_oos_stability.yaml +``` + +Split semantics (locked by tests): + +- train = `[data.start, split)`, test = `[split, data.end]`; the exact realized + dates and day counts are written into the report. +- Evaluation is **walk-forward (rolling subperiod)**: weights at any date d use + only observations REALIZED by d (`t + horizon <= d`) — perturbing every + post-split forward return cannot change any train-period date's weights + (split-boundary no-leakage test). Freezing weights at the split is NOT used + (that would be a new alpha mode; P3-3 adds no alpha complexity). +- The config's `alpha` section carries the ic_weighted params; the run ALSO + evaluates `equal_weight` on the same data as the control. + +Report (`artifacts/reports/phase3_oos_stability.md`): split boundaries; +per-subperiod performance for both models (annual / vol / Sharpe / maxDD / +turnover / rebalances); per-series IC stability (mean / IR / hit rate / n + +train-vs-test sign consistency) for every raw factor and both combo scores; +ic_weighted weight stability (per-rebalance weights with train/test labels, +sign-flip counts on trained rows, fallback count + reasons); and the explicit +caveat that this is a small-sample stability check, NOT a return claim. + ## Quality gate ```bash diff --git a/TEST_REPORT.md b/TEST_REPORT.md index 53a764e..07c98a1 100644 --- a/TEST_REPORT.md +++ b/TEST_REPORT.md @@ -1,4 +1,4 @@ -# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha) +# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha → OOS stability) ## Commands @@ -12,6 +12,7 @@ Run from the repo root with the project python (env `quant_mf`): /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase2_real_baseline.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_multifactor.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_ic_weighted.yaml +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_oos_stability.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli run-phase0 --config config/example.yaml ``` @@ -19,12 +20,12 @@ Run from the repo root with the project python (env `quant_mf`): | Gate | Command | Result | |---|---|---| -| Unit + integration | `pytest -q` | **269 passed, 0 failed** | +| Unit + integration | `pytest -q` | **282 passed, 0 failed** | | Lint | `ruff check .` | **All checks passed** | -| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`) | exit `0`, prints `OK` | +| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`) | exit `0`, prints `OK` | | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | -Counts below are the actual per-file `pytest` numbers (sum = 269). +Counts below are the actual per-file `pytest` numbers (sum = 282). ## Per-file breakdown — Phase 0 core (97) @@ -109,7 +110,13 @@ Counts below are the actual per-file `pytest` numbers (sum = 269). |---|---|---| | `test_ic_weight_alpha.py` | 12 | **lookahead red-line**: perturbing unrealized forward returns cannot change weights; exact `t + h <= d` realization cutoff (min_periods boundary); insufficient-history equal-weight fallback (== EqualWeightAlpha row mean); single-factor degeneration to ±1; L1 normalization + sign preservation; degenerate-IC fallback; rolling-vs-expanding window; fit requires forward_returns; dated-cross-section contract; input immutability; weights/fallback log | | `test_ic_alpha_pipeline.py` | 6 | alpha dispatch by config (equal_weight / ic_weighted + params / unknown = ConfigError), equal-weight default keeps exact demo numbers (ic 0.96 / annual 0.84) + report line, ic_weighted demo e2e (summary, weights log, early-fallback→late-trained, L1 rows, report disclosure, no secret), ic-weights differ from equal weight on diverging-IC synthetic data | -| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2)** | **269** | | + +## Per-file breakdown — Phase 3-3 OOS stability (13) + +| Test file | Tests | Red-line / feature | +|---|---|---| +| `test_oos_stability.py` | 13 | **split-boundary no-leakage**: perturbing every post-split forward return leaves all train-period weights bit-identical; `subperiod_perf` slices strictly + rebased nav + empty-slice NaN; `ic_period_stats` mean/IR/hit-rate/n; `sign_consistent` nonzero-same-sign; `weight_sign_flips` on trained rows only (fallback rows excluded); fallback-reason aggregation; OOS config validates + split-inside-window ConfigError; runner rejects demo source + missing `oos` section; report renders boundaries / OOS metrics / weight stability / caveat / no secret | +| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3)** | **282** | | ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) @@ -182,6 +189,13 @@ Counts below are the actual per-file `pytest` numbers (sum = 269). EqualWeightAlpha combination — and is counted in the report. Weights are L1-normalized, sign-preserving. `EqualWeightAlpha` remains the default; its demo numbers (ic 0.96 / annual 0.84) are locked unchanged. +- **P3-3 OOS stability (locked by tests):** `run-phase3-oos` is a REPORT-ONLY + validation layer — one shared data load, two backtests (equal_weight vs + ic_weighted), every diagnostic split at `oos.split_date` (train strictly + before, test on/after; subperiod navs rebased so nothing bleeds across). + Evaluation is walk-forward (rolling subperiod): the split-boundary test + proves post-split forward returns cannot move any train-period weight. + Portfolio / execution / factor math are untouched. - A duplicate test-function name across two files was found and renamed during P2-2 (it had been silently shadowing one test in the full-suite run). A second, harmless duplicate (`test_enrich_does_not_mutate_input` in two files) was verified NOT to drop diff --git a/config/phase3_real_oos_stability.yaml b/config/phase3_real_oos_stability.yaml new file mode 100644 index 0000000..1e5c31b --- /dev/null +++ b/config/phase3_real_oos_stability.yaml @@ -0,0 +1,112 @@ +# Phase 3-3 — REAL out-of-sample stability validation (equal_weight vs ic_weighted). +# +# Validation layer, NOT a new strategy: same SSE50 universe, same three factors, +# same neutralization / portfolio / execution rules as the phase3 baselines — +# the window is EXTENDED to two years and split at oos.split_date: +# train = [data.start, split) test = [split, data.end] +# Evaluation is walk-forward (rolling subperiod): weights at any date d use only +# observations REALIZED by d (t + horizon <= d), so no test-period forward +# return can reach a train-period computation (locked by tests). The alpha +# section carries the ic_weighted params; the run ALSO evaluates equal_weight +# on the same data as the control. NOT a return claim — a small-sample +# stability check. +# +# Run: python -m qt.cli run-phase3-oos --config config/phase3_real_oos_stability.yaml +# (report: artifacts/reports/phase3_oos_stability.md; needs the tushare token in +# the external .config.json; hits the network; heavy ~15-25 min.) + +project: + name: quantitative_trading_phase3_oos_stability + timezone: Asia/Shanghai + +data: + source: tushare + freq: D + start: "2022-07-01" + end: "2024-06-30" + external_secret_file: "/home/shaofl/Projects/financial_projects/.config.json" + tushare_token_key: "tushare.token" + output_name: phase3oos_daily + +universe: + type: index + index_code: "000016.SH" + symbols: [] + min_listing_days: 60 + filters: + missing_close: true + suspended: true + st: true + limit_up_down: true + +factors: + - name: momentum_20 + enabled: true + params: + window: 20 + price_col: close + - name: roe + enabled: true + params: {} + - name: netprofit_yoy + enabled: true + params: {} + +processing: + drop_missing: true + standardize: + enabled: true + method: zscore + winsorize: + enabled: false + method: mad + n: 3.0 + neutralize: + enabled: true + industry_col: industry + size_col: market_cap + industry_level: L1 # PIT SW level (L1/L2/L3); L1 = 31 broad sectors (standard, DOF-safe) + +alpha: + model: ic_weighted # P3-2 walk-forward rolling-IC weights + params: + window: 60 # trailing trading days of REALIZED ICs (conservative rolling) + min_periods: 20 # < this many valid realized ICs -> equal-weight fallback (disclosed) + mode: rolling # rolling (default) | expanding + +portfolio: + constructor: topn_equal_weight + top_n: 20 + long_only: true + max_weight: null + turnover_cap: null + +backtest: + initial_nav: 1.0 + rebalance: monthly + event_order: close_to_next_period + cash_return: 0.0 + +cost: + fee_rate: 0.001 + slippage_rate: 0.0 + turnover_formula: l1 + +analytics: + forward_return_periods: + - 1 + - 5 + - 20 + quantiles: 5 + benchmark: null + +output: + root_dir: artifacts + data_dir: artifacts/data + factor_dir: artifacts/factors + report_dir: artifacts/reports + log_dir: artifacts/logs + overwrite: true + +oos: + split_date: "2023-07-01" # train = 1y [2022-07-01, split); test = 1y [split, 2024-06-30] diff --git a/qt/cli.py b/qt/cli.py index efc8016..18c9d79 100644 --- a/qt/cli.py +++ b/qt/cli.py @@ -91,6 +91,26 @@ def _cmd_run_phase2_baseline(args: argparse.Namespace) -> int: return 0 +def _cmd_run_phase3_oos(args: argparse.Namespace) -> int: + """Run the OOS stability validation (equal_weight vs ic_weighted) + report.""" + from qt.oos_stability import run_phase3_oos + + try: + result = run_phase3_oos(args.config) + except (ConfigError, ValueError, FileNotFoundError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + eq = result.performance["equal_weight"]["test"].get("annual_return", float("nan")) + ic = result.performance["ic_weighted"]["test"].get("annual_return", float("nan")) + print( + f"OK run-phase3-oos: split={result.split_date.date()}, " + f"test annual eq={eq:.4f} ic={ic:.4f}, fallbacks={result.n_fallback} " + f"({result.elapsed_seconds:.1f}s)\n" + f"report: {result.report_path}" + ) + return 0 + + def _cmd_fetch_data(args: argparse.Namespace) -> int: """Stage helper: run the spine and report the data-fetch stage.""" return _run_pipeline_cmd(args.config, "fetch-data") @@ -129,6 +149,13 @@ def build_parser() -> argparse.ArgumentParser: p_p2.add_argument("--config", required=True, help="Path to the YAML config.") p_p2.set_defaults(func=_cmd_run_phase2_baseline) + p_oos = sub.add_parser( + "run-phase3-oos", + help="Run the REAL-data OOS stability validation (equal_weight vs ic_weighted).", + ) + p_oos.add_argument("--config", required=True, help="Path to the YAML config.") + p_oos.set_defaults(func=_cmd_run_phase3_oos) + for name, func, help_text in ( ("fetch-data", _cmd_fetch_data, "Run the spine, report data fetch."), ("compute-factors", _cmd_compute_factors, "Run the spine, report factor compute."), diff --git a/qt/config.py b/qt/config.py index 2df2151..60e97c5 100644 --- a/qt/config.py +++ b/qt/config.py @@ -184,12 +184,29 @@ class OutputCfg(_Strict): baseline_report_name: str | None = None +class OOSCfg(_Strict): + """P3-3 out-of-sample split: train = [data.start, split_date), test = + [split_date, data.end]. Evaluation is walk-forward (rolling subperiod): + weights at any date use only observations realized by that date, so no + test-period forward return can reach a train-period computation.""" + + split_date: str + + @field_validator("split_date", mode="before") + @classmethod + def _coerce_date_to_str(cls, v: Any) -> Any: + if isinstance(v, (_date, datetime)): + return v.strftime("%Y-%m-%d") + return v + + class RootConfig(_Strict): """Top-level config composing every section. Required top-level sections (CFG-002): data, universe, factors, alpha, portfolio, backtest, cost, output. ``project``, ``processing`` and ``analytics`` have sensible defaults but are present in the template. + ``oos`` is optional and only consumed by ``run-phase3-oos``. """ project: ProjectCfg = Field(default_factory=lambda: ProjectCfg(name="quantitative_trading")) @@ -203,6 +220,28 @@ class RootConfig(_Strict): cost: CostCfg analytics: AnalyticsCfg = Field(default_factory=AnalyticsCfg) output: OutputCfg + oos: OOSCfg | None = None + + @model_validator(mode="after") + def _check_oos_split_inside_window(self) -> "RootConfig": + if self.oos is None: + return self + try: + split = datetime.strptime(self.oos.split_date, "%Y-%m-%d") + except ValueError as exc: + raise ValueError( + f"oos.split_date must be a 'YYYY-MM-DD' date; got " + f"{self.oos.split_date!r} ({exc})." + ) from exc + start = datetime.strptime(self.data.start, "%Y-%m-%d") + end = datetime.strptime(self.data.end, "%Y-%m-%d") + if not (start < split < end): + raise ValueError( + f"oos.split_date ({self.oos.split_date}) must lie STRICTLY inside " + f"the data window ({self.data.start}, {self.data.end}) so both the " + "train and test subperiods are non-empty." + ) + return self # --------------------------------------------------------------------------- # diff --git a/qt/oos_stability.py b/qt/oos_stability.py new file mode 100644 index 0000000..e2aa669 --- /dev/null +++ b/qt/oos_stability.py @@ -0,0 +1,359 @@ +"""Phase 3-3: out-of-sample stability validation (equal_weight vs ic_weighted). + +A REPORT-ONLY validation layer, not a new strategy: on ONE shared data load +(same universe, same factor panel, same processing / portfolio / execution +rules) it runs the backtest twice — once under :class:`EqualWeightAlpha`, once +under :class:`RollingICWeightAlpha` — and splits every diagnostic at +``oos.split_date``: + + train = [data.start, split) test = [split, data.end] + +Evaluation is WALK-FORWARD (rolling subperiod): the ic-weighted model derives +each date's weights from observations realized by that date only (t + h <= d, +P3-2, locked by tests), so no test-period forward return can reach any +train-period computation — the split here is an ACCOUNTING boundary for the +statistics, not a new training mode. Nothing in portfolio / execution / factor +math changes. + +The report (``phase3_oos_stability.md``) carries: the exact split dates, +per-subperiod performance for both models, per-series IC stability (mean / IR / +hit rate / cross-period sign consistency) for every raw factor and both combo +scores, the ic-weighted weight-stability diagnostics (per-rebalance weights, +sign flips, fallback counts + reasons), and the explicit small-sample caveat. +Guard: requires the real tushare source and an ``oos`` config section. +""" + +from __future__ import annotations + +import math +import time +from dataclasses import dataclass +from pathlib import Path + +import pandas as pd + +from alpha.equal_weight import EqualWeightAlpha +from analytics.factor import compute_ic, forward_returns +from analytics.performance import performance_summary +from portfolio.construct import TopNEqualWeight +from qt.config import RootConfig, load_config +from qt.pipeline import ( + _FrameScores, + _alpha_disclosure, + _build_alpha, + _build_factors, + _build_scores, + _build_universe, + _collect_downgrades, + _compute_factor_panel, + _load_panel, + _make_logger, + _maybe_enrich_covariates, + _maybe_enrich_financials, + _maybe_enrich_listing, + _periods_per_year, + _process_factors, +) +from qt.reports import render_oos_stability, write_oos_stability_summary +from runtime.backtest.driver import BacktestDriver +from runtime.backtest.sim_execution import SimExecution + +__all__ = ["OOSResult", "run_phase3_oos", "render_oos_stability", + "subperiod_perf", "ic_period_stats", "sign_consistent", + "weight_sign_flips", "fallback_reason_counts"] + +_LOGGER_NAME = "qt.run_phase3_oos" +_PERF_NAN = { + "annual_return": float("nan"), "volatility": float("nan"), + "sharpe": float("nan"), "max_drawdown": float("nan"), + "avg_turnover": float("nan"), "n_rebalances": 0, +} + + +@dataclass(frozen=True) +class OOSResult: + """Immutable summary of one OOS stability run (what the report consumes).""" + + config: RootConfig + elapsed_seconds: float + # split boundaries (actual panel dates, disclosed verbatim in the report) + split_date: pd.Timestamp + train_start: pd.Timestamp | None + train_end: pd.Timestamp | None + test_start: pd.Timestamp | None + test_end: pd.Timestamp | None + n_train_days: int + n_test_days: int + factor_names: tuple[str, ...] + # performance[model][period] -> {annual_return, volatility, sharpe, + # max_drawdown, avg_turnover, n_rebalances}; model in (equal_weight, + # ic_weighted); period in (train, test). + performance: dict[str, dict[str, dict]] + # ic_stats[series][period] -> {ic_mean, ic_ir, hit_rate, n}; series = every + # raw factor + combo_equal_weight + combo_ic_weighted. + ic_stats: dict[str, dict[str, dict]] + # sign_consistency[series] -> True iff train/test mean ICs share a nonzero sign. + sign_consistency: dict[str, bool] + # ic_weighted weight stability + weights_at_rebalances: pd.DataFrame + sign_flips: dict[str, int] + n_scored: int + n_fallback: int + fallback_reasons: dict[str, int] + alpha_summary: dict + downgrades: tuple[str, ...] + report_path: Path + log_path: Path + + +# --------------------------------------------------------------------------- # +# Pure subperiod statistics (network-free; unit-tested with synthetic inputs). +# --------------------------------------------------------------------------- # +def subperiod_perf( + nav_table: pd.DataFrame, + start: pd.Timestamp | None = None, + end: pd.Timestamp | None = None, + periods_per_year: int = 12, +) -> dict: + """Performance over the nav rows in [start, end) — period-local compounding. + + The subperiod nav is REBASED to 1.0 from the slice's own net returns, so a + drawdown or annualization never bleeds across the split. ``end`` is + exclusive (a rebalance ON the split date belongs to the test period). + Empty slices return NaN metrics (never crash). + """ + if nav_table is None or nav_table.empty: + return dict(_PERF_NAN) + sliced = nav_table + if start is not None: + sliced = sliced[sliced.index >= pd.Timestamp(start)] + if end is not None: + sliced = sliced[sliced.index < pd.Timestamp(end)] + if sliced.empty: + return dict(_PERF_NAN) + local_nav = (1.0 + sliced["net_return"]).cumprod() + perf = performance_summary(local_nav, periods_per_year=periods_per_year) + return { + **perf, + "avg_turnover": float(sliced["turnover"].mean()), + "n_rebalances": int(len(sliced)), + } + + +def ic_period_stats(ic: pd.Series, split: pd.Timestamp) -> dict[str, dict]: + """{'train': {...}, 'test': {...}} stats of a per-date IC series. + + Each period reports the NaN-dropped mean, IR (mean/std, ddof=1), hit rate + (share of positive ICs) and observation count. Train is strictly before the + split; test is on/after it. + """ + split = pd.Timestamp(split) + out: dict[str, dict] = {} + for name, mask in (("train", ic.index < split), ("test", ic.index >= split)): + clean = ic[mask].dropna() + n = int(len(clean)) + mean = float(clean.mean()) if n else float("nan") + std = float(clean.std(ddof=1)) if n > 1 else float("nan") + ir = mean / std if (math.isfinite(std) and std != 0) else float("nan") + hit = float((clean > 0).mean()) if n else float("nan") + out[name] = {"ic_mean": mean, "ic_ir": float(ir), "hit_rate": hit, "n": n} + return out + + +def sign_consistent(stats: dict[str, dict]) -> bool: + """True iff the train and test mean ICs share the SAME nonzero finite sign.""" + tr = float(stats.get("train", {}).get("ic_mean", float("nan"))) + te = float(stats.get("test", {}).get("ic_mean", float("nan"))) + if not (math.isfinite(tr) and math.isfinite(te)) or tr == 0.0 or te == 0.0: + return False + return (tr > 0) == (te > 0) + + +def weight_sign_flips(weights_log: pd.DataFrame) -> dict[str, int]: + """Per-factor count of sign changes between CONSECUTIVE TRAINED rows. + + Fallback rows carry all-positive equal weights, so including them would + inject artificial flips — they are excluded here and disclosed separately + (fallback count / reasons). + """ + if weights_log is None or weights_log.empty: + return {} + trained = weights_log.loc[~weights_log["fallback"].astype(bool)] + factor_cols = [c for c in weights_log.columns if c != "fallback"] + flips: dict[str, int] = {} + for col in factor_cols: + signs = trained[col].apply( + lambda v: 0 if (pd.isna(v) or v == 0) else (1 if v > 0 else -1) + ) + signs = signs[signs != 0] + flips[col] = int((signs.diff().fillna(0) != 0).sum()) + return flips + + +def fallback_reason_counts(fallback_log: dict) -> dict[str, int]: + """Aggregate the per-date fallback reasons into reason -> count (None = trained).""" + counts: dict[str, int] = {} + for reason in fallback_log.values(): + if reason is not None: + counts[reason] = counts.get(reason, 0) + 1 + return counts + + +def _oos_downgrades(cfg: RootConfig) -> tuple[str, ...]: + """Base downgrades + the OOS-validation caveats (INV-007).""" + extra = ( + "OOS VALIDATION SEMANTICS: this run evaluates BOTH equal_weight and " + "ic_weighted on one shared data load; the split is an ACCOUNTING boundary " + "for subperiod statistics. Weight training is walk-forward (a pair enters " + "date d's weights only once realized, t + horizon <= d), so no test-period " + "forward return reaches any train-period computation; freezing weights at " + "the split is NOT used (that would be a new alpha mode, out of scope).", + "This is a SMALL-SAMPLE stability check (one index, two years, ~22 " + "rebalances), NOT a return claim and NOT a tuned result: subperiod " + "metrics carry wide uncertainty and must not be read as expected " + "performance.", + ) + return _collect_downgrades(cfg) + extra + + +# --------------------------------------------------------------------------- # +# Orchestration +# --------------------------------------------------------------------------- # +def _run_backtest_for(cfg: RootConfig, panel, universe, score_panel) -> pd.DataFrame: + """One backtest run for a given score panel (same rules for both models).""" + driver = BacktestDriver( + universe=universe, + scores=_FrameScores(score_panel), + constructor=TopNEqualWeight(cfg.portfolio.top_n, long_only=cfg.portfolio.long_only), + execution=SimExecution(fee_rate=cfg.cost.fee_rate), + prices=panel, + rebalance=cfg.backtest.rebalance, + fee_rate=cfg.cost.fee_rate, + initial_nav=cfg.backtest.initial_nav, + cash_return=cfg.backtest.cash_return, + ) + return driver.run() + + +def run_phase3_oos(config_path: str) -> OOSResult: + """Run the OOS stability validation and write the phase3 OOS report. + + Requires ``data.source='tushare'`` (a stability check on offline demo data + would be a category error) and an ``oos`` config section (the split date). + All writes land under the configured output dirs; no secret is echoed. + """ + cfg = load_config(config_path) + if cfg.data.source != "tushare": + raise ValueError( + "run-phase3-oos is a REAL-data validation and requires " + "data.source='tushare'; the demo path carries no PIT/ann_date meaning. " + f"Got data.source={cfg.data.source!r}." + ) + if cfg.oos is None: + raise ValueError( + "run-phase3-oos requires an 'oos' config section with split_date " + "(train = [data.start, split), test = [split, data.end])." + ) + split = pd.Timestamp(cfg.oos.split_date) + + t0 = time.perf_counter() + log_path = Path(cfg.output.log_dir) / "run_phase3_oos.log" + logger = _make_logger(log_path, name=_LOGGER_NAME) + logger.info("phase3 oos start: project=%s split=%s", cfg.project.name, split.date()) + + # --- ONE shared data load / factor panel (identical inputs for both) ----- # + universe, symbols = _build_universe(cfg, logger) + panel = _load_panel(cfg, symbols, logger) + factors = _build_factors(cfg) + panel = _maybe_enrich_financials(cfg, panel, symbols, factors, logger) + panel = _maybe_enrich_covariates(cfg, panel, symbols, logger) + panel = _maybe_enrich_listing(cfg, panel, symbols, logger) + factor_panel = _compute_factor_panel(cfg, panel, factors, logger) + processed = _process_factors(cfg, factor_panel, panel) + + horizon = int(cfg.analytics.forward_return_periods[0]) + fwd_col = forward_returns(panel, periods=(horizon,))[f"forward_return_{horizon}d"] + + # --- both alphas on the SAME processed panel ----------------------------- # + eq_alpha = EqualWeightAlpha() + eq_scores = _build_scores(processed, eq_alpha) + ic_alpha = _build_alpha(cfg) # config carries the ic_weighted params + ic_scores = _build_scores(processed, ic_alpha, fwd_col) + alpha_summary, weights_log = _alpha_disclosure(cfg, ic_alpha) + logger.info("alpha (ic_weighted leg): %s", alpha_summary) + + nav_eq = _run_backtest_for(cfg, panel, universe, eq_scores) + nav_ic = _run_backtest_for(cfg, panel, universe, ic_scores) + logger.info("backtests: equal_weight %d rows, ic_weighted %d rows", + len(nav_eq), len(nav_ic)) + + # --- subperiod statistics ------------------------------------------------ # + ppy = _periods_per_year(cfg.backtest.rebalance) + performance = { + "equal_weight": { + "train": subperiod_perf(nav_eq, end=split, periods_per_year=ppy), + "test": subperiod_perf(nav_eq, start=split, periods_per_year=ppy), + }, + "ic_weighted": { + "train": subperiod_perf(nav_ic, end=split, periods_per_year=ppy), + "test": subperiod_perf(nav_ic, start=split, periods_per_year=ppy), + }, + } + ic_series: dict[str, pd.Series] = { + name: compute_ic(factor_panel[name], fwd_col) for name in factor_panel.columns + } + ic_series["combo_equal_weight"] = compute_ic(eq_scores, fwd_col) + ic_series["combo_ic_weighted"] = compute_ic(ic_scores, fwd_col) + ic_stats = {name: ic_period_stats(s, split) for name, s in ic_series.items()} + consistency = {name: sign_consistent(stats) for name, stats in ic_stats.items()} + + # weight stability at the SETTLED rebalance dates of the ic run + settled = list(nav_ic.index) + if weights_log is not None and not weights_log.empty: + wanted = [d for d in settled if d in weights_log.index] + weights_at_reb = weights_log.loc[wanted] + n_scored = int(len(weights_log)) + n_fallback = int(weights_log["fallback"].sum()) + else: # pragma: no cover - ic leg always logs + weights_at_reb = pd.DataFrame() + n_scored = n_fallback = 0 + flips = weight_sign_flips(weights_at_reb) + reasons = fallback_reason_counts( + ic_alpha.fallback_log() if hasattr(ic_alpha, "fallback_log") else {} + ) + + dates = panel.index.get_level_values("date") + train_dates = sorted({d for d in dates if d < split}) + test_dates = sorted({d for d in dates if d >= split}) + result = OOSResult( + config=cfg, + elapsed_seconds=time.perf_counter() - t0, + split_date=split, + train_start=train_dates[0] if train_dates else None, + train_end=train_dates[-1] if train_dates else None, + test_start=test_dates[0] if test_dates else None, + test_end=test_dates[-1] if test_dates else None, + n_train_days=len(train_dates), + n_test_days=len(test_dates), + factor_names=tuple(f.name for f in factors), + performance=performance, + ic_stats=ic_stats, + sign_consistency=consistency, + weights_at_rebalances=weights_at_reb, + sign_flips=flips, + n_scored=n_scored, + n_fallback=n_fallback, + fallback_reasons=reasons, + alpha_summary=alpha_summary, + downgrades=_oos_downgrades(cfg), + report_path=Path(cfg.output.report_dir) / "phase3_oos_stability.md", + log_path=log_path, + ) + write_oos_stability_summary(result) + logger.info( + "phase3 oos done: eq test annual=%.4f ic test annual=%.4f report=%s (%.1fs)", + performance["equal_weight"]["test"].get("annual_return", float("nan")), + performance["ic_weighted"]["test"].get("annual_return", float("nan")), + result.report_path, result.elapsed_seconds, + ) + return result diff --git a/qt/reports.py b/qt/reports.py index e803b68..73e8cc7 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -631,6 +631,147 @@ def write_phase2_baseline_summary(result: "Phase2Result") -> Path: return target +# --------------------------------------------------------------------------- # +# Phase 3-3 OOS stability report (qt.oos_stability). +# --------------------------------------------------------------------------- # +def _oos_perf_table(performance: dict) -> str: + """Both models × both subperiods performance table.""" + header = ( + "| Model | Period | annual | vol | sharpe | maxDD | avg turnover | rebalances |\n" + "|---|---|---|---|---|---|---|---|\n" + ) + rows = "" + for model in ("equal_weight", "ic_weighted"): + for period in ("train", "test"): + p = (performance.get(model) or {}).get(period) or {} + rows += ( + f"| `{model}` | {period} | " + f"{_fmt(p.get('annual_return', float('nan')), pct=True)} | " + f"{_fmt(p.get('volatility', float('nan')), pct=True)} | " + f"{_fmt(p.get('sharpe', float('nan')))} | " + f"{_fmt(p.get('max_drawdown', float('nan')), pct=True)} | " + f"{_fmt(p.get('avg_turnover', float('nan')))} | " + f"{int(p.get('n_rebalances', 0))} |\n" + ) + return header + rows + + +def _oos_ic_table(ic_stats: dict, sign_consistency: dict) -> str: + """Per-series IC stability table (train + test rows, sign-consistency flag).""" + header = ( + "| Series | Period | IC mean | IC IR | hit rate | n | sign consistency |\n" + "|---|---|---|---|---|---|---|\n" + ) + rows = "" + for series, stats in ic_stats.items(): + flag = "YES" if sign_consistency.get(series) else "NO" + for period in ("train", "test"): + p = stats.get(period) or {} + rows += ( + f"| `{series}` | {period} | {_fmt(p.get('ic_mean', float('nan')))} | " + f"{_fmt(p.get('ic_ir', float('nan')))} | " + f"{_fmt(p.get('hit_rate', float('nan')), pct=True)} | " + f"{int(p.get('n', 0))} | " + f"{flag if period == 'train' else ''} |\n" + ) + return header + rows + + +def render_oos_stability(result) -> str: + """Build the phase3 OOS stability markdown (pure; no I/O, no secrets).""" + cfg = result.config + lines: list[str] = [] + lines.append("# Phase 3-3 — OOS Stability Validation (equal_weight vs ic_weighted)\n") + lines.append( + f"Project: **{cfg.project.name}** · source: **{cfg.data.source}** · " + f"ran in **{result.elapsed_seconds:.1f}s**\n" + ) + lines.append( + "\n> **This is NOT a return claim.** A small-sample stability check on one " + "index over two years: it compares the same three factors under " + "equal-weight vs walk-forward IC-weighted combination across a train/test " + "split. Subperiod metrics carry wide uncertainty.\n" + ) + + lines.append("\n## Split boundaries\n") + lines.append( + f"- split_date: **{_date_str(result.split_date)}** (train = " + f"[data.start, split), test = [split, data.end])\n" + f"- train: **{_date_str(result.train_start)} → {_date_str(result.train_end)}** " + f"({result.n_train_days} trading days)\n" + f"- test: **{_date_str(result.test_start)} → {_date_str(result.test_end)}** " + f"({result.n_test_days} trading days)\n" + f"- semantics: evaluation is WALK-FORWARD (rolling subperiod) — weights at " + f"any date d use only observations REALIZED by d (t + horizon <= d), so no " + f"test-period forward return reaches a train-period computation (locked by " + f"tests). The split is an accounting boundary for the statistics below.\n" + ) + + lines.append("\n## Config echo\n") + lines.append( + f"- universe: type=`{cfg.universe.type}`, index_code=`{cfg.universe.index_code}`, " + f"top_n=`{cfg.portfolio.top_n}`\n" + f"- factors: `{list(result.factor_names)}` · " + f"neutralize=`{cfg.processing.neutralize.enabled}`\n" + f"- ic_weighted params: {result.alpha_summary}\n" + ) + + lines.append("\n## Subperiod performance (same data, same rules, two alphas)\n") + lines.append(_oos_perf_table(result.performance)) + + lines.append("\n## IC stability (raw factors + both combo scores)\n") + lines.append( + "_hit rate = share of positive daily ICs; sign consistency = train and " + "test mean ICs share one nonzero sign._\n\n" + ) + lines.append(_oos_ic_table(result.ic_stats, result.sign_consistency)) + + lines.append("\n## Weight stability (ic_weighted)\n") + lines.append( + f"- scored dates: **{result.n_scored}** · equal-weight fallbacks: " + f"**{result.n_fallback}**\n" + ) + for reason, count in (result.fallback_reasons or {}).items(): + lines.append(f" - fallback reason ×{count}: {reason}\n") + flips = ", ".join(f"`{k}` {v}" for k, v in (result.sign_flips or {}).items()) + lines.append( + f"- sign flips between consecutive TRAINED rebalance weights " + f"(fallback rows excluded): {flips or '_n/a_'}\n\n" + ) + w = result.weights_at_rebalances + if w is not None and not w.empty: + factor_cols = [c for c in w.columns if c != "fallback"] + lines.append("Effective weights at each settled rebalance date " + "(ic_weighted leg):\n\n") + header = "| Date | period | " + " | ".join(f"`{c}`" for c in factor_cols) + header += " | fallback |\n|" + "---|" * (len(factor_cols) + 3) + "\n" + body = "" + for date, row in w.iterrows(): + period = "test" if pd.Timestamp(date) >= result.split_date else "train" + cells = " | ".join(_fmt(float(row[c])) for c in factor_cols) + body += ( + f"| {_date_str(date)} | {period} | {cells} | " + f"{'YES' if row['fallback'] else 'no'} |\n" + ) + lines.append(header + body) + + lines.append("\n## DOWNGRADES / caveats (INV-007 — must be disclosed)\n") + for item in result.downgrades: + lines.append(f"- {item}\n") + + lines.append("\n## Artifacts\n") + lines.append(f"- report: `{result.report_path}`\n- log: `{result.log_path}`\n") + return "".join(lines) + + +def write_oos_stability_summary(result) -> Path: + """Render and write the OOS stability report; return the path (SEC-003).""" + target = result.report_path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(render_oos_stability(result), encoding="utf-8") + return target + + # --------------------------------------------------------------------------- # # Repo-root delivery docs (framework spec §10). These describe the run; they # carry no secrets and are regenerable. diff --git a/tests/test_oos_stability.py b/tests/test_oos_stability.py new file mode 100644 index 0000000..81cf800 --- /dev/null +++ b/tests/test_oos_stability.py @@ -0,0 +1,276 @@ +"""P3-3: OOS stability validation (network-free). + +Locks the validation-layer contract: + * split boundary carries NO leakage: perturbing every forward return realized + AFTER the split cannot change any train-period date's weights (the + walk-forward property restated at the split boundary); + * subperiod statistics are computed strictly within their period; + * weight-stability diagnostics (sign flips on trained rows, fallback counts / + reasons) are correct on synthetic logs; + * the report disclosed the split dates, OOS metrics, weight stability, the + small-sample caveat, and leaks no secret; + * the OOS runner refuses the demo source and a config without an ``oos`` + section; the split date must fall inside the data window. +""" + +from __future__ import annotations + +import math +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest +import yaml + +from alpha.ic_weight import RollingICWeightAlpha +from qt.config import ConfigError, load_config +from qt.oos_stability import ( + fallback_reason_counts, + ic_period_stats, + sign_consistent, + subperiod_perf, + weight_sign_flips, +) + +_OOS_CONFIG = str( + Path(__file__).resolve().parents[1] / "config" / "phase3_real_oos_stability.yaml" +) + + +# --------------------------------------------------------------------------- # +# split-boundary no-leakage (the P3-3 red-line) +# --------------------------------------------------------------------------- # +def test_perturbing_post_split_returns_leaves_train_weights_unchanged(): + rng = np.random.default_rng(11) + dates = pd.bdate_range("2023-01-02", periods=120) + syms = [f"S{i}" for i in range(8)] + idx = pd.MultiIndex.from_product([dates, syms], names=["date", "symbol"]) + base = rng.normal(size=len(idx)) + factors = pd.DataFrame({"a": base, "b": rng.normal(size=len(idx))}, index=idx) + fwd = pd.Series(base, index=idx) + + split = dates[60] + h = 1 + model = RollingICWeightAlpha(window=30, min_periods=10, horizon=h) + model.fit(factors, fwd) + train_dates = [d for d in dates if d < split] + before = {d: model.weights_for(d) for d in train_dates} + + # poison EVERY forward return of factor dates on/after the split: none of + # them is realized before the split, so no train-period weight may move. + poisoned = fwd.copy() + mask = factors.index.get_level_values("date") >= split + poisoned[mask] = rng.normal(size=int(mask.sum())) * 1e3 + model2 = RollingICWeightAlpha(window=30, min_periods=10, horizon=h) + model2.fit(factors, poisoned) + for d in train_dates: + w_b, w_a = before[d], model2.weights_for(d) + if w_b is None: + assert w_a is None + else: + pd.testing.assert_series_equal(w_b, w_a) + + +# --------------------------------------------------------------------------- # +# subperiod statistics (pure functions) +# --------------------------------------------------------------------------- # +def _nav_table(): + dates = pd.to_datetime( + ["2023-01-31", "2023-02-28", "2023-03-31", "2023-04-28", "2023-05-31"] + ) + return pd.DataFrame( + { + "net_return": [0.01, -0.02, 0.03, 0.01, -0.01], + "turnover": [1.0, 0.4, 0.5, 0.3, 0.2], + "cost": [0.001] * 5, + "gross_return": [0.011, -0.019, 0.031, 0.011, -0.009], + "nav": (1 + pd.Series([0.01, -0.02, 0.03, 0.01, -0.01])).cumprod().values, + }, + index=pd.Index(dates, name="date"), + ) + + +def test_subperiod_perf_slices_strictly_by_period(): + nav = _nav_table() + split = pd.Timestamp("2023-03-01") + train = subperiod_perf(nav, end=split, periods_per_year=12) + test = subperiod_perf(nav, start=split, periods_per_year=12) + assert train["n_rebalances"] == 2 and test["n_rebalances"] == 3 + # turnover means are computed within the slice only + assert train["avg_turnover"] == pytest.approx((1.0 + 0.4) / 2) + assert test["avg_turnover"] == pytest.approx((0.5 + 0.3 + 0.2) / 3) + # the subperiod nav is rebased to 1.0 (period-local compounding) + assert train["annual_return"] != test["annual_return"] + for key in ("annual_return", "volatility", "sharpe", "max_drawdown"): + assert key in train and key in test + + +def test_subperiod_perf_empty_slice_is_nan_not_crash(): + nav = _nav_table() + out = subperiod_perf(nav, start=pd.Timestamp("2030-01-01"), periods_per_year=12) + assert out["n_rebalances"] == 0 + assert math.isnan(out["annual_return"]) + + +def test_ic_period_stats_mean_ir_hit_rate(): + dates = pd.bdate_range("2023-01-02", periods=10) + ic = pd.Series([0.1, 0.2, -0.1, 0.3, 0.1, -0.2, -0.3, -0.1, 0.2, np.nan], + index=dates) + split = dates[5] + stats = ic_period_stats(ic, split) + tr, te = stats["train"], stats["test"] + assert tr["n"] == 5 and te["n"] == 4 # NaN dropped from the test slice + assert tr["ic_mean"] == pytest.approx(np.mean([0.1, 0.2, -0.1, 0.3, 0.1])) + assert tr["hit_rate"] == pytest.approx(4 / 5) + assert te["hit_rate"] == pytest.approx(1 / 4) + assert math.isfinite(tr["ic_ir"]) and math.isfinite(te["ic_ir"]) + + +def test_sign_consistent_requires_same_nonzero_sign(): + assert sign_consistent({"train": {"ic_mean": 0.02}, "test": {"ic_mean": 0.01}}) + assert sign_consistent({"train": {"ic_mean": -0.02}, "test": {"ic_mean": -0.01}}) + assert not sign_consistent({"train": {"ic_mean": 0.02}, "test": {"ic_mean": -0.01}}) + assert not sign_consistent({"train": {"ic_mean": float("nan")}, + "test": {"ic_mean": 0.01}}) + + +# --------------------------------------------------------------------------- # +# weight stability diagnostics +# --------------------------------------------------------------------------- # +def _weights_log(): + dates = pd.to_datetime( + ["2023-01-31", "2023-02-28", "2023-03-31", "2023-04-28", "2023-05-31"] + ) + return pd.DataFrame( + { + "a": [0.33, 0.5, -0.4, 0.6, -0.2], # trained signs: +, -, +, - -> 3 flips + "b": [0.33, -0.5, -0.6, -0.4, -0.8], # trained signs: -, -, -, - -> 0 flips + "fallback": [True, False, False, False, False], + }, + index=pd.Index(dates, name="date"), + ) + + +def test_weight_sign_flips_counts_trained_rows_only(): + flips = weight_sign_flips(_weights_log()) + # the fallback row (all-positive equal weights) must NOT inject flips + assert flips == {"a": 3, "b": 0} + + +def test_fallback_reason_counts_aggregates(): + log = { + pd.Timestamp("2023-01-31"): "insufficient realized IC history", + pd.Timestamp("2023-02-28"): None, + pd.Timestamp("2023-03-31"): "insufficient realized IC history", + } + counts = fallback_reason_counts(log) + assert counts == {"insufficient realized IC history": 2} + + +# --------------------------------------------------------------------------- # +# config + guards +# --------------------------------------------------------------------------- # +def test_oos_config_validates_with_split_inside_window(): + cfg = load_config(_OOS_CONFIG) + assert cfg.oos is not None + split = pd.Timestamp(cfg.oos.split_date) + assert pd.Timestamp(cfg.data.start) < split < pd.Timestamp(cfg.data.end) + assert cfg.alpha.model == "ic_weighted" # carries the ic params for the run + + +def test_split_outside_window_is_a_config_error(tmp_path): + raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) + raw["oos"]["split_date"] = "2031-01-01" + p = tmp_path / "bad.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + with pytest.raises(ConfigError, match="split_date"): + load_config(str(p)) + + +def test_oos_runner_rejects_demo_source(example_config_path): + from qt.oos_stability import run_phase3_oos + + with pytest.raises(ValueError, match="tushare|REAL"): + run_phase3_oos(example_config_path) + + +def test_oos_runner_requires_oos_section(tmp_path): + raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) + raw.pop("oos") + p = tmp_path / "no_oos.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + from qt.oos_stability import run_phase3_oos + + with pytest.raises(ValueError, match="oos"): + run_phase3_oos(str(p)) + + +# --------------------------------------------------------------------------- # +# report rendering +# --------------------------------------------------------------------------- # +def _synthetic_oos_result(): + from qt.oos_stability import OOSResult + + cfg = load_config(_OOS_CONFIG) + period = {"annual_return": -0.05, "volatility": 0.15, "sharpe": -0.3, + "max_drawdown": -0.12, "avg_turnover": 0.8, "n_rebalances": 11} + ic_p = {"ic_mean": 0.01, "ic_ir": 0.05, "hit_rate": 0.55, "n": 110} + return OOSResult( + config=cfg, + elapsed_seconds=100.0, + split_date=pd.Timestamp("2023-07-01"), + train_start=pd.Timestamp("2022-07-01"), train_end=pd.Timestamp("2023-06-30"), + test_start=pd.Timestamp("2023-07-03"), test_end=pd.Timestamp("2024-06-28"), + n_train_days=240, n_test_days=238, + factor_names=("momentum_20", "roe", "netprofit_yoy"), + performance={ + "equal_weight": {"train": dict(period), "test": dict(period)}, + "ic_weighted": {"train": dict(period), "test": dict(period)}, + }, + ic_stats={ + "momentum_20": {"train": dict(ic_p), "test": dict(ic_p)}, + "combo_equal_weight": {"train": dict(ic_p), "test": dict(ic_p)}, + "combo_ic_weighted": {"train": dict(ic_p), "test": dict(ic_p)}, + }, + sign_consistency={"momentum_20": True, "combo_equal_weight": True, + "combo_ic_weighted": False}, + weights_at_rebalances=_weights_log(), + sign_flips={"a": 3, "b": 0}, + n_scored=221, n_fallback=20, + fallback_reasons={"insufficient realized IC history": 20}, + alpha_summary={"model": "ic_weighted", "window": 60, "min_periods": 20, + "horizon": 1, "mode": "rolling"}, + downgrades=("DATA PATH = REAL tushare: ...",), + report_path=Path("artifacts/reports/phase3_oos_stability.md"), + log_path=Path("artifacts/logs/run_phase3_oos.log"), + ) + + +def test_render_oos_report_disclosed_boundaries_and_metrics(): + from qt.oos_stability import render_oos_stability + + md = render_oos_stability(_synthetic_oos_result()) + # split boundaries written into the report + assert "2023-07-01" in md and "2022-07-01" in md and "2024-06-28" in md + assert "train" in md.lower() and "test" in md.lower() + # walk-forward boundary semantics stated + assert "realized" in md.lower() + # both models' subperiod performance + IC stability + weight stability + assert "equal_weight" in md and "ic_weighted" in md + assert "hit rate" in md.lower() + assert "sign consistency" in md.lower() or "sign-consistency" in md.lower() + assert "sign flip" in md.lower() + assert "fallback" in md.lower() + # the caveat: a stability CHECK, not a return claim + assert "not a" in md.lower() and "claim" in md.lower() + assert "small-sample" in md.lower() or "small sample" in md.lower() + + +def test_render_oos_report_leaks_no_secret(): + from qt.oos_stability import render_oos_stability + + result = _synthetic_oos_result() + md = render_oos_stability(result) + assert result.config.data.external_secret_file not in md + assert "token" not in md.lower() From 0035a04b300fe895d2539af23efd4b3bd319a910 Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Wed, 10 Jun 2026 17:25:35 +0800 Subject: [PATCH 2/5] fix(oos): holding-window subperiod slicing + ic_weighted leg guard Review HIGH fixes on PR #12: 1. Subperiod performance was sliced by the nav row's SIGNAL date, but a row's return covers the holding window [rebalance, next rebalance] -- with split=2023-07-01 the 2023-06-30 row (held into July) credited test-period return to train. Now split_nav_by_holding assigns train <=> holding END on/before the split, test <=> holding START on/after it; straddling rows (incl. the last row, whose end is the unknown terminal candidate) are EXCLUDED from both and disclosed in the report. IC stats are likewise sliced by the realization date (t + horizon), not the factor date alone. Real impact (rerun, 929s): train eq -6.81% -> -11.92%, ic -1.69% -> -8.31% (the old train was polluted by post-split July returns); test unchanged (-5.27% / -2.70%); IC sign-flip finding unchanged (all NO). 2. run-phase3-oos now REQUIRES alpha.model='ic_weighted': any other model would silently run equal_weight twice and label one leg ic_weighted (a fake comparison). Readable ValueError + network-free test. Tests: 285 passed (was 282): +2 holding-window slicing (straddler + unknown -end), +1 fake-comparison guard; ic_period_stats test rewritten for realization-date slicing; render test asserts the straddler disclosure. Docs (RUNBOOK/TEST_REPORT/CLAUDE/AGENTS) updated with the new semantics and corrected real numbers. --- AGENTS.md | 8 +-- CLAUDE.md | 8 +-- RUNBOOK.md | 14 +++- TEST_REPORT.md | 20 +++--- qt/oos_stability.py | 139 +++++++++++++++++++++++++++--------- qt/reports.py | 13 +++- tests/test_oos_stability.py | 89 ++++++++++++++++++----- 7 files changed, 219 insertions(+), 72 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fb1f165..9b48c82 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -98,9 +98,9 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - `config/phase3_real_ic_weighted.yaml` 与 phase3_real_multifactor 唯一差异 alpha.model(直接可比)。 - 真实结果:annual **−3.57%**(等权 −9.05%),训练覆盖 201/221(20 fallback 全在窗口攒满前)。⚠️ 优于等权非业绩声明——单年窗口+权重逐期翻号即小样本不稳定,照实披露。回归不破:等权 rerun −9.05%/0.0083 不变,demo 0.96/0.84 不变。 - 🔧 **Phase 3-3 OOS 稳定性验证**（`p3-oos-stability-validation` 分支,代劳待验收）：报告型验证层（`run-phase3-oos` + `qt/oos_stability.py` + `config/phase3_real_oos_stability.yaml`,SSE50 2 年,split 2023-07-01）;不加 alpha 复杂度、不改 portfolio/execution/factor math。 - - 一次数据加载、两次回测(equal_weight vs ic_weighted),诊断按 split 切段(子段 nav 重归一)。walk-forward 边界测试锁定:扰动 split 后全部 fwd,train 期权重逐 bit 不变。 - - 报告:split 边界/分期绩效/逐序列 IC(mean/IR/hit/sign consistency)/权重稳定性(sign flips、fallback+原因)/小样本 caveat。 - - **真实关键发现**:三因子 train→test IC **全部翻号**(sign consistency 全 NO),hit 46~53%;权重 sign flips 7/3/4;eq −6.81%/−5.27%,ic −1.69%/−2.70%(train/test)。**P3-2 单年跑赢不可外推——这正是验证层要拿到的证据;非收益声明。**回归不破:eq −9.05%/ic −3.57%/demo 0.96/0.84 全不变;报告 secret scan 干净。 -- ✅ 当前质量门：`pytest -p no:cacheprovider` **282 passed**；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）OK；`run-phase0`（demo）OK。 + - 一次数据加载、两次回测(equal_weight vs ic_weighted)。walk-forward 边界测试锁定:扰动 split 后全部 fwd,train 期权重逐 bit 不变。**绩效按持有窗口切片**(train 持有 end≤split、test start≥split,跨界调仓排除并披露;IC 按实现日 t+h 切;review HIGH 修复——旧 signal-date 切法把跨界收益记进 train);runner 强制 alpha.model=ic_weighted(防假对比)。 + - 报告:split 边界+跨界行/分期绩效/逐序列 IC(mean/IR/hit/sign consistency)/权重稳定性(sign flips、fallback+原因)/小样本 caveat。 + - **真实关键发现**:三因子 train→test IC **全部翻号**(sign consistency 全 NO),hit 46~53%;权重 sign flips 7/3/4;eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(修切片前 train 被污染到 −6.81%/−1.69%)。**P3-2 单年跑赢不可外推——这正是验证层要拿到的证据;非收益声明。**回归不破:eq −9.05%/ic −3.57%/demo 0.96/0.84 全不变;报告 secret scan 干净。 +- ✅ 当前质量门：`pytest -p no:cacheprovider` **285 passed**；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）OK；`run-phase0`（demo）OK。 - ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、因子 IC 小样本不稳定（P3-3 实证）。 - 路线图下一步：更长历史/更宽 universe 的稳定性复检,或分钟级（architecture.html §11）。 diff --git a/CLAUDE.md b/CLAUDE.md index 8110bbb..5970b5a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -103,10 +103,10 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变（测试锁定）。 - 🔧 **Phase 3-3 OOS 稳定性验证**（`p3-oos-stability-validation` 分支,代劳待验收）：**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`（`qt/oos_stability.py`）+ `config/phase3_real_oos_stability.yaml`（SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照）。 - **一次数据加载、同一 processed 因子面板、两次回测**（equal_weight vs ic_weighted）;所有诊断按 split 切段（子段 nav 重新归一,绝不跨段串味）。 - - **边界语义（测试锁定）**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。 - - 报告 `phase3_oos_stability.md`:split 边界/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 - - **真实结果（关键发现,~16min,77 成分/2 年）**:三个原始因子 train→test **IC 全部翻号**(momentum −0.024→+0.006 / roe −0.029→+0.007 / np_yoy −0.010→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −6.81%/test −5.27%,ic train −1.69%/test −2.70%。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** + - **边界语义（测试锁定）**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。**绩效切片按持有窗口**(train 行持有期 end≤split、test 行 start≥split,跨界调仓从两段排除并披露;IC 按实现日 t+h 切)——绝不按 signal date 单切(review HIGH 修复:旧切法把跨界持有期的 test 收益记进 train);runner 强制 `alpha.model: ic_weighted`(否则假对比,可读报错)。 + - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 + - **真实结果（关键发现,~16min,77 成分/2 年,持有窗口切片）**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 -- ✅ 质量门：`pytest` **282 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=13）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）+ `run-phase0`（demo）均 OK。 +- ✅ 质量门：`pytest` **285 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）+ `run-phase0`（demo）均 OK。 - ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、因子 IC 小样本不稳定（P3-3 实证）。 - 路线图下一步：更长历史/更宽 universe 的稳定性复检,或分钟级（architecture.html §11）。 diff --git a/RUNBOOK.md b/RUNBOOK.md index 9a87c5f..f269429 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -287,8 +287,18 @@ Split semantics (locked by tests): post-split forward return cannot change any train-period date's weights (split-boundary no-leakage test). Freezing weights at the split is NOT used (that would be a new alpha mode; P3-3 adds no alpha complexity). -- The config's `alpha` section carries the ic_weighted params; the run ALSO - evaluates `equal_weight` on the same data as the control. +- **Performance slicing is HOLDING-WINDOW aware**: a nav row is indexed by its + rebalance (signal) date but its return covers [that rebalance, the next one] — + so train rows must have their holding END on/before the split, test rows their + holding START on/after it, and a straddling rebalance is EXCLUDED from both + subperiods and disclosed in the report. IC stats are sliced by the realization + date (`t + horizon`) the same way. The test subperiod therefore starts with + the first post-split holding period — the same place the 1-year phase3 + baselines start. +- The config's `alpha` section carries the ic_weighted params and MUST set + `alpha.model: ic_weighted` (guarded: any other model is refused — running + equal_weight twice and labelling one leg ic_weighted would be a fake + comparison); the `equal_weight` control leg is built internally. Report (`artifacts/reports/phase3_oos_stability.md`): split boundaries; per-subperiod performance for both models (annual / vol / Sharpe / maxDD / diff --git a/TEST_REPORT.md b/TEST_REPORT.md index 07c98a1..4e1874a 100644 --- a/TEST_REPORT.md +++ b/TEST_REPORT.md @@ -20,12 +20,12 @@ Run from the repo root with the project python (env `quant_mf`): | Gate | Command | Result | |---|---|---| -| Unit + integration | `pytest -q` | **282 passed, 0 failed** | +| Unit + integration | `pytest -q` | **285 passed, 0 failed** | | Lint | `ruff check .` | **All checks passed** | | Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`) | exit `0`, prints `OK` | | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | -Counts below are the actual per-file `pytest` numbers (sum = 282). +Counts below are the actual per-file `pytest` numbers (sum = 285). ## Per-file breakdown — Phase 0 core (97) @@ -111,12 +111,12 @@ Counts below are the actual per-file `pytest` numbers (sum = 282). | `test_ic_weight_alpha.py` | 12 | **lookahead red-line**: perturbing unrealized forward returns cannot change weights; exact `t + h <= d` realization cutoff (min_periods boundary); insufficient-history equal-weight fallback (== EqualWeightAlpha row mean); single-factor degeneration to ±1; L1 normalization + sign preservation; degenerate-IC fallback; rolling-vs-expanding window; fit requires forward_returns; dated-cross-section contract; input immutability; weights/fallback log | | `test_ic_alpha_pipeline.py` | 6 | alpha dispatch by config (equal_weight / ic_weighted + params / unknown = ConfigError), equal-weight default keeps exact demo numbers (ic 0.96 / annual 0.84) + report line, ic_weighted demo e2e (summary, weights log, early-fallback→late-trained, L1 rows, report disclosure, no secret), ic-weights differ from equal weight on diverging-IC synthetic data | -## Per-file breakdown — Phase 3-3 OOS stability (13) +## Per-file breakdown — Phase 3-3 OOS stability (16) | Test file | Tests | Red-line / feature | |---|---|---| -| `test_oos_stability.py` | 13 | **split-boundary no-leakage**: perturbing every post-split forward return leaves all train-period weights bit-identical; `subperiod_perf` slices strictly + rebased nav + empty-slice NaN; `ic_period_stats` mean/IR/hit-rate/n; `sign_consistent` nonzero-same-sign; `weight_sign_flips` on trained rows only (fallback rows excluded); fallback-reason aggregation; OOS config validates + split-inside-window ConfigError; runner rejects demo source + missing `oos` section; report renders boundaries / OOS metrics / weight stability / caveat / no secret | -| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3)** | **282** | | +| `test_oos_stability.py` | 16 | **split-boundary no-leakage**: perturbing every post-split forward return leaves all train-period weights bit-identical; **holding-window slicing** (`split_nav_by_holding`: train = holding end ≤ split, test = start ≥ split, straddlers + unknown-end rows excluded and disclosed); `subperiod_perf` rebased nav + empty-slice NaN; `ic_period_stats` sliced by realization date (t+h) with mean/IR/hit-rate/n; `sign_consistent` nonzero-same-sign; `weight_sign_flips` on trained rows only (fallback rows excluded); fallback-reason aggregation; OOS config validates + split-inside-window ConfigError; runner rejects demo source / missing `oos` section / **non-ic_weighted alpha (fake-comparison guard)**; report renders boundaries / straddler disclosure / OOS metrics / weight stability / caveat / no secret | +| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3)** | **285** | | ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) @@ -191,10 +191,14 @@ Counts below are the actual per-file `pytest` numbers (sum = 282). demo numbers (ic 0.96 / annual 0.84) are locked unchanged. - **P3-3 OOS stability (locked by tests):** `run-phase3-oos` is a REPORT-ONLY validation layer — one shared data load, two backtests (equal_weight vs - ic_weighted), every diagnostic split at `oos.split_date` (train strictly - before, test on/after; subperiod navs rebased so nothing bleeds across). + ic_weighted), every diagnostic split at `oos.split_date`. Performance slicing + is HOLDING-WINDOW aware (a nav row's return covers [rebalance, next + rebalance]: train rows END on/before the split, test rows START on/after it, + straddlers are excluded from both and disclosed); IC stats slice by the + realization date (t+h); subperiod navs are rebased so nothing bleeds across. Evaluation is walk-forward (rolling subperiod): the split-boundary test - proves post-split forward returns cannot move any train-period weight. + proves post-split forward returns cannot move any train-period weight. The + runner refuses a non-ic_weighted `alpha.model` (a fake comparison guard). Portfolio / execution / factor math are untouched. - A duplicate test-function name across two files was found and renamed during P2-2 (it had been silently shadowing one test in the full-suite run). A second, harmless diff --git a/qt/oos_stability.py b/qt/oos_stability.py index e2aa669..70a7489 100644 --- a/qt/oos_stability.py +++ b/qt/oos_stability.py @@ -59,8 +59,8 @@ from runtime.backtest.sim_execution import SimExecution __all__ = ["OOSResult", "run_phase3_oos", "render_oos_stability", - "subperiod_perf", "ic_period_stats", "sign_consistent", - "weight_sign_flips", "fallback_reason_counts"] + "split_nav_by_holding", "subperiod_perf", "ic_period_stats", + "sign_consistent", "weight_sign_flips", "fallback_reason_counts"] _LOGGER_NAME = "qt.run_phase3_oos" _PERF_NAN = { @@ -84,6 +84,9 @@ class OOSResult: test_end: pd.Timestamp | None n_train_days: int n_test_days: int + # rebalance dates whose HOLDING WINDOW straddles the split (excluded from + # both subperiods' performance and disclosed in the report). + boundary_dates: tuple[pd.Timestamp, ...] factor_names: tuple[str, ...] # performance[model][period] -> {annual_return, volatility, sharpe, # max_drawdown, avg_turnover, n_rebalances}; model in (equal_weight, @@ -109,47 +112,93 @@ class OOSResult: # --------------------------------------------------------------------------- # # Pure subperiod statistics (network-free; unit-tested with synthetic inputs). # --------------------------------------------------------------------------- # -def subperiod_perf( - nav_table: pd.DataFrame, - start: pd.Timestamp | None = None, - end: pd.Timestamp | None = None, - periods_per_year: int = 12, -) -> dict: - """Performance over the nav rows in [start, end) — period-local compounding. - - The subperiod nav is REBASED to 1.0 from the slice's own net returns, so a - drawdown or annualization never bleeds across the split. ``end`` is - exclusive (a rebalance ON the split date belongs to the test period). - Empty slices return NaN metrics (never crash). +def split_nav_by_holding( + nav_table: pd.DataFrame, split: pd.Timestamp +) -> tuple[pd.DataFrame, pd.DataFrame, list[pd.Timestamp]]: + """(train_rows, test_rows, boundary_dates) sliced by the HOLDING WINDOW. + + A driver nav row is INDEXED by its rebalance (signal) date, but its return + covers the holding window [index[i], index[i+1]] — slicing by the row index + alone would credit a straddling holding period's post-split return to the + train period. Calendar-pure subperiods therefore require: + + train <=> holding END on/before the split (return fully realized + pre-split); + test <=> holding START on/after the split; + boundary <=> the holding window straddles the split (start < split < end, + or the end is unknown — the LAST row's end is the skipped + terminal candidate, not recoverable here — with a pre-split + start). Boundary rows are EXCLUDED from both subperiods and + disclosed in the report. """ if nav_table is None or nav_table.empty: + empty = nav_table if nav_table is not None else pd.DataFrame() + return empty, empty, [] + split = pd.Timestamp(split) + starts = [pd.Timestamp(d) for d in nav_table.index] + ends: list[pd.Timestamp | None] = starts[1:] + [None] + train_mask: list[bool] = [] + test_mask: list[bool] = [] + boundary: list[pd.Timestamp] = [] + for start, end in zip(starts, ends): + if start >= split: + train_mask.append(False) + test_mask.append(True) + elif end is not None and end <= split: + train_mask.append(True) + test_mask.append(False) + else: # straddles the split, or unknown end with a pre-split start + train_mask.append(False) + test_mask.append(False) + boundary.append(start) + return nav_table[train_mask], nav_table[test_mask], boundary + + +def subperiod_perf(nav_slice: pd.DataFrame, periods_per_year: int = 12) -> dict: + """Performance of one pre-sliced nav segment — period-local compounding. + + The segment nav is REBASED to 1.0 from its own net returns, so a drawdown + or annualization never bleeds across the split. Empty slices return NaN + metrics (never crash). Slicing itself lives in + :func:`split_nav_by_holding` (holding-window aware). + """ + if nav_slice is None or nav_slice.empty: return dict(_PERF_NAN) - sliced = nav_table - if start is not None: - sliced = sliced[sliced.index >= pd.Timestamp(start)] - if end is not None: - sliced = sliced[sliced.index < pd.Timestamp(end)] - if sliced.empty: - return dict(_PERF_NAN) - local_nav = (1.0 + sliced["net_return"]).cumprod() + local_nav = (1.0 + nav_slice["net_return"]).cumprod() perf = performance_summary(local_nav, periods_per_year=periods_per_year) return { **perf, - "avg_turnover": float(sliced["turnover"].mean()), - "n_rebalances": int(len(sliced)), + "avg_turnover": float(nav_slice["turnover"].mean()), + "n_rebalances": int(len(nav_slice)), } -def ic_period_stats(ic: pd.Series, split: pd.Timestamp) -> dict[str, dict]: +def ic_period_stats( + ic: pd.Series, split: pd.Timestamp, horizon: int = 1 +) -> dict[str, dict]: """{'train': {...}, 'test': {...}} stats of a per-date IC series. + The IC at factor date t uses the h-day forward return REALIZED at trading + position pos(t) + h, so (mirroring the nav slicing): + train <=> realization date strictly BEFORE the split; + test <=> factor date on/after the split; + straddlers (t < split <= realization) are excluded from both. Each period reports the NaN-dropped mean, IR (mean/std, ddof=1), hit rate - (share of positive ICs) and observation count. Train is strictly before the - split; test is on/after it. + (share of positive ICs) and observation count. """ split = pd.Timestamp(split) + idx = list(ic.index) + n_idx = len(idx) + realization = [ + idx[pos + horizon] if pos + horizon < n_idx else None + for pos in range(n_idx) + ] + train_mask = [ + r is not None and pd.Timestamp(r) < split for r in realization + ] + test_mask = [pd.Timestamp(t) >= split for t in idx] out: dict[str, dict] = {} - for name, mask in (("train", ic.index < split), ("test", ic.index >= split)): + for name, mask in (("train", train_mask), ("test", test_mask)): clean = ic[mask].dropna() n = int(len(clean)) mean = float(clean.mean()) if n else float("nan") @@ -207,7 +256,11 @@ def _oos_downgrades(cfg: RootConfig) -> tuple[str, ...]: "for subperiod statistics. Weight training is walk-forward (a pair enters " "date d's weights only once realized, t + horizon <= d), so no test-period " "forward return reaches any train-period computation; freezing weights at " - "the split is NOT used (that would be a new alpha mode, out of scope).", + "the split is NOT used (that would be a new alpha mode, out of scope). " + "Subperiod PERFORMANCE is sliced by the HOLDING WINDOW (train rows end " + "on/before the split, test rows start on/after it; a straddling rebalance " + "is excluded from both and disclosed) and IC stats by the realization " + "date — never by the row's signal date alone.", "This is a SMALL-SAMPLE stability check (one index, two years, ~22 " "rebalances), NOT a return claim and NOT a tuned result: subperiod " "metrics carry wide uncertainty and must not be read as expected " @@ -254,6 +307,14 @@ def run_phase3_oos(config_path: str) -> OOSResult: "run-phase3-oos requires an 'oos' config section with split_date " "(train = [data.start, split), test = [split, data.end])." ) + if cfg.alpha.model != "ic_weighted": + raise ValueError( + "run-phase3-oos compares equal_weight vs ic_weighted: the config's " + "alpha section must set model='ic_weighted' (it carries the " + "ic-weighted leg's params; the equal-weight control is built " + f"internally). Got alpha.model={cfg.alpha.model!r} — running that " + "would silently label an equal_weight leg as ic_weighted." + ) split = pd.Timestamp(cfg.oos.split_date) t0 = time.perf_counter() @@ -287,16 +348,20 @@ def run_phase3_oos(config_path: str) -> OOSResult: logger.info("backtests: equal_weight %d rows, ic_weighted %d rows", len(nav_eq), len(nav_ic)) - # --- subperiod statistics ------------------------------------------------ # + # --- subperiod statistics (holding-window aware slicing) ------------------ # ppy = _periods_per_year(cfg.backtest.rebalance) + eq_train, eq_test, eq_boundary = split_nav_by_holding(nav_eq, split) + ic_train, ic_test, ic_boundary = split_nav_by_holding(nav_ic, split) + # both legs share the driver's rebalance calendar -> identical boundary rows + boundary_dates = tuple(sorted(set(eq_boundary) | set(ic_boundary))) performance = { "equal_weight": { - "train": subperiod_perf(nav_eq, end=split, periods_per_year=ppy), - "test": subperiod_perf(nav_eq, start=split, periods_per_year=ppy), + "train": subperiod_perf(eq_train, periods_per_year=ppy), + "test": subperiod_perf(eq_test, periods_per_year=ppy), }, "ic_weighted": { - "train": subperiod_perf(nav_ic, end=split, periods_per_year=ppy), - "test": subperiod_perf(nav_ic, start=split, periods_per_year=ppy), + "train": subperiod_perf(ic_train, periods_per_year=ppy), + "test": subperiod_perf(ic_test, periods_per_year=ppy), }, } ic_series: dict[str, pd.Series] = { @@ -304,7 +369,10 @@ def run_phase3_oos(config_path: str) -> OOSResult: } ic_series["combo_equal_weight"] = compute_ic(eq_scores, fwd_col) ic_series["combo_ic_weighted"] = compute_ic(ic_scores, fwd_col) - ic_stats = {name: ic_period_stats(s, split) for name, s in ic_series.items()} + ic_stats = { + name: ic_period_stats(s, split, horizon=horizon) + for name, s in ic_series.items() + } consistency = {name: sign_consistent(stats) for name, stats in ic_stats.items()} # weight stability at the SETTLED rebalance dates of the ic run @@ -335,6 +403,7 @@ def run_phase3_oos(config_path: str) -> OOSResult: test_end=test_dates[-1] if test_dates else None, n_train_days=len(train_dates), n_test_days=len(test_dates), + boundary_dates=boundary_dates, factor_names=tuple(f.name for f in factors), performance=performance, ic_stats=ic_stats, diff --git a/qt/reports.py b/qt/reports.py index 73e8cc7..f072855 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -705,7 +705,18 @@ def render_oos_stability(result) -> str: f"any date d use only observations REALIZED by d (t + horizon <= d), so no " f"test-period forward return reaches a train-period computation (locked by " f"tests). The split is an accounting boundary for the statistics below.\n" - ) + f"- performance slicing is HOLDING-WINDOW aware: a rebalance row's return " + f"covers [that rebalance, the next one], so train rows must have their " + f"holding END on/before the split and test rows their holding START on/" + f"after it. IC stats are sliced by the realization date (t + horizon) the " + f"same way.\n" + ) + if result.boundary_dates: + bd = ", ".join(_date_str(d) for d in result.boundary_dates) + lines.append( + f"- straddling rebalance(s) excluded from BOTH subperiods (holding " + f"window crosses the split): {bd}\n" + ) lines.append("\n## Config echo\n") lines.append( diff --git a/tests/test_oos_stability.py b/tests/test_oos_stability.py index 81cf800..9bfc191 100644 --- a/tests/test_oos_stability.py +++ b/tests/test_oos_stability.py @@ -29,6 +29,7 @@ fallback_reason_counts, ic_period_stats, sign_consistent, + split_nav_by_holding, subperiod_perf, weight_sign_flips, ) @@ -91,38 +92,74 @@ def _nav_table(): ) -def test_subperiod_perf_slices_strictly_by_period(): +def test_split_nav_by_holding_assigns_rows_by_holding_window(): + """A row's return covers [index[i], index[i+1]] — the split must respect it. + + With split 2023-03-01: the 2023-01-31 row (held to 2023-02-28) is fully + pre-split -> train; the 2023-02-28 row is held to 2023-03-31, STRADDLING the + split -> excluded from BOTH and disclosed; rows starting on/after the split + -> test. The last row's end is unknown (terminal candidate) but it starts + post-split here, so it is test. + """ + nav = _nav_table() + train, test, boundary = split_nav_by_holding(nav, pd.Timestamp("2023-03-01")) + assert list(train.index) == [pd.Timestamp("2023-01-31")] + assert list(test.index) == [ + pd.Timestamp("2023-03-31"), pd.Timestamp("2023-04-28"), + pd.Timestamp("2023-05-31"), + ] + assert boundary == [pd.Timestamp("2023-02-28")] # straddler disclosed + + +def test_split_nav_by_holding_unknown_end_prestart_is_boundary(): + # split AFTER the last row's start: the last row's holding end is unknown + # (terminal candidate), so it cannot be proven pre-split -> boundary. nav = _nav_table() - split = pd.Timestamp("2023-03-01") - train = subperiod_perf(nav, end=split, periods_per_year=12) - test = subperiod_perf(nav, start=split, periods_per_year=12) - assert train["n_rebalances"] == 2 and test["n_rebalances"] == 3 - # turnover means are computed within the slice only - assert train["avg_turnover"] == pytest.approx((1.0 + 0.4) / 2) - assert test["avg_turnover"] == pytest.approx((0.5 + 0.3 + 0.2) / 3) - # the subperiod nav is rebased to 1.0 (period-local compounding) - assert train["annual_return"] != test["annual_return"] + train, test, boundary = split_nav_by_holding(nav, pd.Timestamp("2023-06-15")) + assert pd.Timestamp("2023-05-31") in boundary + assert len(test) == 0 + assert list(train.index) == [ + pd.Timestamp("2023-01-31"), pd.Timestamp("2023-02-28"), + pd.Timestamp("2023-03-31"), pd.Timestamp("2023-04-28"), + ] + + +def test_subperiod_perf_stats_within_slice_only(): + nav = _nav_table() + train, test, _ = split_nav_by_holding(nav, pd.Timestamp("2023-03-01")) + tr = subperiod_perf(train, periods_per_year=12) + te = subperiod_perf(test, periods_per_year=12) + assert tr["n_rebalances"] == 1 and te["n_rebalances"] == 3 + assert tr["avg_turnover"] == pytest.approx(1.0) + assert te["avg_turnover"] == pytest.approx((0.5 + 0.3 + 0.2) / 3) for key in ("annual_return", "volatility", "sharpe", "max_drawdown"): - assert key in train and key in test + assert key in tr and key in te def test_subperiod_perf_empty_slice_is_nan_not_crash(): nav = _nav_table() - out = subperiod_perf(nav, start=pd.Timestamp("2030-01-01"), periods_per_year=12) + out = subperiod_perf(nav.iloc[0:0], periods_per_year=12) assert out["n_rebalances"] == 0 assert math.isnan(out["annual_return"]) -def test_ic_period_stats_mean_ir_hit_rate(): +def test_ic_period_stats_splits_by_realization_date(): + """IC at factor date t realizes at t+h: train requires realization < split. + + 10 trading days, h=1, split = dates[5]: ICs of t in dates[0..3] realize at + dates[1..4] (< split) -> train; t = dates[4] realizes ON dates[5] (== split, + not strictly before) -> excluded (boundary); t >= dates[5] -> test. + """ dates = pd.bdate_range("2023-01-02", periods=10) ic = pd.Series([0.1, 0.2, -0.1, 0.3, 0.1, -0.2, -0.3, -0.1, 0.2, np.nan], index=dates) split = dates[5] - stats = ic_period_stats(ic, split) + stats = ic_period_stats(ic, split, horizon=1) tr, te = stats["train"], stats["test"] - assert tr["n"] == 5 and te["n"] == 4 # NaN dropped from the test slice - assert tr["ic_mean"] == pytest.approx(np.mean([0.1, 0.2, -0.1, 0.3, 0.1])) - assert tr["hit_rate"] == pytest.approx(4 / 5) + assert tr["n"] == 4 # dates[0..3] only; dates[4] straddles -> excluded + assert te["n"] == 4 # dates[5..8]; dates[9] is NaN -> dropped + assert tr["ic_mean"] == pytest.approx(np.mean([0.1, 0.2, -0.1, 0.3])) + assert tr["hit_rate"] == pytest.approx(3 / 4) assert te["hit_rate"] == pytest.approx(1 / 4) assert math.isfinite(tr["ic_ir"]) and math.isfinite(te["ic_ir"]) @@ -195,6 +232,19 @@ def test_oos_runner_rejects_demo_source(example_config_path): run_phase3_oos(example_config_path) +def test_oos_runner_rejects_non_ic_weighted_alpha(tmp_path): + """alpha.model=equal_weight would silently produce a FAKE comparison + (equal_weight vs equal_weight labelled ic_weighted) — must be refused.""" + raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) + raw["alpha"] = {"model": "equal_weight", "params": {}} + p = tmp_path / "eq_alpha.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + from qt.oos_stability import run_phase3_oos + + with pytest.raises(ValueError, match="ic_weighted"): + run_phase3_oos(str(p)) + + def test_oos_runner_requires_oos_section(tmp_path): raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) raw.pop("oos") @@ -223,6 +273,7 @@ def _synthetic_oos_result(): train_start=pd.Timestamp("2022-07-01"), train_end=pd.Timestamp("2023-06-30"), test_start=pd.Timestamp("2023-07-03"), test_end=pd.Timestamp("2024-06-28"), n_train_days=240, n_test_days=238, + boundary_dates=(pd.Timestamp("2023-06-30"),), factor_names=("momentum_20", "roe", "netprofit_yoy"), performance={ "equal_weight": {"train": dict(period), "test": dict(period)}, @@ -254,8 +305,10 @@ def test_render_oos_report_disclosed_boundaries_and_metrics(): # split boundaries written into the report assert "2023-07-01" in md and "2022-07-01" in md and "2024-06-28" in md assert "train" in md.lower() and "test" in md.lower() - # walk-forward boundary semantics stated + # walk-forward boundary semantics + holding-window slicing stated assert "realized" in md.lower() + assert "holding" in md.lower() + assert "2023-06-30" in md # the straddling rebalance is disclosed # both models' subperiod performance + IC stability + weight stability assert "equal_weight" in md and "ic_weighted" in md assert "hit rate" in md.lower() From f90c36d442580c99d75bc3de05b256019e779726 Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Wed, 10 Jun 2026 17:35:16 +0800 Subject: [PATCH 3/5] fix(oos): label straddling rebalance rows 'boundary' in the weights table Review LOW on PR #12: the per-rebalance weights table labelled the straddling 2023-06-30 row 'train' while the performance slicing had already excluded it from both subperiods as a boundary row. The period label in the weights table now matches the performance slicing (boundary, not train/test). Display-only; no metric changes. Render test asserts the boundary label. --- qt/reports.py | 9 ++++++++- tests/test_oos_stability.py | 7 ++++++- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/qt/reports.py b/qt/reports.py index f072855..36c9b4e 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -757,8 +757,15 @@ def render_oos_stability(result) -> str: header = "| Date | period | " + " | ".join(f"`{c}`" for c in factor_cols) header += " | fallback |\n|" + "---|" * (len(factor_cols) + 3) + "\n" body = "" + # period labels match the PERFORMANCE slicing: a straddling rebalance + # (holding window crosses the split) is labelled boundary, not train. + boundary = {pd.Timestamp(d) for d in (result.boundary_dates or ())} for date, row in w.iterrows(): - period = "test" if pd.Timestamp(date) >= result.split_date else "train" + ts = pd.Timestamp(date) + if ts in boundary: + period = "boundary" + else: + period = "test" if ts >= result.split_date else "train" cells = " | ".join(_fmt(float(row[c])) for c in factor_cols) body += ( f"| {_date_str(date)} | {period} | {cells} | " diff --git a/tests/test_oos_stability.py b/tests/test_oos_stability.py index 9bfc191..14325df 100644 --- a/tests/test_oos_stability.py +++ b/tests/test_oos_stability.py @@ -273,7 +273,9 @@ def _synthetic_oos_result(): train_start=pd.Timestamp("2022-07-01"), train_end=pd.Timestamp("2023-06-30"), test_start=pd.Timestamp("2023-07-03"), test_end=pd.Timestamp("2024-06-28"), n_train_days=240, n_test_days=238, - boundary_dates=(pd.Timestamp("2023-06-30"),), + # one boundary date inside the weights table (2023-05-31) so the + # weights-row labelling is exercised, plus one outside it. + boundary_dates=(pd.Timestamp("2023-05-31"), pd.Timestamp("2023-06-30")), factor_names=("momentum_20", "roe", "netprofit_yoy"), performance={ "equal_weight": {"train": dict(period), "test": dict(period)}, @@ -309,6 +311,9 @@ def test_render_oos_report_disclosed_boundaries_and_metrics(): assert "realized" in md.lower() assert "holding" in md.lower() assert "2023-06-30" in md # the straddling rebalance is disclosed + # a straddling rebalance row in the WEIGHTS table is labelled boundary, + # matching the performance slicing (not mislabelled train). + assert "| 2023-05-31 | boundary |" in md # both models' subperiod performance + IC stability + weight stability assert "equal_weight" in md and "ic_weighted" in md assert "hit rate" in md.lower() From 6cf1ad9f0b2dc029c374320102e6f411e44e622c Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Wed, 10 Jun 2026 17:39:30 +0800 Subject: [PATCH 4/5] =?UTF-8?q?docs:=20sync=20progress=20=E2=80=94=20PR=20?= =?UTF-8?q?#10/#11=20merged,=20P3-2=20done,=20P3-3=20=3D=20PR=20#12=20unde?= =?UTF-8?q?r=20review?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Doc-only sync in CLAUDE.md + AGENTS.md (both files, same drift): - merged-PR lists: add #10 (progress docs) and #11 (P3-2). - P3-2 entry: 'branch, awaiting acceptance' -> 'PR #11 merged to main'. - P3-3 entry: 'branch, awaiting acceptance' -> 'PR #12 OPEN, review HIGHs+LOW fixed, awaiting acceptance/merge'. --- AGENTS.md | 6 +++--- CLAUDE.md | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 9b48c82..4a193d7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**；代码/注释/commit message 用**英文**。 -- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3,默认 SW-L1 可配置）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）均已 merge 到 `main`。** commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3,默认 SW-L1 可配置）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）、#10（进度文档）、#11（P3-2）均已 merge 到 `main`。** commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 - **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 - **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 - 文件小而专（<800 行），immutable 优先。 @@ -92,12 +92,12 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 财务字段**一次 fetch + 一次 as-of 对齐**;合成仍 EqualWeightAlpha 等权;`drop_missing` 要求全因子齐备（披露）;demo+财务因子仍可读报错;重名因子报错。 - 报告：active factor list / per-factor coverage+IC+分位 / combo score 诊断 / 财务 coverage 按字段（TRADED vs diagnostic）;`output.baseline_report_name` 分离 phase3 报告。 - 真实结果:多因子 annual **−9.05%**（单因子 −10.19%）;momentum_20 IC 0.0083（跨 run 一致）/ roe 0.0006 / netprofit_yoy 0.0001 / combo −0.0038;财务 ann_date 覆盖 100%;PIT SW-L1 98.53%。回归不破:phase2 rerun −10.19%/0.0083 不变,demo 0.96/0.84 不变。 -- 🔧 **Phase 3-2 walk-forward IC 加权 alpha**（`p3-ic-weighted-alpha` 分支,代劳待验收）：`alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认+回归基线;不调参、非收益声明。 +- ✅ **Phase 3-2 walk-forward IC 加权 alpha**（**PR #11 已 merge 到 `main`**）：`alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认+回归基线;不调参、非收益声明。 - Lookahead 边界测试锁定:(factor[t], fwd_h[t]) 仅**已实现**(`t+h <= d` 交易日序)进权重;扰动未实现 fwd 权重不变;fwd 只进 `alpha.fit`,factors 层绝不接触。 - rolling(默认 60d/min20)/expanding;历史不足→该日退回等权(计数披露);权重 L1 归一化、保留符号。报告新增 **Alpha model** 必含小节(模型/超参/覆盖率/每期权重表/fallback/非调参声明)。 - `config/phase3_real_ic_weighted.yaml` 与 phase3_real_multifactor 唯一差异 alpha.model(直接可比)。 - 真实结果:annual **−3.57%**(等权 −9.05%),训练覆盖 201/221(20 fallback 全在窗口攒满前)。⚠️ 优于等权非业绩声明——单年窗口+权重逐期翻号即小样本不稳定,照实披露。回归不破:等权 rerun −9.05%/0.0083 不变,demo 0.96/0.84 不变。 -- 🔧 **Phase 3-3 OOS 稳定性验证**（`p3-oos-stability-validation` 分支,代劳待验收）：报告型验证层（`run-phase3-oos` + `qt/oos_stability.py` + `config/phase3_real_oos_stability.yaml`,SSE50 2 年,split 2023-07-01）;不加 alpha 复杂度、不改 portfolio/execution/factor math。 +- 🔧 **Phase 3-3 OOS 稳定性验证**（**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并）：报告型验证层（`run-phase3-oos` + `qt/oos_stability.py` + `config/phase3_real_oos_stability.yaml`,SSE50 2 年,split 2023-07-01）;不加 alpha 复杂度、不改 portfolio/execution/factor math。 - 一次数据加载、两次回测(equal_weight vs ic_weighted)。walk-forward 边界测试锁定:扰动 split 后全部 fwd,train 期权重逐 bit 不变。**绩效按持有窗口切片**(train 持有 end≤split、test start≥split,跨界调仓排除并披露;IC 按实现日 t+h 切;review HIGH 修复——旧 signal-date 切法把跨界收益记进 train);runner 强制 alpha.model=ic_weighted(防假对比)。 - 报告:split 边界+跨界行/分期绩效/逐序列 IC(mean/IR/hit/sign consistency)/权重稳定性(sign flips、fallback+原因)/小样本 caveat。 - **真实关键发现**:三因子 train→test IC **全部翻号**(sign consistency 全 NO),hit 46~53%;权重 sign flips 7/3/4;eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(修切片前 train 被污染到 −6.81%/−1.69%)。**P3-2 单年跑赢不可外推——这正是验证层要拿到的证据;非收益声明。**回归不破:eq −9.05%/ic −3.57%/demo 0.96/0.84 全不变;报告 secret scan 干净。 diff --git a/CLAUDE.md b/CLAUDE.md index 5970b5a..969931d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**；代码/注释/commit message 用**英文**。 -- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）均已 merge 到 `main`**。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）、#10（进度文档）、#11（P3-2）均已 merge 到 `main`**。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 - **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 - **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 - 文件小而专（<800 行），immutable 优先。 @@ -94,14 +94,14 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 报告增强：active factor list / per-factor coverage+IC+分位 / **combo score** 诊断 / 财务 coverage **按字段**披露（TRADED vs diagnostic 角色标注）;`output.baseline_report_name` 使 phase3 报告独立于 phase2。 - **真实结果**（SSE50 2023-07~2024-06,~14min）:多因子 annual **−9.05%**（单因子 −10.19%）;per-factor IC: momentum_20 0.0083（与 phase2 run 完全一致,跨 run 一致性实证）/ roe 0.0006 / netprofit_yoy 0.0001;combo IC −0.0038;财务两字段 ann_date 覆盖 **100%**;PIT SW-L1 98.53%。财务因子在该小截面短窗口 IC≈0,照实披露——这是 plumbing 验证。 - **回归不破**:phase2 单因子真实 rerun annual −10.19% / IC 0.0083 不变;demo ic 0.96/annual 0.84 不变。 -- 🔧 **Phase 3-2 walk-forward IC 加权 alpha**（`p3-ic-weighted-alpha` 分支,代劳待验收）：新增 `alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。 +- ✅ **Phase 3-2 walk-forward IC 加权 alpha**（**PR #11 已 merge 到 `main`**）：新增 `alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。 - **Lookahead 边界（测试锁定）**：训练严格 walk-forward——(factor[t], fwd_h[t]) 只有**已实现**（交易日序 `t+h <= d`）才进日 d 的权重;**扰动未实现 forward returns 权重不变**（扰动测试）+ `t+h` 切片精确边界测试。forward returns 由 pipeline 在 alpha 边界计算、只传 `alpha.fit`,factors 层照旧绝不接触（不变量 #1）。 - rolling（默认保守,窗口=60 交易日,min_periods=20）/ expanding 可配;历史不足 → 该日**退回等权**（与 EqualWeightAlpha 逐 bit 一致）并计数披露;权重 L1 归一化、保留符号（负 IC 因子负权重）。 - 报告新增 **Alpha model** 必含小节：active model / 超参 / 训练覆盖率 + fallback 次数 / 每调仓日生效权重表（fallback 行标注）/ 非调参声明 + 等权基线对比指引。 - `config/phase3_real_ic_weighted.yaml`：与 phase3_real_multifactor **唯一差异是 alpha.model**（universe/window/因子/中性化全同,直接可比）。 - **真实结果**（SSE50 2023-07~2024-06,~14min）:annual **−3.57%**（等权 −9.05% / 单因子 −10.19%）,maxDD −12.93%,训练覆盖 **201/221**（20 个 fallback 全在窗口攒满前,90.95%）。⚠️ 优于等权**不是**业绩声明——单年窗口 + 权重逐期翻号（如 momentum_20 从 −0.58 到 +0.36）正是小样本不稳定的体现,照实披露。 - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变（测试锁定）。 -- 🔧 **Phase 3-3 OOS 稳定性验证**（`p3-oos-stability-validation` 分支,代劳待验收）：**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`（`qt/oos_stability.py`）+ `config/phase3_real_oos_stability.yaml`（SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照）。 +- 🔧 **Phase 3-3 OOS 稳定性验证**（**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并）：**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`（`qt/oos_stability.py`）+ `config/phase3_real_oos_stability.yaml`（SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照）。 - **一次数据加载、同一 processed 因子面板、两次回测**（equal_weight vs ic_weighted）;所有诊断按 split 切段（子段 nav 重新归一,绝不跨段串味）。 - **边界语义（测试锁定）**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。**绩效切片按持有窗口**(train 行持有期 end≤split、test 行 start≥split,跨界调仓从两段排除并披露;IC 按实现日 t+h 切)——绝不按 signal date 单切(review HIGH 修复:旧切法把跨界持有期的 test 收益记进 train);runner 强制 `alpha.model: ic_weighted`(否则假对比,可读报错)。 - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 From 90aed74365a7acb87e76e9116335e78fbdf873cb Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Wed, 10 Jun 2026 17:40:33 +0800 Subject: [PATCH 5/5] docs: make AGENTS.md a verbatim copy of CLAUDE.md Maintaining two slightly-divergent progress docs invited drift (three review findings so far were exactly this). AGENTS.md is now a straight copy of CLAUDE.md; future updates edit CLAUDE.md and copy over. --- AGENTS.md | 68 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 37 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 4a193d7..969931d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,8 +39,8 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 数据：tushare - **token**：`/home/shaofl/Projects/financial_projects/.config.json`（key `tushare.token`）。 ⚠️ **绝不打印、绝不写进 repo、绝不 commit。** 代码里从该文件读取，不硬编码。 -- **权限**：实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业历史(`index_member_all`/`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上，无需先退回日线。 -- **MCP（可选开发工具）**：`financial_projects/.mcp.json` 有 tushare MCP，仅供开发期交互查数；**从 `financial_projects/` 启动 Codex 才加载**。 +- **权限**：实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业(`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上，无需先退回日线。 +- **MCP（可选开发工具）**：`financial_projects/.mcp.json` 有 tushare MCP，仅供开发期交互查数；**从 `financial_projects/` 启动 claude 才加载**。 - **数据层 ETL 一律用 Python SDK（批量/增量），不要建在 MCP 上。** 注意 tushare 各接口有每分钟调用上限，批量拉取需限流+重试。 ## 技术选型 @@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**；代码/注释/commit message 用**英文**。 -- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3,默认 SW-L1 可配置）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）、#10（进度文档）、#11（P3-2）均已 merge 到 `main`。** commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）、#10（进度文档）、#11（P3-2）均已 merge 到 `main`**。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 - **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 - **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 - 文件小而专（<800 行），immutable 优先。 @@ -73,34 +73,40 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 行业 + 市值中性化（按 date 截面 OLS 残差；欠定/无自由度截面 → NaN） - 路径感知降级披露（demo/static vs tushare/index/ann_date，绝不把 demo 当真实验证） - ✅ 真数据实证（tushare，非 CI）：复权除权日 raw−5.74%→qfq+0.99% / CSI300 全年 24 快照 328 名换手 / ann_date Q1 延后至 04-20 / 中性化 corr −0.617→0。详见 `BIAS_AUDIT.md`、`artifacts/reports/phase1_summary.md`。 -- ✅ **Phase 2-1 真实数据可复现基准**（PR #2 已 merge）：新 run mode `run-phase2-baseline` + `config/phase2_real_baseline.yaml`（上证50 `000016.SH`，2023-07~2024-06）。复用 P0/P1 全套机器，不扩因子、不调参。一次真实跑约 11 min，输出 `artifacts/reports/phase2_real_baseline.md`（gitignored）：数据窗口 / PIT 成分摘要 / ann_date 覆盖率 / 可交易过滤 / 每期持仓 / 换手成本 / IC / 绩效 / 全部 P2 降级。 -- ✅ **Phase 2-2 执行真实性**（PR #3 已 merge）：拆分 selection（选谁）与 execution feasibility（能否成交）。 - - `runtime/fills.py::simulate_fills` 方向感知执行：涨停挡买 / 跌停挡卖 / 停牌·缺收盘双向挡；按 panel flag 实时判定，与选股 toggle 无关。 - - 现金一致 sell-then-buy：卖在前释放现金、买在后，现金不足按比例部分成交 → 无杠杆；被挡交易 carry forward；换手/成本只算实际成交；闲置现金按 driver 的 `cash_return` 计息。 - - `universe.min_listing_days` 真实路径已执行（`stock_basic.list_date` 富化，买入资格过滤，边界 age==min 放行，缺 list_date 保留并披露）；demo 无上市日 → 披露 no-op。 - - 回测 `feasibility_log()` 记录 blocked buys/sells/carried/executed turnover/invested；phase2 报告有 **Execution feasibility** 小节。 -- ✅ **Phase 2-3 历史 PIT 行业**（**PR #5 已 merge**,默认 SW-L1 可配置）：把行业中性化协变量从 `stock_basic.industry` 当前标签升级为按 trade_date as-of 的历史申万行业。 - - `tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 的 `in_date`/`out_date` 区间；`data/clean/pit_industry.py::asof_industry` 按 `[in_date, out_date)` 覆盖该日取行业，改分类日新行业生效，PIT-safe，起始日前成分 carry forward 到窗口开始。 - - **SW 层级可配置** `processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,中性化标准 + 小截面自由度更稳)。**关键实证**:旧 tag 年化 −17.6% / SW-L1 −10.2% / SW-L2 −9.3% → **L1≈L2**,−17.6→−10 大跳主因是 **tushare→SW 分类切换**(补 PIT 必然:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**。(曾误判粒度、默认 L2,经 L2 实测推翻,改默认 L1。) - - 不静默退回 current：无 SW 历史的票 → 行业 NaN，被 neutralize 按截面丢弃；每次运行 **实际 level + PIT 覆盖率**进 phase2 报告。 - - 真实 baseline(SW-L1,默认):`run-phase2-baseline` OK,symbols=68,settled=11,**PIT SW-L1 coverage 98.53%**(67/68),IC 0.0083,**annual −10.19%**。 +- ✅ **Phase 2-1 真实数据可复现基准**（**PR #2 已 merge 到 `main`**）：新 run mode `run-phase2-baseline` + `config/phase2_real_baseline.yaml`（上证50 `000016.SH`，2023-07~2024-06）。**复用 P0/P1 全套机器，不扩因子、不调参**。一次真实跑 ~11min（68 成分 / 25 loaded 快照 / in-window distinct 60 / 11 settled 调仓，候选 12 末日跳过），输出 `artifacts/reports/phase2_real_baseline.md`（gitignored）：数据窗口 / PIT 成分摘要(loaded vs in-window) / ann_date 覆盖率(100%) / 可交易过滤命中(首命中互斥) / 每期持仓 / 换手成本 / IC(≈0.008) / 绩效(年化−17.6%，**亏损动量基准，非业绩声明**) / 全部 P2 降级。诊断只读、demo 源拒绝、token 不入报告。 +- ✅ **Phase 2-2 执行真实性**（PR #3 已 merge 到 `main`）：**拆分 selection（选谁）与 execution feasibility（能否成交）**。 + - 方向感知执行 `runtime/fills.py::simulate_fills`：涨停挡买 / 跌停挡卖 / 停牌·缺收盘双向挡;按 panel flag 实时判定,与选股 toggle 无关。 + - **现金一致 sell-then-buy**:卖在前释放现金、买在后,现金不足按比例部分成交 → **无杠杆**;被挡交易 carry forward;换手/成本只算实际成交;闲置现金按 driver 的 `cash_return` 计息（BT-007）。 + - `universe.min_listing_days` **真实路径已执行**（`stock_basic.list_date` 富化,买入资格过滤,边界 age==min 放行,缺 list_date 保留并披露）;demo 无上市日 → 披露 no-op。 + - 回测 `feasibility_log()`：每调仓期 blocked buys/sells/carried/executed turnover/invested;phase2 报告新增 **Execution feasibility** 小节。 + - **保不变量**:demo 无 flag → 全可成交 → P0/P1 数字不变;无未来函数、PIT/ann_date/real-demo 分离不变。 +- ✅ **Phase 2-3 历史 PIT 行业**（**PR #5 已 merge 到 `main`**,默认 SW-L1 可配置）：把行业中性化协变量从 `stock_basic.industry` **当前**标签升级为按 trade_date **as-of** 的历史申万行业。 + - `tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 取每股 SW 行业 `in_date`/`out_date` 区间;`data/clean/pit_industry.py::asof_industry` 按 `[in_date,out_date)` 覆盖该日取行业(改分类日新行业生效,PIT-safe;起始日前成分 carry forward 到窗口开始)。 + - **SW 层级可配置** `processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,中性化标准 + 小截面自由度更稳)。**关键实证**:旧 tag 年化 −17.6% / SW-L1 −10.2% / SW-L2 −9.3% → **L1≈L2**,大跳的主因是 **tushare→SW 分类切换**(补 PIT 的必然:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**。(注:曾误以为是粒度、默认 L2,经 L2 实测推翻,改默认 L1。) + - **不静默退回 current**:无 SW 历史的票 → 行业 NaN,被 neutralize 按截面丢弃(neutralize 数学不变,本就丢 NaN 行);每次运行 **实际 level + PIT 覆盖率**进 phase2 报告。 - ✅ **Phase 2-4 标准分析集成**（**PR #7 已 merge 到 `main`**）：alphalens-reloaded + quantstats 接入报告(**report-only cross-check**)。 - - `analytics/alphalens_adapter.py`(IC mean/IR + 分位均值)+ `analytics/quantstats_adapter.py`(CAGR/Sharpe/maxDD/vol)薄 adapter,各带 `backend` 字段;`_import_*` 间接层使 import-missing 路径可测;alphalens stdout/warnings 已抑制。 - - **简版仍权威**(驱动回测 + cross-check);依赖不可用/报错 → 报告披露 backend(unavailable/error,只记异常类型),**绝不静默假装**。 - - **不改 alpha/portfolio/runtime/fills/universe**:P0/P2 交易数字不变(demo ic 0.96/annual 0.84 不变;alphalens IC=简版 IC 吻合),只新增 **Standard analytics** 报告段。 -- ✅ **Phase 3-1 首个真实多因子 baseline**（**PR #9 已 merge 到 `main`**）：pipeline 消费**全部 enabled factors**（曾只用第一个）;`config/phase3_real_multifactor.yaml` = momentum_20 + roe + netprofit_yoy（SSE50 同窗口,与 phase2 可比;不调参、无 learned weights、非收益承诺）。 - - 财务字段**一次 fetch + 一次 as-of 对齐**;合成仍 EqualWeightAlpha 等权;`drop_missing` 要求全因子齐备（披露）;demo+财务因子仍可读报错;重名因子报错。 - - 报告：active factor list / per-factor coverage+IC+分位 / combo score 诊断 / 财务 coverage 按字段（TRADED vs diagnostic）;`output.baseline_report_name` 分离 phase3 报告。 - - 真实结果:多因子 annual **−9.05%**（单因子 −10.19%）;momentum_20 IC 0.0083（跨 run 一致）/ roe 0.0006 / netprofit_yoy 0.0001 / combo −0.0038;财务 ann_date 覆盖 100%;PIT SW-L1 98.53%。回归不破:phase2 rerun −10.19%/0.0083 不变,demo 0.96/0.84 不变。 -- ✅ **Phase 3-2 walk-forward IC 加权 alpha**（**PR #11 已 merge 到 `main`**）：`alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认+回归基线;不调参、非收益声明。 - - Lookahead 边界测试锁定:(factor[t], fwd_h[t]) 仅**已实现**(`t+h <= d` 交易日序)进权重;扰动未实现 fwd 权重不变;fwd 只进 `alpha.fit`,factors 层绝不接触。 - - rolling(默认 60d/min20)/expanding;历史不足→该日退回等权(计数披露);权重 L1 归一化、保留符号。报告新增 **Alpha model** 必含小节(模型/超参/覆盖率/每期权重表/fallback/非调参声明)。 - - `config/phase3_real_ic_weighted.yaml` 与 phase3_real_multifactor 唯一差异 alpha.model(直接可比)。 - - 真实结果:annual **−3.57%**(等权 −9.05%),训练覆盖 201/221(20 fallback 全在窗口攒满前)。⚠️ 优于等权非业绩声明——单年窗口+权重逐期翻号即小样本不稳定,照实披露。回归不破:等权 rerun −9.05%/0.0083 不变,demo 0.96/0.84 不变。 -- 🔧 **Phase 3-3 OOS 稳定性验证**（**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并）：报告型验证层（`run-phase3-oos` + `qt/oos_stability.py` + `config/phase3_real_oos_stability.yaml`,SSE50 2 年,split 2023-07-01）;不加 alpha 复杂度、不改 portfolio/execution/factor math。 - - 一次数据加载、两次回测(equal_weight vs ic_weighted)。walk-forward 边界测试锁定:扰动 split 后全部 fwd,train 期权重逐 bit 不变。**绩效按持有窗口切片**(train 持有 end≤split、test start≥split,跨界调仓排除并披露;IC 按实现日 t+h 切;review HIGH 修复——旧 signal-date 切法把跨界收益记进 train);runner 强制 alpha.model=ic_weighted(防假对比)。 - - 报告:split 边界+跨界行/分期绩效/逐序列 IC(mean/IR/hit/sign consistency)/权重稳定性(sign flips、fallback+原因)/小样本 caveat。 - - **真实关键发现**:三因子 train→test IC **全部翻号**(sign consistency 全 NO),hit 46~53%;权重 sign flips 7/3/4;eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(修切片前 train 被污染到 −6.81%/−1.69%)。**P3-2 单年跑赢不可外推——这正是验证层要拿到的证据;非收益声明。**回归不破:eq −9.05%/ic −3.57%/demo 0.96/0.84 全不变;报告 secret scan 干净。 -- ✅ 当前质量门：`pytest -p no:cacheprovider` **285 passed**；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）OK；`run-phase0`（demo）OK。 + - `analytics/alphalens_adapter.py`(IC mean/IR + 分位均值)+ `analytics/quantstats_adapter.py`(CAGR/Sharpe/maxDD/vol)薄 adapter,各带 `backend` 字段。 + - **简版 numpy/pandas 仍权威**(驱动回测 + cross-check);依赖不可用/报错 → 报告显式披露 backend(unavailable/error,只记异常**类型**不记消息),**绝不静默假装用了标准库**。 + - **不改 alpha/portfolio/runtime/fills/universe**:P0/P2 交易数字不变(demo ic 0.96/annual 0.84 不变;实测 alphalens IC=简版 IC 完全吻合),只新增 **Standard analytics** 报告段。 +- ✅ **Phase 3-1 首个真实多因子 baseline**（**PR #9 已 merge 到 `main`**）：pipeline 从"只用第一个 enabled factor"升级为**消费全部 enabled factors**;新增 `config/phase3_real_multifactor.yaml`（momentum_20 + roe + netprofit_yoy,SSE50 同窗口,与 phase2 可比）。**不调参、无 learned weights、非收益承诺**。 + - `_build_factors` 全量实例化（配置序;重名报错）;财务字段**一次 fetch + 一次 as-of 对齐**（逐字段独立守 `ann_date <= trade_date`,无每因子重复拉取）;demo+财务因子仍可读报错。 + - 合成仍 `EqualWeightAlpha`（处理后各列等权平均,不看 forward returns）;`drop_missing` 要求该日该票**所有**因子齐备（显式披露）。 + - 报告增强：active factor list / per-factor coverage+IC+分位 / **combo score** 诊断 / 财务 coverage **按字段**披露（TRADED vs diagnostic 角色标注）;`output.baseline_report_name` 使 phase3 报告独立于 phase2。 + - **真实结果**（SSE50 2023-07~2024-06,~14min）:多因子 annual **−9.05%**（单因子 −10.19%）;per-factor IC: momentum_20 0.0083（与 phase2 run 完全一致,跨 run 一致性实证）/ roe 0.0006 / netprofit_yoy 0.0001;combo IC −0.0038;财务两字段 ann_date 覆盖 **100%**;PIT SW-L1 98.53%。财务因子在该小截面短窗口 IC≈0,照实披露——这是 plumbing 验证。 + - **回归不破**:phase2 单因子真实 rerun annual −10.19% / IC 0.0083 不变;demo ic 0.96/annual 0.84 不变。 +- ✅ **Phase 3-2 walk-forward IC 加权 alpha**（**PR #11 已 merge 到 `main`**）：新增 `alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。 + - **Lookahead 边界（测试锁定）**：训练严格 walk-forward——(factor[t], fwd_h[t]) 只有**已实现**（交易日序 `t+h <= d`）才进日 d 的权重;**扰动未实现 forward returns 权重不变**（扰动测试）+ `t+h` 切片精确边界测试。forward returns 由 pipeline 在 alpha 边界计算、只传 `alpha.fit`,factors 层照旧绝不接触（不变量 #1）。 + - rolling（默认保守,窗口=60 交易日,min_periods=20）/ expanding 可配;历史不足 → 该日**退回等权**（与 EqualWeightAlpha 逐 bit 一致）并计数披露;权重 L1 归一化、保留符号（负 IC 因子负权重）。 + - 报告新增 **Alpha model** 必含小节：active model / 超参 / 训练覆盖率 + fallback 次数 / 每调仓日生效权重表（fallback 行标注）/ 非调参声明 + 等权基线对比指引。 + - `config/phase3_real_ic_weighted.yaml`：与 phase3_real_multifactor **唯一差异是 alpha.model**（universe/window/因子/中性化全同,直接可比）。 + - **真实结果**（SSE50 2023-07~2024-06,~14min）:annual **−3.57%**（等权 −9.05% / 单因子 −10.19%）,maxDD −12.93%,训练覆盖 **201/221**（20 个 fallback 全在窗口攒满前,90.95%）。⚠️ 优于等权**不是**业绩声明——单年窗口 + 权重逐期翻号（如 momentum_20 从 −0.58 到 +0.36）正是小样本不稳定的体现,照实披露。 + - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变（测试锁定）。 +- 🔧 **Phase 3-3 OOS 稳定性验证**（**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并）：**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`（`qt/oos_stability.py`）+ `config/phase3_real_oos_stability.yaml`（SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照）。 + - **一次数据加载、同一 processed 因子面板、两次回测**（equal_weight vs ic_weighted）;所有诊断按 split 切段（子段 nav 重新归一,绝不跨段串味）。 + - **边界语义（测试锁定）**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。**绩效切片按持有窗口**(train 行持有期 end≤split、test 行 start≥split,跨界调仓从两段排除并披露;IC 按实现日 t+h 切)——绝不按 signal date 单切(review HIGH 修复:旧切法把跨界持有期的 test 收益记进 train);runner 强制 `alpha.model: ic_weighted`(否则假对比,可读报错)。 + - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 + - **真实结果（关键发现,~16min,77 成分/2 年,持有窗口切片）**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** + - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 +- ✅ 质量门：`pytest` **285 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）+ `run-phase0`（demo）均 OK。 - ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、因子 IC 小样本不稳定（P3-3 实证）。 - 路线图下一步：更长历史/更宽 universe 的稳定性复检,或分钟级（architecture.html §11）。