feat(validation): P3-3 OOS stability validation (equal_weight vs ic_weighted)

CLAUDE.md

@@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri
 
 ## 开发约定
 - **交流中文**；代码/注释/commit message 用**英文**。
-- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）均已 merge 到 `main`**。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。
+- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）、#5（P2-3）、#6（进度文档）、#7（P2-4）、#8（进度文档）、#9（P3-1）、#10（进度文档）、#11（P3-2）均已 merge 到 `main`**。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。
 - **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。
 - **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。
 - 文件小而专（<800 行），immutable 优先。
@@ -94,13 +94,19 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri
   - 报告增强：active factor list / per-factor coverage+IC+分位 / **combo score** 诊断 / 财务 coverage **按字段**披露（TRADED vs diagnostic 角色标注）;`output.baseline_report_name` 使 phase3 报告独立于 phase2。
   - **真实结果**（SSE50 2023-07~2024-06,~14min）:多因子 annual **−9.05%**（单因子 −10.19%）;per-factor IC: momentum_20 0.0083（与 phase2 run 完全一致,跨 run 一致性实证）/ roe 0.0006 / netprofit_yoy 0.0001;combo IC −0.0038;财务两字段 ann_date 覆盖 **100%**;PIT SW-L1 98.53%。财务因子在该小截面短窗口 IC≈0,照实披露——这是 plumbing 验证。
   - **回归不破**:phase2 单因子真实 rerun annual −10.19% / IC 0.0083 不变;demo ic 0.96/annual 0.84 不变。
-- 🔧 **Phase 3-2 walk-forward IC 加权 alpha**（`p3-ic-weighted-alpha` 分支,代劳待验收）：新增 `alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。
+- ✅ **Phase 3-2 walk-forward IC 加权 alpha**（**PR #11 已 merge 到 `main`**）：新增 `alpha/ic_weight.py::RollingICWeightAlpha`（`alpha.model: ic_weighted`）;EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。
   - **Lookahead 边界（测试锁定）**：训练严格 walk-forward——(factor[t], fwd_h[t]) 只有**已实现**（交易日序 `t+h <= d`）才进日 d 的权重;**扰动未实现 forward returns 权重不变**（扰动测试）+ `t+h` 切片精确边界测试。forward returns 由 pipeline 在 alpha 边界计算、只传 `alpha.fit`,factors 层照旧绝不接触（不变量 #1）。
   - rolling（默认保守,窗口=60 交易日,min_periods=20）/ expanding 可配;历史不足 → 该日**退回等权**（与 EqualWeightAlpha 逐 bit 一致）并计数披露;权重 L1 归一化、保留符号（负 IC 因子负权重）。
   - 报告新增 **Alpha model** 必含小节：active model / 超参 / 训练覆盖率 + fallback 次数 / 每调仓日生效权重表（fallback 行标注）/ 非调参声明 + 等权基线对比指引。
   - `config/phase3_real_ic_weighted.yaml`：与 phase3_real_multifactor **唯一差异是 alpha.model**（universe/window/因子/中性化全同,直接可比）。
   - **真实结果**（SSE50 2023-07~2024-06,~14min）:annual **−3.57%**（等权 −9.05% / 单因子 −10.19%）,maxDD −12.93%,训练覆盖 **201/221**（20 个 fallback 全在窗口攒满前,90.95%）。⚠️ 优于等权**不是**业绩声明——单年窗口 + 权重逐期翻号（如 momentum_20 从 −0.58 到 +0.36）正是小样本不稳定的体现,照实披露。
   - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变（测试锁定）。
-- ✅ 质量门：`pytest` **269 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`）+ `run-phase0`（demo）均 OK。
-- ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据。
-- 路线图下一步：样本外/IC 稳定性检验 / 分钟级（architecture.html §11）。
+- 🔧 **Phase 3-3 OOS 稳定性验证**（**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并）：**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`（`qt/oos_stability.py`）+ `config/phase3_real_oos_stability.yaml`（SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照）。
+  - **一次数据加载、同一 processed 因子面板、两次回测**（equal_weight vs ic_weighted）;所有诊断按 split 切段（子段 nav 重新归一,绝不跨段串味）。
+  - **边界语义（测试锁定）**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。**绩效切片按持有窗口**(train 行持有期 end≤split、test 行 start≥split,跨界调仓从两段排除并披露;IC 按实现日 t+h 切)——绝不按 signal date 单切(review HIGH 修复:旧切法把跨界持有期的 test 收益记进 train);runner 强制 `alpha.model: ic_weighted`(否则假对比,可读报错)。
+  - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。
+  - **真实结果（关键发现,~16min,77 成分/2 年,持有窗口切片）**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。**
+  - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。
+- ✅ 质量门：`pytest` **285 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`）+ `run-phase0`（demo）均 OK。
+- ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、因子 IC 小样本不稳定（P3-3 实证）。
+- 路线图下一步：更长历史/更宽 universe 的稳定性复检,或分钟级（architecture.html §11）。

RUNBOOK.md

@@ -262,6 +262,52 @@ The baseline report gains an **Alpha model** section: active model,
 hyper-params, training coverage (fallback count), and the effective weights at
 every settled rebalance date (fallback rows flagged).
 
+## Phase 3-3 — OOS stability validation (equal_weight vs ic_weighted)
+
+A REPORT-ONLY validation layer (no portfolio / execution / factor-math change):
+one shared data load, the SAME processed factor panel, two backtests — one per
+alpha — and every diagnostic split at `oos.split_date`. Documented by
+`config/phase3_real_oos_stability.yaml` (SSE50, two years 2022-07 ~ 2024-06,
+split 2023-07-01 → train 1y / test 1y; the test year equals the phase3
+baselines' window, so numbers line up).
+
+```bash
+# validate (no network)
+... -m qt.cli validate-config  --config config/phase3_real_oos_stability.yaml
+# run the OOS validation (network + token; heavy, ~15-30 min)
+... -m qt.cli run-phase3-oos   --config config/phase3_real_oos_stability.yaml
+```
+
+Split semantics (locked by tests):
+
+- train = `[data.start, split)`, test = `[split, data.end]`; the exact realized
+  dates and day counts are written into the report.
+- Evaluation is **walk-forward (rolling subperiod)**: weights at any date d use
+  only observations REALIZED by d (`t + horizon <= d`) — perturbing every
+  post-split forward return cannot change any train-period date's weights
+  (split-boundary no-leakage test). Freezing weights at the split is NOT used
+  (that would be a new alpha mode; P3-3 adds no alpha complexity).
+- **Performance slicing is HOLDING-WINDOW aware**: a nav row is indexed by its
+  rebalance (signal) date but its return covers [that rebalance, the next one] —
+  so train rows must have their holding END on/before the split, test rows their
+  holding START on/after it, and a straddling rebalance is EXCLUDED from both
+  subperiods and disclosed in the report. IC stats are sliced by the realization
+  date (`t + horizon`) the same way. The test subperiod therefore starts with
+  the first post-split holding period — the same place the 1-year phase3
+  baselines start.
+- The config's `alpha` section carries the ic_weighted params and MUST set
+  `alpha.model: ic_weighted` (guarded: any other model is refused — running
+  equal_weight twice and labelling one leg ic_weighted would be a fake
+  comparison); the `equal_weight` control leg is built internally.
+
+Report (`artifacts/reports/phase3_oos_stability.md`): split boundaries;
+per-subperiod performance for both models (annual / vol / Sharpe / maxDD /
+turnover / rebalances); per-series IC stability (mean / IR / hit rate / n +
+train-vs-test sign consistency) for every raw factor and both combo scores;
+ic_weighted weight stability (per-rebalance weights with train/test labels,
+sign-flip counts on trained rows, fallback count + reasons); and the explicit
+caveat that this is a small-sample stability check, NOT a return claim.
+
 ## Quality gate
 
 ```bash

TEST_REPORT.md

@@ -1,4 +1,4 @@
-# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha)
+# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha → OOS stability)
 
 ## Commands
 
@@ -12,19 +12,20 @@ Run from the repo root with the project python (env `quant_mf`):
 /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase2_real_baseline.yaml
 /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_multifactor.yaml
 /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_ic_weighted.yaml
+/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_oos_stability.yaml
 /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli run-phase0 --config config/example.yaml
 ```
 
 ## Results
 
 | Gate | Command | Result |
 |---|---|---|
-| Unit + integration | `pytest -q` | **269 passed, 0 failed** |
+| Unit + integration | `pytest -q` | **285 passed, 0 failed** |
 | Lint | `ruff check .` | **All checks passed** |
-| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`) | exit `0`, prints `OK` |
+| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`) | exit `0`, prints `OK` |
 | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` |
 
-Counts below are the actual per-file `pytest` numbers (sum = 269).
+Counts below are the actual per-file `pytest` numbers (sum = 285).
 
 ## Per-file breakdown — Phase 0 core (97)
 
@@ -109,7 +110,13 @@ Counts below are the actual per-file `pytest` numbers (sum = 269).
 |---|---|---|
 | `test_ic_weight_alpha.py` | 12 | **lookahead red-line**: perturbing unrealized forward returns cannot change weights; exact `t + h <= d` realization cutoff (min_periods boundary); insufficient-history equal-weight fallback (== EqualWeightAlpha row mean); single-factor degeneration to ±1; L1 normalization + sign preservation; degenerate-IC fallback; rolling-vs-expanding window; fit requires forward_returns; dated-cross-section contract; input immutability; weights/fallback log |
 | `test_ic_alpha_pipeline.py` | 6 | alpha dispatch by config (equal_weight / ic_weighted + params / unknown = ConfigError), equal-weight default keeps exact demo numbers (ic 0.96 / annual 0.84) + report line, ic_weighted demo e2e (summary, weights log, early-fallback→late-trained, L1 rows, report disclosure, no secret), ic-weights differ from equal weight on diverging-IC synthetic data |
-| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2)** | **269** | |
+
+## Per-file breakdown — Phase 3-3 OOS stability (16)
+
+| Test file | Tests | Red-line / feature |
+|---|---|---|
+| `test_oos_stability.py` | 16 | **split-boundary no-leakage**: perturbing every post-split forward return leaves all train-period weights bit-identical; **holding-window slicing** (`split_nav_by_holding`: train = holding end ≤ split, test = start ≥ split, straddlers + unknown-end rows excluded and disclosed); `subperiod_perf` rebased nav + empty-slice NaN; `ic_period_stats` sliced by realization date (t+h) with mean/IR/hit-rate/n; `sign_consistent` nonzero-same-sign; `weight_sign_flips` on trained rows only (fallback rows excluded); fallback-reason aggregation; OOS config validates + split-inside-window ConfigError; runner rejects demo source / missing `oos` section / **non-ic_weighted alpha (fake-comparison guard)**; report renders boundaries / straddler disclosure / OOS metrics / weight stability / caveat / no secret |
+| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3)** | **285** | |
 
 ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free)
 
@@ -182,6 +189,17 @@ Counts below are the actual per-file `pytest` numbers (sum = 269).
   EqualWeightAlpha combination — and is counted in the report. Weights are
   L1-normalized, sign-preserving. `EqualWeightAlpha` remains the default; its
   demo numbers (ic 0.96 / annual 0.84) are locked unchanged.
+- **P3-3 OOS stability (locked by tests):** `run-phase3-oos` is a REPORT-ONLY
+  validation layer — one shared data load, two backtests (equal_weight vs
+  ic_weighted), every diagnostic split at `oos.split_date`. Performance slicing
+  is HOLDING-WINDOW aware (a nav row's return covers [rebalance, next
+  rebalance]: train rows END on/before the split, test rows START on/after it,
+  straddlers are excluded from both and disclosed); IC stats slice by the
+  realization date (t+h); subperiod navs are rebased so nothing bleeds across.
+  Evaluation is walk-forward (rolling subperiod): the split-boundary test
+  proves post-split forward returns cannot move any train-period weight. The
+  runner refuses a non-ic_weighted `alpha.model` (a fake comparison guard).
+  Portfolio / execution / factor math are untouched.
 - A duplicate test-function name across two files was found and renamed during P2-2
   (it had been silently shadowing one test in the full-suite run). A second, harmless
   duplicate (`test_enrich_does_not_mutate_input` in two files) was verified NOT to drop

config/phase3_real_oos_stability.yaml

@@ -0,0 +1,112 @@
+# Phase 3-3 — REAL out-of-sample stability validation (equal_weight vs ic_weighted).
+#
+# Validation layer, NOT a new strategy: same SSE50 universe, same three factors,
+# same neutralization / portfolio / execution rules as the phase3 baselines —
+# the window is EXTENDED to two years and split at oos.split_date:
+#   train = [data.start, split)   test = [split, data.end]
+# Evaluation is walk-forward (rolling subperiod): weights at any date d use only
+# observations REALIZED by d (t + horizon <= d), so no test-period forward
+# return can reach a train-period computation (locked by tests). The alpha
+# section carries the ic_weighted params; the run ALSO evaluates equal_weight
+# on the same data as the control. NOT a return claim — a small-sample
+# stability check.
+#
+# Run:  python -m qt.cli run-phase3-oos --config config/phase3_real_oos_stability.yaml
+# (report: artifacts/reports/phase3_oos_stability.md; needs the tushare token in
+# the external .config.json; hits the network; heavy ~15-25 min.)
+
+project:
+  name: quantitative_trading_phase3_oos_stability
+  timezone: Asia/Shanghai
+
+data:
+  source: tushare
+  freq: D
+  start: "2022-07-01"
+  end: "2024-06-30"
+  external_secret_file: "/home/shaofl/Projects/financial_projects/.config.json"
+  tushare_token_key: "tushare.token"
+  output_name: phase3oos_daily
+
+universe:
+  type: index
+  index_code: "000016.SH"
+  symbols: []
+  min_listing_days: 60
+  filters:
+    missing_close: true
+    suspended: true
+    st: true
+    limit_up_down: true
+
+factors:
+  - name: momentum_20
+    enabled: true
+    params:
+      window: 20
+      price_col: close
+  - name: roe
+    enabled: true
+    params: {}
+  - name: netprofit_yoy
+    enabled: true
+    params: {}
+
+processing:
+  drop_missing: true
+  standardize:
+    enabled: true
+    method: zscore
+  winsorize:
+    enabled: false
+    method: mad
+    n: 3.0
+  neutralize:
+    enabled: true
+    industry_col: industry
+    size_col: market_cap
+    industry_level: L1   # PIT SW level (L1/L2/L3); L1 = 31 broad sectors (standard, DOF-safe)
+
+alpha:
+  model: ic_weighted          # P3-2 walk-forward rolling-IC weights
+  params:
+    window: 60                # trailing trading days of REALIZED ICs (conservative rolling)
+    min_periods: 20           # < this many valid realized ICs -> equal-weight fallback (disclosed)
+    mode: rolling             # rolling (default) | expanding
+
+portfolio:
+  constructor: topn_equal_weight
+  top_n: 20
+  long_only: true
+  max_weight: null
+  turnover_cap: null
+
+backtest:
+  initial_nav: 1.0
+  rebalance: monthly
+  event_order: close_to_next_period
+  cash_return: 0.0
+
+cost:
+  fee_rate: 0.001
+  slippage_rate: 0.0
+  turnover_formula: l1
+
+analytics:
+  forward_return_periods:
+    - 1
+    - 5
+    - 20
+  quantiles: 5
+  benchmark: null
+
+output:
+  root_dir: artifacts
+  data_dir: artifacts/data
+  factor_dir: artifacts/factors
+  report_dir: artifacts/reports
+  log_dir: artifacts/logs
+  overwrite: true
+
+oos:
+  split_date: "2023-07-01"   # train = 1y [2022-07-01, split); test = 1y [split, 2024-06-30]