diff --git a/AGENTS.md b/AGENTS.md index d7d2ce8..969931d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,8 +39,8 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 数据:tushare - **token**:`/home/shaofl/Projects/financial_projects/.config.json`(key `tushare.token`)。 ⚠️ **绝不打印、绝不写进 repo、绝不 commit。** 代码里从该文件读取,不硬编码。 -- **权限**:实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业历史(`index_member_all`/`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上,无需先退回日线。 -- **MCP(可选开发工具)**:`financial_projects/.mcp.json` 有 tushare MCP,仅供开发期交互查数;**从 `financial_projects/` 启动 Codex 才加载**。 +- **权限**:实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业(`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上,无需先退回日线。 +- **MCP(可选开发工具)**:`financial_projects/.mcp.json` 有 tushare MCP,仅供开发期交互查数;**从 `financial_projects/` 启动 claude 才加载**。 - **数据层 ETL 一律用 Python SDK(批量/增量),不要建在 MCP 上。** 注意 tushare 各接口有每分钟调用上限,批量拉取需限流+重试。 ## 技术选型 @@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**;代码/注释/commit message 用**英文**。 -- **Git**:feature 分支 + PR。**PR #1(P0+P1)、#2(P2-1)、#3(P2-2)、#4(进度文档)、#5(P2-3,默认 SW-L1 可配置)、#6(进度文档)、#7(P2-4)、#8(进度文档)、#9(P3-1)均已 merge 到 `main`。** commit 用 conventional 格式,**无 attribution**(不加 Co-Authored-By)。 +- **Git**:feature 分支 + PR。**PR #1(P0+P1)、#2(P2-1)、#3(P2-2)、#4(进度文档)、#5(P2-3)、#6(进度文档)、#7(P2-4)、#8(进度文档)、#9(P3-1)、#10(进度文档)、#11(P3-2)均已 merge 到 `main`**。commit 用 conventional 格式,**无 attribution**(不加 Co-Authored-By)。 - **不过度设计**:按路线图 MVP 先打通一条端到端链路,再加层(architecture.html §11,Phase 0→3)。 - **secrets** 一律走外部 `.config.json`;repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`(仅留架构文档)。 - 文件小而专(<800 行),immutable 优先。 @@ -73,30 +73,40 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 行业 + 市值中性化(按 date 截面 OLS 残差;欠定/无自由度截面 → NaN) - 路径感知降级披露(demo/static vs tushare/index/ann_date,绝不把 demo 当真实验证) - ✅ 真数据实证(tushare,非 CI):复权除权日 raw−5.74%→qfq+0.99% / CSI300 全年 24 快照 328 名换手 / ann_date Q1 延后至 04-20 / 中性化 corr −0.617→0。详见 `BIAS_AUDIT.md`、`artifacts/reports/phase1_summary.md`。 -- ✅ **Phase 2-1 真实数据可复现基准**(PR #2 已 merge):新 run mode `run-phase2-baseline` + `config/phase2_real_baseline.yaml`(上证50 `000016.SH`,2023-07~2024-06)。复用 P0/P1 全套机器,不扩因子、不调参。一次真实跑约 11 min,输出 `artifacts/reports/phase2_real_baseline.md`(gitignored):数据窗口 / PIT 成分摘要 / ann_date 覆盖率 / 可交易过滤 / 每期持仓 / 换手成本 / IC / 绩效 / 全部 P2 降级。 -- ✅ **Phase 2-2 执行真实性**(PR #3 已 merge):拆分 selection(选谁)与 execution feasibility(能否成交)。 - - `runtime/fills.py::simulate_fills` 方向感知执行:涨停挡买 / 跌停挡卖 / 停牌·缺收盘双向挡;按 panel flag 实时判定,与选股 toggle 无关。 - - 现金一致 sell-then-buy:卖在前释放现金、买在后,现金不足按比例部分成交 → 无杠杆;被挡交易 carry forward;换手/成本只算实际成交;闲置现金按 driver 的 `cash_return` 计息。 - - `universe.min_listing_days` 真实路径已执行(`stock_basic.list_date` 富化,买入资格过滤,边界 age==min 放行,缺 list_date 保留并披露);demo 无上市日 → 披露 no-op。 - - 回测 `feasibility_log()` 记录 blocked buys/sells/carried/executed turnover/invested;phase2 报告有 **Execution feasibility** 小节。 -- ✅ **Phase 2-3 历史 PIT 行业**(**PR #5 已 merge**,默认 SW-L1 可配置):把行业中性化协变量从 `stock_basic.industry` 当前标签升级为按 trade_date as-of 的历史申万行业。 - - `tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 的 `in_date`/`out_date` 区间;`data/clean/pit_industry.py::asof_industry` 按 `[in_date, out_date)` 覆盖该日取行业,改分类日新行业生效,PIT-safe,起始日前成分 carry forward 到窗口开始。 - - **SW 层级可配置** `processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,中性化标准 + 小截面自由度更稳)。**关键实证**:旧 tag 年化 −17.6% / SW-L1 −10.2% / SW-L2 −9.3% → **L1≈L2**,−17.6→−10 大跳主因是 **tushare→SW 分类切换**(补 PIT 必然:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**。(曾误判粒度、默认 L2,经 L2 实测推翻,改默认 L1。) - - 不静默退回 current:无 SW 历史的票 → 行业 NaN,被 neutralize 按截面丢弃;每次运行 **实际 level + PIT 覆盖率**进 phase2 报告。 - - 真实 baseline(SW-L1,默认):`run-phase2-baseline` OK,symbols=68,settled=11,**PIT SW-L1 coverage 98.53%**(67/68),IC 0.0083,**annual −10.19%**。 +- ✅ **Phase 2-1 真实数据可复现基准**(**PR #2 已 merge 到 `main`**):新 run mode `run-phase2-baseline` + `config/phase2_real_baseline.yaml`(上证50 `000016.SH`,2023-07~2024-06)。**复用 P0/P1 全套机器,不扩因子、不调参**。一次真实跑 ~11min(68 成分 / 25 loaded 快照 / in-window distinct 60 / 11 settled 调仓,候选 12 末日跳过),输出 `artifacts/reports/phase2_real_baseline.md`(gitignored):数据窗口 / PIT 成分摘要(loaded vs in-window) / ann_date 覆盖率(100%) / 可交易过滤命中(首命中互斥) / 每期持仓 / 换手成本 / IC(≈0.008) / 绩效(年化−17.6%,**亏损动量基准,非业绩声明**) / 全部 P2 降级。诊断只读、demo 源拒绝、token 不入报告。 +- ✅ **Phase 2-2 执行真实性**(PR #3 已 merge 到 `main`):**拆分 selection(选谁)与 execution feasibility(能否成交)**。 + - 方向感知执行 `runtime/fills.py::simulate_fills`:涨停挡买 / 跌停挡卖 / 停牌·缺收盘双向挡;按 panel flag 实时判定,与选股 toggle 无关。 + - **现金一致 sell-then-buy**:卖在前释放现金、买在后,现金不足按比例部分成交 → **无杠杆**;被挡交易 carry forward;换手/成本只算实际成交;闲置现金按 driver 的 `cash_return` 计息(BT-007)。 + - `universe.min_listing_days` **真实路径已执行**(`stock_basic.list_date` 富化,买入资格过滤,边界 age==min 放行,缺 list_date 保留并披露);demo 无上市日 → 披露 no-op。 + - 回测 `feasibility_log()`:每调仓期 blocked buys/sells/carried/executed turnover/invested;phase2 报告新增 **Execution feasibility** 小节。 + - **保不变量**:demo 无 flag → 全可成交 → P0/P1 数字不变;无未来函数、PIT/ann_date/real-demo 分离不变。 +- ✅ **Phase 2-3 历史 PIT 行业**(**PR #5 已 merge 到 `main`**,默认 SW-L1 可配置):把行业中性化协变量从 `stock_basic.industry` **当前**标签升级为按 trade_date **as-of** 的历史申万行业。 + - `tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 取每股 SW 行业 `in_date`/`out_date` 区间;`data/clean/pit_industry.py::asof_industry` 按 `[in_date,out_date)` 覆盖该日取行业(改分类日新行业生效,PIT-safe;起始日前成分 carry forward 到窗口开始)。 + - **SW 层级可配置** `processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,中性化标准 + 小截面自由度更稳)。**关键实证**:旧 tag 年化 −17.6% / SW-L1 −10.2% / SW-L2 −9.3% → **L1≈L2**,大跳的主因是 **tushare→SW 分类切换**(补 PIT 的必然:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**。(注:曾误以为是粒度、默认 L2,经 L2 实测推翻,改默认 L1。) + - **不静默退回 current**:无 SW 历史的票 → 行业 NaN,被 neutralize 按截面丢弃(neutralize 数学不变,本就丢 NaN 行);每次运行 **实际 level + PIT 覆盖率**进 phase2 报告。 - ✅ **Phase 2-4 标准分析集成**(**PR #7 已 merge 到 `main`**):alphalens-reloaded + quantstats 接入报告(**report-only cross-check**)。 - - `analytics/alphalens_adapter.py`(IC mean/IR + 分位均值)+ `analytics/quantstats_adapter.py`(CAGR/Sharpe/maxDD/vol)薄 adapter,各带 `backend` 字段;`_import_*` 间接层使 import-missing 路径可测;alphalens stdout/warnings 已抑制。 - - **简版仍权威**(驱动回测 + cross-check);依赖不可用/报错 → 报告披露 backend(unavailable/error,只记异常类型),**绝不静默假装**。 - - **不改 alpha/portfolio/runtime/fills/universe**:P0/P2 交易数字不变(demo ic 0.96/annual 0.84 不变;alphalens IC=简版 IC 吻合),只新增 **Standard analytics** 报告段。 -- ✅ **Phase 3-1 首个真实多因子 baseline**(**PR #9 已 merge 到 `main`**):pipeline 消费**全部 enabled factors**(曾只用第一个);`config/phase3_real_multifactor.yaml` = momentum_20 + roe + netprofit_yoy(SSE50 同窗口,与 phase2 可比;不调参、无 learned weights、非收益承诺)。 - - 财务字段**一次 fetch + 一次 as-of 对齐**;合成仍 EqualWeightAlpha 等权;`drop_missing` 要求全因子齐备(披露);demo+财务因子仍可读报错;重名因子报错。 - - 报告:active factor list / per-factor coverage+IC+分位 / combo score 诊断 / 财务 coverage 按字段(TRADED vs diagnostic);`output.baseline_report_name` 分离 phase3 报告。 - - 真实结果:多因子 annual **−9.05%**(单因子 −10.19%);momentum_20 IC 0.0083(跨 run 一致)/ roe 0.0006 / netprofit_yoy 0.0001 / combo −0.0038;财务 ann_date 覆盖 100%;PIT SW-L1 98.53%。回归不破:phase2 rerun −10.19%/0.0083 不变,demo 0.96/0.84 不变。 -- 🔧 **Phase 3-2 walk-forward IC 加权 alpha**(`p3-ic-weighted-alpha` 分支,代劳待验收):`alpha/ic_weight.py::RollingICWeightAlpha`(`alpha.model: ic_weighted`);EqualWeightAlpha 仍默认+回归基线;不调参、非收益声明。 - - Lookahead 边界测试锁定:(factor[t], fwd_h[t]) 仅**已实现**(`t+h <= d` 交易日序)进权重;扰动未实现 fwd 权重不变;fwd 只进 `alpha.fit`,factors 层绝不接触。 - - rolling(默认 60d/min20)/expanding;历史不足→该日退回等权(计数披露);权重 L1 归一化、保留符号。报告新增 **Alpha model** 必含小节(模型/超参/覆盖率/每期权重表/fallback/非调参声明)。 - - `config/phase3_real_ic_weighted.yaml` 与 phase3_real_multifactor 唯一差异 alpha.model(直接可比)。 - - 真实结果:annual **−3.57%**(等权 −9.05%),训练覆盖 201/221(20 fallback 全在窗口攒满前)。⚠️ 优于等权非业绩声明——单年窗口+权重逐期翻号即小样本不稳定,照实披露。回归不破:等权 rerun −9.05%/0.0083 不变,demo 0.96/0.84 不变。 -- ✅ 当前质量门:`pytest -p no:cacheprovider` **269 passed**;`ruff` clean;`validate-config`(demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`)OK;`run-phase0`(demo)OK。 -- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据。 -- 路线图下一步:样本外/IC 稳定性检验 / 分钟级(architecture.html §11)。 + - `analytics/alphalens_adapter.py`(IC mean/IR + 分位均值)+ `analytics/quantstats_adapter.py`(CAGR/Sharpe/maxDD/vol)薄 adapter,各带 `backend` 字段。 + - **简版 numpy/pandas 仍权威**(驱动回测 + cross-check);依赖不可用/报错 → 报告显式披露 backend(unavailable/error,只记异常**类型**不记消息),**绝不静默假装用了标准库**。 + - **不改 alpha/portfolio/runtime/fills/universe**:P0/P2 交易数字不变(demo ic 0.96/annual 0.84 不变;实测 alphalens IC=简版 IC 完全吻合),只新增 **Standard analytics** 报告段。 +- ✅ **Phase 3-1 首个真实多因子 baseline**(**PR #9 已 merge 到 `main`**):pipeline 从"只用第一个 enabled factor"升级为**消费全部 enabled factors**;新增 `config/phase3_real_multifactor.yaml`(momentum_20 + roe + netprofit_yoy,SSE50 同窗口,与 phase2 可比)。**不调参、无 learned weights、非收益承诺**。 + - `_build_factors` 全量实例化(配置序;重名报错);财务字段**一次 fetch + 一次 as-of 对齐**(逐字段独立守 `ann_date <= trade_date`,无每因子重复拉取);demo+财务因子仍可读报错。 + - 合成仍 `EqualWeightAlpha`(处理后各列等权平均,不看 forward returns);`drop_missing` 要求该日该票**所有**因子齐备(显式披露)。 + - 报告增强:active factor list / per-factor coverage+IC+分位 / **combo score** 诊断 / 财务 coverage **按字段**披露(TRADED vs diagnostic 角色标注);`output.baseline_report_name` 使 phase3 报告独立于 phase2。 + - **真实结果**(SSE50 2023-07~2024-06,~14min):多因子 annual **−9.05%**(单因子 −10.19%);per-factor IC: momentum_20 0.0083(与 phase2 run 完全一致,跨 run 一致性实证)/ roe 0.0006 / netprofit_yoy 0.0001;combo IC −0.0038;财务两字段 ann_date 覆盖 **100%**;PIT SW-L1 98.53%。财务因子在该小截面短窗口 IC≈0,照实披露——这是 plumbing 验证。 + - **回归不破**:phase2 单因子真实 rerun annual −10.19% / IC 0.0083 不变;demo ic 0.96/annual 0.84 不变。 +- ✅ **Phase 3-2 walk-forward IC 加权 alpha**(**PR #11 已 merge 到 `main`**):新增 `alpha/ic_weight.py::RollingICWeightAlpha`(`alpha.model: ic_weighted`);EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。 + - **Lookahead 边界(测试锁定)**:训练严格 walk-forward——(factor[t], fwd_h[t]) 只有**已实现**(交易日序 `t+h <= d`)才进日 d 的权重;**扰动未实现 forward returns 权重不变**(扰动测试)+ `t+h` 切片精确边界测试。forward returns 由 pipeline 在 alpha 边界计算、只传 `alpha.fit`,factors 层照旧绝不接触(不变量 #1)。 + - rolling(默认保守,窗口=60 交易日,min_periods=20)/ expanding 可配;历史不足 → 该日**退回等权**(与 EqualWeightAlpha 逐 bit 一致)并计数披露;权重 L1 归一化、保留符号(负 IC 因子负权重)。 + - 报告新增 **Alpha model** 必含小节:active model / 超参 / 训练覆盖率 + fallback 次数 / 每调仓日生效权重表(fallback 行标注)/ 非调参声明 + 等权基线对比指引。 + - `config/phase3_real_ic_weighted.yaml`:与 phase3_real_multifactor **唯一差异是 alpha.model**(universe/window/因子/中性化全同,直接可比)。 + - **真实结果**(SSE50 2023-07~2024-06,~14min):annual **−3.57%**(等权 −9.05% / 单因子 −10.19%),maxDD −12.93%,训练覆盖 **201/221**(20 个 fallback 全在窗口攒满前,90.95%)。⚠️ 优于等权**不是**业绩声明——单年窗口 + 权重逐期翻号(如 momentum_20 从 −0.58 到 +0.36)正是小样本不稳定的体现,照实披露。 + - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变(测试锁定)。 +- 🔧 **Phase 3-3 OOS 稳定性验证**(**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并):**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`(`qt/oos_stability.py`)+ `config/phase3_real_oos_stability.yaml`(SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照)。 + - **一次数据加载、同一 processed 因子面板、两次回测**(equal_weight vs ic_weighted);所有诊断按 split 切段(子段 nav 重新归一,绝不跨段串味)。 + - **边界语义(测试锁定)**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。**绩效切片按持有窗口**(train 行持有期 end≤split、test 行 start≥split,跨界调仓从两段排除并披露;IC 按实现日 t+h 切)——绝不按 signal date 单切(review HIGH 修复:旧切法把跨界持有期的 test 收益记进 train);runner 强制 `alpha.model: ic_weighted`(否则假对比,可读报错)。 + - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 + - **真实结果(关键发现,~16min,77 成分/2 年,持有窗口切片)**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** + - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 +- ✅ 质量门:`pytest` **285 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16);`ruff` clean;`validate-config`(demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`)+ `run-phase0`(demo)均 OK。 +- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据、因子 IC 小样本不稳定(P3-3 实证)。 +- 路线图下一步:更长历史/更宽 universe 的稳定性复检,或分钟级(architecture.html §11)。 diff --git a/CLAUDE.md b/CLAUDE.md index 0987dcd..969931d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**;代码/注释/commit message 用**英文**。 -- **Git**:feature 分支 + PR。**PR #1(P0+P1)、#2(P2-1)、#3(P2-2)、#4(进度文档)、#5(P2-3)、#6(进度文档)、#7(P2-4)、#8(进度文档)、#9(P3-1)均已 merge 到 `main`**。commit 用 conventional 格式,**无 attribution**(不加 Co-Authored-By)。 +- **Git**:feature 分支 + PR。**PR #1(P0+P1)、#2(P2-1)、#3(P2-2)、#4(进度文档)、#5(P2-3)、#6(进度文档)、#7(P2-4)、#8(进度文档)、#9(P3-1)、#10(进度文档)、#11(P3-2)均已 merge 到 `main`**。commit 用 conventional 格式,**无 attribution**(不加 Co-Authored-By)。 - **不过度设计**:按路线图 MVP 先打通一条端到端链路,再加层(architecture.html §11,Phase 0→3)。 - **secrets** 一律走外部 `.config.json`;repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`(仅留架构文档)。 - 文件小而专(<800 行),immutable 优先。 @@ -94,13 +94,19 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 报告增强:active factor list / per-factor coverage+IC+分位 / **combo score** 诊断 / 财务 coverage **按字段**披露(TRADED vs diagnostic 角色标注);`output.baseline_report_name` 使 phase3 报告独立于 phase2。 - **真实结果**(SSE50 2023-07~2024-06,~14min):多因子 annual **−9.05%**(单因子 −10.19%);per-factor IC: momentum_20 0.0083(与 phase2 run 完全一致,跨 run 一致性实证)/ roe 0.0006 / netprofit_yoy 0.0001;combo IC −0.0038;财务两字段 ann_date 覆盖 **100%**;PIT SW-L1 98.53%。财务因子在该小截面短窗口 IC≈0,照实披露——这是 plumbing 验证。 - **回归不破**:phase2 单因子真实 rerun annual −10.19% / IC 0.0083 不变;demo ic 0.96/annual 0.84 不变。 -- 🔧 **Phase 3-2 walk-forward IC 加权 alpha**(`p3-ic-weighted-alpha` 分支,代劳待验收):新增 `alpha/ic_weight.py::RollingICWeightAlpha`(`alpha.model: ic_weighted`);EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。 +- ✅ **Phase 3-2 walk-forward IC 加权 alpha**(**PR #11 已 merge 到 `main`**):新增 `alpha/ic_weight.py::RollingICWeightAlpha`(`alpha.model: ic_weighted`);EqualWeightAlpha 仍默认 + 回归基线。**不调参、非收益声明**。 - **Lookahead 边界(测试锁定)**:训练严格 walk-forward——(factor[t], fwd_h[t]) 只有**已实现**(交易日序 `t+h <= d`)才进日 d 的权重;**扰动未实现 forward returns 权重不变**(扰动测试)+ `t+h` 切片精确边界测试。forward returns 由 pipeline 在 alpha 边界计算、只传 `alpha.fit`,factors 层照旧绝不接触(不变量 #1)。 - rolling(默认保守,窗口=60 交易日,min_periods=20)/ expanding 可配;历史不足 → 该日**退回等权**(与 EqualWeightAlpha 逐 bit 一致)并计数披露;权重 L1 归一化、保留符号(负 IC 因子负权重)。 - 报告新增 **Alpha model** 必含小节:active model / 超参 / 训练覆盖率 + fallback 次数 / 每调仓日生效权重表(fallback 行标注)/ 非调参声明 + 等权基线对比指引。 - `config/phase3_real_ic_weighted.yaml`:与 phase3_real_multifactor **唯一差异是 alpha.model**(universe/window/因子/中性化全同,直接可比)。 - **真实结果**(SSE50 2023-07~2024-06,~14min):annual **−3.57%**(等权 −9.05% / 单因子 −10.19%),maxDD −12.93%,训练覆盖 **201/221**(20 个 fallback 全在窗口攒满前,90.95%)。⚠️ 优于等权**不是**业绩声明——单年窗口 + 权重逐期翻号(如 momentum_20 从 −0.58 到 +0.36)正是小样本不稳定的体现,照实披露。 - **回归不破**:phase3 等权真实 rerun annual −9.05% / IC 0.0083 不变;demo equal_weight ic 0.96/annual 0.84 不变(测试锁定)。 -- ✅ 质量门:`pytest` **269 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18);`ruff` clean;`validate-config`(demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`)+ `run-phase0`(demo)均 OK。 -- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据。 -- 路线图下一步:样本外/IC 稳定性检验 / 分钟级(architecture.html §11)。 +- 🔧 **Phase 3-3 OOS 稳定性验证**(**PR #12 OPEN**,review 两 HIGH+一 LOW 已修,待验收/合并):**报告型验证层**,不加新 alpha 复杂度、不改 portfolio/execution/factor math。新 run mode `run-phase3-oos`(`qt/oos_stability.py`)+ `config/phase3_real_oos_stability.yaml`(SSE50 扩到 **2 年** 2022-07~2024-06,split 2023-07-01 → train 1y / test 1y,test 年=旧 baseline 窗口可对照)。 + - **一次数据加载、同一 processed 因子面板、两次回测**(equal_weight vs ic_weighted);所有诊断按 split 切段(子段 nav 重新归一,绝不跨段串味)。 + - **边界语义(测试锁定)**:walk-forward(rolling subperiod)——任何日期的权重只用该日已实现观测(`t+h <= d`);**扰动 split 后全部 forward returns,train 期所有日期权重逐 bit 不变**(split 无泄漏测试);不用 freeze-at-split(那是新 alpha 模式,超范围)。**绩效切片按持有窗口**(train 行持有期 end≤split、test 行 start≥split,跨界调仓从两段排除并披露;IC 按实现日 t+h 切)——绝不按 signal date 单切(review HIGH 修复:旧切法把跨界持有期的 test 收益记进 train);runner 强制 `alpha.model: ic_weighted`(否则假对比,可读报错)。 + - 报告 `phase3_oos_stability.md`:split 边界+跨界行披露/分期绩效(annual/vol/sharpe/maxDD/turnover)/逐序列 IC 分期(mean/IR/hit rate/sign consistency)/权重稳定性(每期权重含 train-test 标注、trained 行 sign flips、fallback 次数+原因)/小样本 caveat。 + - **真实结果(关键发现,~16min,77 成分/2 年,持有窗口切片)**:三个原始因子 train→test **IC 全部翻号**(momentum −0.023→+0.006 / roe −0.029→+0.007 / np_yoy −0.011→+0.005,sign consistency 全 NO),hit rate 46~53%≈抛硬币;权重 23 期 sign flips 7/3/4;绩效 eq train −11.92%/test −5.27%,ic train −8.31%/test −2.70%(跨界行 2023-06-30 排除;修切片前 train 被 test 期收益污染到 −6.81%/−1.69%,修正幅度本身就是边界 bug 的实证)。**结论:ic_weighted 两段都略好但 IC≈0 且翻号——P3-2 单年跑赢不可外推,这正是本验证层要拿到的证据;非收益声明。** + - **回归不破**:phase3 equal_weight rerun −9.05%/0.0083、ic_weighted rerun −3.57% 均不变;demo 0.96/0.84 不变;secret scan 报告 0 处 token/config.json。 +- ✅ 质量门:`pytest` **285 passed**(P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16);`ruff` clean;`validate-config`(demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`)+ `run-phase0`(demo)均 OK。 +- ⚠️ 剩余(已显式披露):日线 only、demo 路径非真数据、因子 IC 小样本不稳定(P3-3 实证)。 +- 路线图下一步:更长历史/更宽 universe 的稳定性复检,或分钟级(architecture.html §11)。 diff --git a/RUNBOOK.md b/RUNBOOK.md index e4f0684..f269429 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -262,6 +262,52 @@ The baseline report gains an **Alpha model** section: active model, hyper-params, training coverage (fallback count), and the effective weights at every settled rebalance date (fallback rows flagged). +## Phase 3-3 — OOS stability validation (equal_weight vs ic_weighted) + +A REPORT-ONLY validation layer (no portfolio / execution / factor-math change): +one shared data load, the SAME processed factor panel, two backtests — one per +alpha — and every diagnostic split at `oos.split_date`. Documented by +`config/phase3_real_oos_stability.yaml` (SSE50, two years 2022-07 ~ 2024-06, +split 2023-07-01 → train 1y / test 1y; the test year equals the phase3 +baselines' window, so numbers line up). + +```bash +# validate (no network) +... -m qt.cli validate-config --config config/phase3_real_oos_stability.yaml +# run the OOS validation (network + token; heavy, ~15-30 min) +... -m qt.cli run-phase3-oos --config config/phase3_real_oos_stability.yaml +``` + +Split semantics (locked by tests): + +- train = `[data.start, split)`, test = `[split, data.end]`; the exact realized + dates and day counts are written into the report. +- Evaluation is **walk-forward (rolling subperiod)**: weights at any date d use + only observations REALIZED by d (`t + horizon <= d`) — perturbing every + post-split forward return cannot change any train-period date's weights + (split-boundary no-leakage test). Freezing weights at the split is NOT used + (that would be a new alpha mode; P3-3 adds no alpha complexity). +- **Performance slicing is HOLDING-WINDOW aware**: a nav row is indexed by its + rebalance (signal) date but its return covers [that rebalance, the next one] — + so train rows must have their holding END on/before the split, test rows their + holding START on/after it, and a straddling rebalance is EXCLUDED from both + subperiods and disclosed in the report. IC stats are sliced by the realization + date (`t + horizon`) the same way. The test subperiod therefore starts with + the first post-split holding period — the same place the 1-year phase3 + baselines start. +- The config's `alpha` section carries the ic_weighted params and MUST set + `alpha.model: ic_weighted` (guarded: any other model is refused — running + equal_weight twice and labelling one leg ic_weighted would be a fake + comparison); the `equal_weight` control leg is built internally. + +Report (`artifacts/reports/phase3_oos_stability.md`): split boundaries; +per-subperiod performance for both models (annual / vol / Sharpe / maxDD / +turnover / rebalances); per-series IC stability (mean / IR / hit rate / n + +train-vs-test sign consistency) for every raw factor and both combo scores; +ic_weighted weight stability (per-rebalance weights with train/test labels, +sign-flip counts on trained rows, fallback count + reasons); and the explicit +caveat that this is a small-sample stability check, NOT a return claim. + ## Quality gate ```bash diff --git a/TEST_REPORT.md b/TEST_REPORT.md index 53a764e..4e1874a 100644 --- a/TEST_REPORT.md +++ b/TEST_REPORT.md @@ -1,4 +1,4 @@ -# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha) +# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 + Phase 3 (bias-boundary → execution realism → PIT industry → standard analytics → multi-factor → walk-forward IC alpha → OOS stability) ## Commands @@ -12,6 +12,7 @@ Run from the repo root with the project python (env `quant_mf`): /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase2_real_baseline.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_multifactor.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_ic_weighted.yaml +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase3_real_oos_stability.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli run-phase0 --config config/example.yaml ``` @@ -19,12 +20,12 @@ Run from the repo root with the project python (env `quant_mf`): | Gate | Command | Result | |---|---|---| -| Unit + integration | `pytest -q` | **269 passed, 0 failed** | +| Unit + integration | `pytest -q` | **285 passed, 0 failed** | | Lint | `ruff check .` | **All checks passed** | -| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml`) | exit `0`, prints `OK` | +| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml`) | exit `0`, prints `OK` | | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | -Counts below are the actual per-file `pytest` numbers (sum = 269). +Counts below are the actual per-file `pytest` numbers (sum = 285). ## Per-file breakdown — Phase 0 core (97) @@ -109,7 +110,13 @@ Counts below are the actual per-file `pytest` numbers (sum = 269). |---|---|---| | `test_ic_weight_alpha.py` | 12 | **lookahead red-line**: perturbing unrealized forward returns cannot change weights; exact `t + h <= d` realization cutoff (min_periods boundary); insufficient-history equal-weight fallback (== EqualWeightAlpha row mean); single-factor degeneration to ±1; L1 normalization + sign preservation; degenerate-IC fallback; rolling-vs-expanding window; fit requires forward_returns; dated-cross-section contract; input immutability; weights/fallback log | | `test_ic_alpha_pipeline.py` | 6 | alpha dispatch by config (equal_weight / ic_weighted + params / unknown = ConfigError), equal-weight default keeps exact demo numbers (ic 0.96 / annual 0.84) + report line, ic_weighted demo e2e (summary, weights log, early-fallback→late-trained, L1 rows, report disclosure, no secret), ic-weights differ from equal weight on diverging-IC synthetic data | -| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2)** | **269** | | + +## Per-file breakdown — Phase 3-3 OOS stability (16) + +| Test file | Tests | Red-line / feature | +|---|---|---| +| `test_oos_stability.py` | 16 | **split-boundary no-leakage**: perturbing every post-split forward return leaves all train-period weights bit-identical; **holding-window slicing** (`split_nav_by_holding`: train = holding end ≤ split, test = start ≥ split, straddlers + unknown-end rows excluded and disclosed); `subperiod_perf` rebased nav + empty-slice NaN; `ic_period_stats` sliced by realization date (t+h) with mean/IR/hit-rate/n; `sign_consistent` nonzero-same-sign; `weight_sign_flips` on trained rows only (fallback rows excluded); fallback-reason aggregation; OOS config validates + split-inside-window ConfigError; runner rejects demo source / missing `oos` section / **non-ic_weighted alpha (fake-comparison guard)**; report renders boundaries / straddler disclosure / OOS metrics / weight stability / caveat / no secret | +| **Total (P0 + P1 + P2-1..P2-4 + P3-1 + P3-2 + P3-3)** | **285** | | ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) @@ -182,6 +189,17 @@ Counts below are the actual per-file `pytest` numbers (sum = 269). EqualWeightAlpha combination — and is counted in the report. Weights are L1-normalized, sign-preserving. `EqualWeightAlpha` remains the default; its demo numbers (ic 0.96 / annual 0.84) are locked unchanged. +- **P3-3 OOS stability (locked by tests):** `run-phase3-oos` is a REPORT-ONLY + validation layer — one shared data load, two backtests (equal_weight vs + ic_weighted), every diagnostic split at `oos.split_date`. Performance slicing + is HOLDING-WINDOW aware (a nav row's return covers [rebalance, next + rebalance]: train rows END on/before the split, test rows START on/after it, + straddlers are excluded from both and disclosed); IC stats slice by the + realization date (t+h); subperiod navs are rebased so nothing bleeds across. + Evaluation is walk-forward (rolling subperiod): the split-boundary test + proves post-split forward returns cannot move any train-period weight. The + runner refuses a non-ic_weighted `alpha.model` (a fake comparison guard). + Portfolio / execution / factor math are untouched. - A duplicate test-function name across two files was found and renamed during P2-2 (it had been silently shadowing one test in the full-suite run). A second, harmless duplicate (`test_enrich_does_not_mutate_input` in two files) was verified NOT to drop diff --git a/config/phase3_real_oos_stability.yaml b/config/phase3_real_oos_stability.yaml new file mode 100644 index 0000000..1e5c31b --- /dev/null +++ b/config/phase3_real_oos_stability.yaml @@ -0,0 +1,112 @@ +# Phase 3-3 — REAL out-of-sample stability validation (equal_weight vs ic_weighted). +# +# Validation layer, NOT a new strategy: same SSE50 universe, same three factors, +# same neutralization / portfolio / execution rules as the phase3 baselines — +# the window is EXTENDED to two years and split at oos.split_date: +# train = [data.start, split) test = [split, data.end] +# Evaluation is walk-forward (rolling subperiod): weights at any date d use only +# observations REALIZED by d (t + horizon <= d), so no test-period forward +# return can reach a train-period computation (locked by tests). The alpha +# section carries the ic_weighted params; the run ALSO evaluates equal_weight +# on the same data as the control. NOT a return claim — a small-sample +# stability check. +# +# Run: python -m qt.cli run-phase3-oos --config config/phase3_real_oos_stability.yaml +# (report: artifacts/reports/phase3_oos_stability.md; needs the tushare token in +# the external .config.json; hits the network; heavy ~15-25 min.) + +project: + name: quantitative_trading_phase3_oos_stability + timezone: Asia/Shanghai + +data: + source: tushare + freq: D + start: "2022-07-01" + end: "2024-06-30" + external_secret_file: "/home/shaofl/Projects/financial_projects/.config.json" + tushare_token_key: "tushare.token" + output_name: phase3oos_daily + +universe: + type: index + index_code: "000016.SH" + symbols: [] + min_listing_days: 60 + filters: + missing_close: true + suspended: true + st: true + limit_up_down: true + +factors: + - name: momentum_20 + enabled: true + params: + window: 20 + price_col: close + - name: roe + enabled: true + params: {} + - name: netprofit_yoy + enabled: true + params: {} + +processing: + drop_missing: true + standardize: + enabled: true + method: zscore + winsorize: + enabled: false + method: mad + n: 3.0 + neutralize: + enabled: true + industry_col: industry + size_col: market_cap + industry_level: L1 # PIT SW level (L1/L2/L3); L1 = 31 broad sectors (standard, DOF-safe) + +alpha: + model: ic_weighted # P3-2 walk-forward rolling-IC weights + params: + window: 60 # trailing trading days of REALIZED ICs (conservative rolling) + min_periods: 20 # < this many valid realized ICs -> equal-weight fallback (disclosed) + mode: rolling # rolling (default) | expanding + +portfolio: + constructor: topn_equal_weight + top_n: 20 + long_only: true + max_weight: null + turnover_cap: null + +backtest: + initial_nav: 1.0 + rebalance: monthly + event_order: close_to_next_period + cash_return: 0.0 + +cost: + fee_rate: 0.001 + slippage_rate: 0.0 + turnover_formula: l1 + +analytics: + forward_return_periods: + - 1 + - 5 + - 20 + quantiles: 5 + benchmark: null + +output: + root_dir: artifacts + data_dir: artifacts/data + factor_dir: artifacts/factors + report_dir: artifacts/reports + log_dir: artifacts/logs + overwrite: true + +oos: + split_date: "2023-07-01" # train = 1y [2022-07-01, split); test = 1y [split, 2024-06-30] diff --git a/qt/cli.py b/qt/cli.py index efc8016..18c9d79 100644 --- a/qt/cli.py +++ b/qt/cli.py @@ -91,6 +91,26 @@ def _cmd_run_phase2_baseline(args: argparse.Namespace) -> int: return 0 +def _cmd_run_phase3_oos(args: argparse.Namespace) -> int: + """Run the OOS stability validation (equal_weight vs ic_weighted) + report.""" + from qt.oos_stability import run_phase3_oos + + try: + result = run_phase3_oos(args.config) + except (ConfigError, ValueError, FileNotFoundError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + eq = result.performance["equal_weight"]["test"].get("annual_return", float("nan")) + ic = result.performance["ic_weighted"]["test"].get("annual_return", float("nan")) + print( + f"OK run-phase3-oos: split={result.split_date.date()}, " + f"test annual eq={eq:.4f} ic={ic:.4f}, fallbacks={result.n_fallback} " + f"({result.elapsed_seconds:.1f}s)\n" + f"report: {result.report_path}" + ) + return 0 + + def _cmd_fetch_data(args: argparse.Namespace) -> int: """Stage helper: run the spine and report the data-fetch stage.""" return _run_pipeline_cmd(args.config, "fetch-data") @@ -129,6 +149,13 @@ def build_parser() -> argparse.ArgumentParser: p_p2.add_argument("--config", required=True, help="Path to the YAML config.") p_p2.set_defaults(func=_cmd_run_phase2_baseline) + p_oos = sub.add_parser( + "run-phase3-oos", + help="Run the REAL-data OOS stability validation (equal_weight vs ic_weighted).", + ) + p_oos.add_argument("--config", required=True, help="Path to the YAML config.") + p_oos.set_defaults(func=_cmd_run_phase3_oos) + for name, func, help_text in ( ("fetch-data", _cmd_fetch_data, "Run the spine, report data fetch."), ("compute-factors", _cmd_compute_factors, "Run the spine, report factor compute."), diff --git a/qt/config.py b/qt/config.py index 2df2151..60e97c5 100644 --- a/qt/config.py +++ b/qt/config.py @@ -184,12 +184,29 @@ class OutputCfg(_Strict): baseline_report_name: str | None = None +class OOSCfg(_Strict): + """P3-3 out-of-sample split: train = [data.start, split_date), test = + [split_date, data.end]. Evaluation is walk-forward (rolling subperiod): + weights at any date use only observations realized by that date, so no + test-period forward return can reach a train-period computation.""" + + split_date: str + + @field_validator("split_date", mode="before") + @classmethod + def _coerce_date_to_str(cls, v: Any) -> Any: + if isinstance(v, (_date, datetime)): + return v.strftime("%Y-%m-%d") + return v + + class RootConfig(_Strict): """Top-level config composing every section. Required top-level sections (CFG-002): data, universe, factors, alpha, portfolio, backtest, cost, output. ``project``, ``processing`` and ``analytics`` have sensible defaults but are present in the template. + ``oos`` is optional and only consumed by ``run-phase3-oos``. """ project: ProjectCfg = Field(default_factory=lambda: ProjectCfg(name="quantitative_trading")) @@ -203,6 +220,28 @@ class RootConfig(_Strict): cost: CostCfg analytics: AnalyticsCfg = Field(default_factory=AnalyticsCfg) output: OutputCfg + oos: OOSCfg | None = None + + @model_validator(mode="after") + def _check_oos_split_inside_window(self) -> "RootConfig": + if self.oos is None: + return self + try: + split = datetime.strptime(self.oos.split_date, "%Y-%m-%d") + except ValueError as exc: + raise ValueError( + f"oos.split_date must be a 'YYYY-MM-DD' date; got " + f"{self.oos.split_date!r} ({exc})." + ) from exc + start = datetime.strptime(self.data.start, "%Y-%m-%d") + end = datetime.strptime(self.data.end, "%Y-%m-%d") + if not (start < split < end): + raise ValueError( + f"oos.split_date ({self.oos.split_date}) must lie STRICTLY inside " + f"the data window ({self.data.start}, {self.data.end}) so both the " + "train and test subperiods are non-empty." + ) + return self # --------------------------------------------------------------------------- # diff --git a/qt/oos_stability.py b/qt/oos_stability.py new file mode 100644 index 0000000..70a7489 --- /dev/null +++ b/qt/oos_stability.py @@ -0,0 +1,428 @@ +"""Phase 3-3: out-of-sample stability validation (equal_weight vs ic_weighted). + +A REPORT-ONLY validation layer, not a new strategy: on ONE shared data load +(same universe, same factor panel, same processing / portfolio / execution +rules) it runs the backtest twice — once under :class:`EqualWeightAlpha`, once +under :class:`RollingICWeightAlpha` — and splits every diagnostic at +``oos.split_date``: + + train = [data.start, split) test = [split, data.end] + +Evaluation is WALK-FORWARD (rolling subperiod): the ic-weighted model derives +each date's weights from observations realized by that date only (t + h <= d, +P3-2, locked by tests), so no test-period forward return can reach any +train-period computation — the split here is an ACCOUNTING boundary for the +statistics, not a new training mode. Nothing in portfolio / execution / factor +math changes. + +The report (``phase3_oos_stability.md``) carries: the exact split dates, +per-subperiod performance for both models, per-series IC stability (mean / IR / +hit rate / cross-period sign consistency) for every raw factor and both combo +scores, the ic-weighted weight-stability diagnostics (per-rebalance weights, +sign flips, fallback counts + reasons), and the explicit small-sample caveat. +Guard: requires the real tushare source and an ``oos`` config section. +""" + +from __future__ import annotations + +import math +import time +from dataclasses import dataclass +from pathlib import Path + +import pandas as pd + +from alpha.equal_weight import EqualWeightAlpha +from analytics.factor import compute_ic, forward_returns +from analytics.performance import performance_summary +from portfolio.construct import TopNEqualWeight +from qt.config import RootConfig, load_config +from qt.pipeline import ( + _FrameScores, + _alpha_disclosure, + _build_alpha, + _build_factors, + _build_scores, + _build_universe, + _collect_downgrades, + _compute_factor_panel, + _load_panel, + _make_logger, + _maybe_enrich_covariates, + _maybe_enrich_financials, + _maybe_enrich_listing, + _periods_per_year, + _process_factors, +) +from qt.reports import render_oos_stability, write_oos_stability_summary +from runtime.backtest.driver import BacktestDriver +from runtime.backtest.sim_execution import SimExecution + +__all__ = ["OOSResult", "run_phase3_oos", "render_oos_stability", + "split_nav_by_holding", "subperiod_perf", "ic_period_stats", + "sign_consistent", "weight_sign_flips", "fallback_reason_counts"] + +_LOGGER_NAME = "qt.run_phase3_oos" +_PERF_NAN = { + "annual_return": float("nan"), "volatility": float("nan"), + "sharpe": float("nan"), "max_drawdown": float("nan"), + "avg_turnover": float("nan"), "n_rebalances": 0, +} + + +@dataclass(frozen=True) +class OOSResult: + """Immutable summary of one OOS stability run (what the report consumes).""" + + config: RootConfig + elapsed_seconds: float + # split boundaries (actual panel dates, disclosed verbatim in the report) + split_date: pd.Timestamp + train_start: pd.Timestamp | None + train_end: pd.Timestamp | None + test_start: pd.Timestamp | None + test_end: pd.Timestamp | None + n_train_days: int + n_test_days: int + # rebalance dates whose HOLDING WINDOW straddles the split (excluded from + # both subperiods' performance and disclosed in the report). + boundary_dates: tuple[pd.Timestamp, ...] + factor_names: tuple[str, ...] + # performance[model][period] -> {annual_return, volatility, sharpe, + # max_drawdown, avg_turnover, n_rebalances}; model in (equal_weight, + # ic_weighted); period in (train, test). + performance: dict[str, dict[str, dict]] + # ic_stats[series][period] -> {ic_mean, ic_ir, hit_rate, n}; series = every + # raw factor + combo_equal_weight + combo_ic_weighted. + ic_stats: dict[str, dict[str, dict]] + # sign_consistency[series] -> True iff train/test mean ICs share a nonzero sign. + sign_consistency: dict[str, bool] + # ic_weighted weight stability + weights_at_rebalances: pd.DataFrame + sign_flips: dict[str, int] + n_scored: int + n_fallback: int + fallback_reasons: dict[str, int] + alpha_summary: dict + downgrades: tuple[str, ...] + report_path: Path + log_path: Path + + +# --------------------------------------------------------------------------- # +# Pure subperiod statistics (network-free; unit-tested with synthetic inputs). +# --------------------------------------------------------------------------- # +def split_nav_by_holding( + nav_table: pd.DataFrame, split: pd.Timestamp +) -> tuple[pd.DataFrame, pd.DataFrame, list[pd.Timestamp]]: + """(train_rows, test_rows, boundary_dates) sliced by the HOLDING WINDOW. + + A driver nav row is INDEXED by its rebalance (signal) date, but its return + covers the holding window [index[i], index[i+1]] — slicing by the row index + alone would credit a straddling holding period's post-split return to the + train period. Calendar-pure subperiods therefore require: + + train <=> holding END on/before the split (return fully realized + pre-split); + test <=> holding START on/after the split; + boundary <=> the holding window straddles the split (start < split < end, + or the end is unknown — the LAST row's end is the skipped + terminal candidate, not recoverable here — with a pre-split + start). Boundary rows are EXCLUDED from both subperiods and + disclosed in the report. + """ + if nav_table is None or nav_table.empty: + empty = nav_table if nav_table is not None else pd.DataFrame() + return empty, empty, [] + split = pd.Timestamp(split) + starts = [pd.Timestamp(d) for d in nav_table.index] + ends: list[pd.Timestamp | None] = starts[1:] + [None] + train_mask: list[bool] = [] + test_mask: list[bool] = [] + boundary: list[pd.Timestamp] = [] + for start, end in zip(starts, ends): + if start >= split: + train_mask.append(False) + test_mask.append(True) + elif end is not None and end <= split: + train_mask.append(True) + test_mask.append(False) + else: # straddles the split, or unknown end with a pre-split start + train_mask.append(False) + test_mask.append(False) + boundary.append(start) + return nav_table[train_mask], nav_table[test_mask], boundary + + +def subperiod_perf(nav_slice: pd.DataFrame, periods_per_year: int = 12) -> dict: + """Performance of one pre-sliced nav segment — period-local compounding. + + The segment nav is REBASED to 1.0 from its own net returns, so a drawdown + or annualization never bleeds across the split. Empty slices return NaN + metrics (never crash). Slicing itself lives in + :func:`split_nav_by_holding` (holding-window aware). + """ + if nav_slice is None or nav_slice.empty: + return dict(_PERF_NAN) + local_nav = (1.0 + nav_slice["net_return"]).cumprod() + perf = performance_summary(local_nav, periods_per_year=periods_per_year) + return { + **perf, + "avg_turnover": float(nav_slice["turnover"].mean()), + "n_rebalances": int(len(nav_slice)), + } + + +def ic_period_stats( + ic: pd.Series, split: pd.Timestamp, horizon: int = 1 +) -> dict[str, dict]: + """{'train': {...}, 'test': {...}} stats of a per-date IC series. + + The IC at factor date t uses the h-day forward return REALIZED at trading + position pos(t) + h, so (mirroring the nav slicing): + train <=> realization date strictly BEFORE the split; + test <=> factor date on/after the split; + straddlers (t < split <= realization) are excluded from both. + Each period reports the NaN-dropped mean, IR (mean/std, ddof=1), hit rate + (share of positive ICs) and observation count. + """ + split = pd.Timestamp(split) + idx = list(ic.index) + n_idx = len(idx) + realization = [ + idx[pos + horizon] if pos + horizon < n_idx else None + for pos in range(n_idx) + ] + train_mask = [ + r is not None and pd.Timestamp(r) < split for r in realization + ] + test_mask = [pd.Timestamp(t) >= split for t in idx] + out: dict[str, dict] = {} + for name, mask in (("train", train_mask), ("test", test_mask)): + clean = ic[mask].dropna() + n = int(len(clean)) + mean = float(clean.mean()) if n else float("nan") + std = float(clean.std(ddof=1)) if n > 1 else float("nan") + ir = mean / std if (math.isfinite(std) and std != 0) else float("nan") + hit = float((clean > 0).mean()) if n else float("nan") + out[name] = {"ic_mean": mean, "ic_ir": float(ir), "hit_rate": hit, "n": n} + return out + + +def sign_consistent(stats: dict[str, dict]) -> bool: + """True iff the train and test mean ICs share the SAME nonzero finite sign.""" + tr = float(stats.get("train", {}).get("ic_mean", float("nan"))) + te = float(stats.get("test", {}).get("ic_mean", float("nan"))) + if not (math.isfinite(tr) and math.isfinite(te)) or tr == 0.0 or te == 0.0: + return False + return (tr > 0) == (te > 0) + + +def weight_sign_flips(weights_log: pd.DataFrame) -> dict[str, int]: + """Per-factor count of sign changes between CONSECUTIVE TRAINED rows. + + Fallback rows carry all-positive equal weights, so including them would + inject artificial flips — they are excluded here and disclosed separately + (fallback count / reasons). + """ + if weights_log is None or weights_log.empty: + return {} + trained = weights_log.loc[~weights_log["fallback"].astype(bool)] + factor_cols = [c for c in weights_log.columns if c != "fallback"] + flips: dict[str, int] = {} + for col in factor_cols: + signs = trained[col].apply( + lambda v: 0 if (pd.isna(v) or v == 0) else (1 if v > 0 else -1) + ) + signs = signs[signs != 0] + flips[col] = int((signs.diff().fillna(0) != 0).sum()) + return flips + + +def fallback_reason_counts(fallback_log: dict) -> dict[str, int]: + """Aggregate the per-date fallback reasons into reason -> count (None = trained).""" + counts: dict[str, int] = {} + for reason in fallback_log.values(): + if reason is not None: + counts[reason] = counts.get(reason, 0) + 1 + return counts + + +def _oos_downgrades(cfg: RootConfig) -> tuple[str, ...]: + """Base downgrades + the OOS-validation caveats (INV-007).""" + extra = ( + "OOS VALIDATION SEMANTICS: this run evaluates BOTH equal_weight and " + "ic_weighted on one shared data load; the split is an ACCOUNTING boundary " + "for subperiod statistics. Weight training is walk-forward (a pair enters " + "date d's weights only once realized, t + horizon <= d), so no test-period " + "forward return reaches any train-period computation; freezing weights at " + "the split is NOT used (that would be a new alpha mode, out of scope). " + "Subperiod PERFORMANCE is sliced by the HOLDING WINDOW (train rows end " + "on/before the split, test rows start on/after it; a straddling rebalance " + "is excluded from both and disclosed) and IC stats by the realization " + "date — never by the row's signal date alone.", + "This is a SMALL-SAMPLE stability check (one index, two years, ~22 " + "rebalances), NOT a return claim and NOT a tuned result: subperiod " + "metrics carry wide uncertainty and must not be read as expected " + "performance.", + ) + return _collect_downgrades(cfg) + extra + + +# --------------------------------------------------------------------------- # +# Orchestration +# --------------------------------------------------------------------------- # +def _run_backtest_for(cfg: RootConfig, panel, universe, score_panel) -> pd.DataFrame: + """One backtest run for a given score panel (same rules for both models).""" + driver = BacktestDriver( + universe=universe, + scores=_FrameScores(score_panel), + constructor=TopNEqualWeight(cfg.portfolio.top_n, long_only=cfg.portfolio.long_only), + execution=SimExecution(fee_rate=cfg.cost.fee_rate), + prices=panel, + rebalance=cfg.backtest.rebalance, + fee_rate=cfg.cost.fee_rate, + initial_nav=cfg.backtest.initial_nav, + cash_return=cfg.backtest.cash_return, + ) + return driver.run() + + +def run_phase3_oos(config_path: str) -> OOSResult: + """Run the OOS stability validation and write the phase3 OOS report. + + Requires ``data.source='tushare'`` (a stability check on offline demo data + would be a category error) and an ``oos`` config section (the split date). + All writes land under the configured output dirs; no secret is echoed. + """ + cfg = load_config(config_path) + if cfg.data.source != "tushare": + raise ValueError( + "run-phase3-oos is a REAL-data validation and requires " + "data.source='tushare'; the demo path carries no PIT/ann_date meaning. " + f"Got data.source={cfg.data.source!r}." + ) + if cfg.oos is None: + raise ValueError( + "run-phase3-oos requires an 'oos' config section with split_date " + "(train = [data.start, split), test = [split, data.end])." + ) + if cfg.alpha.model != "ic_weighted": + raise ValueError( + "run-phase3-oos compares equal_weight vs ic_weighted: the config's " + "alpha section must set model='ic_weighted' (it carries the " + "ic-weighted leg's params; the equal-weight control is built " + f"internally). Got alpha.model={cfg.alpha.model!r} — running that " + "would silently label an equal_weight leg as ic_weighted." + ) + split = pd.Timestamp(cfg.oos.split_date) + + t0 = time.perf_counter() + log_path = Path(cfg.output.log_dir) / "run_phase3_oos.log" + logger = _make_logger(log_path, name=_LOGGER_NAME) + logger.info("phase3 oos start: project=%s split=%s", cfg.project.name, split.date()) + + # --- ONE shared data load / factor panel (identical inputs for both) ----- # + universe, symbols = _build_universe(cfg, logger) + panel = _load_panel(cfg, symbols, logger) + factors = _build_factors(cfg) + panel = _maybe_enrich_financials(cfg, panel, symbols, factors, logger) + panel = _maybe_enrich_covariates(cfg, panel, symbols, logger) + panel = _maybe_enrich_listing(cfg, panel, symbols, logger) + factor_panel = _compute_factor_panel(cfg, panel, factors, logger) + processed = _process_factors(cfg, factor_panel, panel) + + horizon = int(cfg.analytics.forward_return_periods[0]) + fwd_col = forward_returns(panel, periods=(horizon,))[f"forward_return_{horizon}d"] + + # --- both alphas on the SAME processed panel ----------------------------- # + eq_alpha = EqualWeightAlpha() + eq_scores = _build_scores(processed, eq_alpha) + ic_alpha = _build_alpha(cfg) # config carries the ic_weighted params + ic_scores = _build_scores(processed, ic_alpha, fwd_col) + alpha_summary, weights_log = _alpha_disclosure(cfg, ic_alpha) + logger.info("alpha (ic_weighted leg): %s", alpha_summary) + + nav_eq = _run_backtest_for(cfg, panel, universe, eq_scores) + nav_ic = _run_backtest_for(cfg, panel, universe, ic_scores) + logger.info("backtests: equal_weight %d rows, ic_weighted %d rows", + len(nav_eq), len(nav_ic)) + + # --- subperiod statistics (holding-window aware slicing) ------------------ # + ppy = _periods_per_year(cfg.backtest.rebalance) + eq_train, eq_test, eq_boundary = split_nav_by_holding(nav_eq, split) + ic_train, ic_test, ic_boundary = split_nav_by_holding(nav_ic, split) + # both legs share the driver's rebalance calendar -> identical boundary rows + boundary_dates = tuple(sorted(set(eq_boundary) | set(ic_boundary))) + performance = { + "equal_weight": { + "train": subperiod_perf(eq_train, periods_per_year=ppy), + "test": subperiod_perf(eq_test, periods_per_year=ppy), + }, + "ic_weighted": { + "train": subperiod_perf(ic_train, periods_per_year=ppy), + "test": subperiod_perf(ic_test, periods_per_year=ppy), + }, + } + ic_series: dict[str, pd.Series] = { + name: compute_ic(factor_panel[name], fwd_col) for name in factor_panel.columns + } + ic_series["combo_equal_weight"] = compute_ic(eq_scores, fwd_col) + ic_series["combo_ic_weighted"] = compute_ic(ic_scores, fwd_col) + ic_stats = { + name: ic_period_stats(s, split, horizon=horizon) + for name, s in ic_series.items() + } + consistency = {name: sign_consistent(stats) for name, stats in ic_stats.items()} + + # weight stability at the SETTLED rebalance dates of the ic run + settled = list(nav_ic.index) + if weights_log is not None and not weights_log.empty: + wanted = [d for d in settled if d in weights_log.index] + weights_at_reb = weights_log.loc[wanted] + n_scored = int(len(weights_log)) + n_fallback = int(weights_log["fallback"].sum()) + else: # pragma: no cover - ic leg always logs + weights_at_reb = pd.DataFrame() + n_scored = n_fallback = 0 + flips = weight_sign_flips(weights_at_reb) + reasons = fallback_reason_counts( + ic_alpha.fallback_log() if hasattr(ic_alpha, "fallback_log") else {} + ) + + dates = panel.index.get_level_values("date") + train_dates = sorted({d for d in dates if d < split}) + test_dates = sorted({d for d in dates if d >= split}) + result = OOSResult( + config=cfg, + elapsed_seconds=time.perf_counter() - t0, + split_date=split, + train_start=train_dates[0] if train_dates else None, + train_end=train_dates[-1] if train_dates else None, + test_start=test_dates[0] if test_dates else None, + test_end=test_dates[-1] if test_dates else None, + n_train_days=len(train_dates), + n_test_days=len(test_dates), + boundary_dates=boundary_dates, + factor_names=tuple(f.name for f in factors), + performance=performance, + ic_stats=ic_stats, + sign_consistency=consistency, + weights_at_rebalances=weights_at_reb, + sign_flips=flips, + n_scored=n_scored, + n_fallback=n_fallback, + fallback_reasons=reasons, + alpha_summary=alpha_summary, + downgrades=_oos_downgrades(cfg), + report_path=Path(cfg.output.report_dir) / "phase3_oos_stability.md", + log_path=log_path, + ) + write_oos_stability_summary(result) + logger.info( + "phase3 oos done: eq test annual=%.4f ic test annual=%.4f report=%s (%.1fs)", + performance["equal_weight"]["test"].get("annual_return", float("nan")), + performance["ic_weighted"]["test"].get("annual_return", float("nan")), + result.report_path, result.elapsed_seconds, + ) + return result diff --git a/qt/reports.py b/qt/reports.py index e803b68..36c9b4e 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -631,6 +631,165 @@ def write_phase2_baseline_summary(result: "Phase2Result") -> Path: return target +# --------------------------------------------------------------------------- # +# Phase 3-3 OOS stability report (qt.oos_stability). +# --------------------------------------------------------------------------- # +def _oos_perf_table(performance: dict) -> str: + """Both models × both subperiods performance table.""" + header = ( + "| Model | Period | annual | vol | sharpe | maxDD | avg turnover | rebalances |\n" + "|---|---|---|---|---|---|---|---|\n" + ) + rows = "" + for model in ("equal_weight", "ic_weighted"): + for period in ("train", "test"): + p = (performance.get(model) or {}).get(period) or {} + rows += ( + f"| `{model}` | {period} | " + f"{_fmt(p.get('annual_return', float('nan')), pct=True)} | " + f"{_fmt(p.get('volatility', float('nan')), pct=True)} | " + f"{_fmt(p.get('sharpe', float('nan')))} | " + f"{_fmt(p.get('max_drawdown', float('nan')), pct=True)} | " + f"{_fmt(p.get('avg_turnover', float('nan')))} | " + f"{int(p.get('n_rebalances', 0))} |\n" + ) + return header + rows + + +def _oos_ic_table(ic_stats: dict, sign_consistency: dict) -> str: + """Per-series IC stability table (train + test rows, sign-consistency flag).""" + header = ( + "| Series | Period | IC mean | IC IR | hit rate | n | sign consistency |\n" + "|---|---|---|---|---|---|---|\n" + ) + rows = "" + for series, stats in ic_stats.items(): + flag = "YES" if sign_consistency.get(series) else "NO" + for period in ("train", "test"): + p = stats.get(period) or {} + rows += ( + f"| `{series}` | {period} | {_fmt(p.get('ic_mean', float('nan')))} | " + f"{_fmt(p.get('ic_ir', float('nan')))} | " + f"{_fmt(p.get('hit_rate', float('nan')), pct=True)} | " + f"{int(p.get('n', 0))} | " + f"{flag if period == 'train' else ''} |\n" + ) + return header + rows + + +def render_oos_stability(result) -> str: + """Build the phase3 OOS stability markdown (pure; no I/O, no secrets).""" + cfg = result.config + lines: list[str] = [] + lines.append("# Phase 3-3 — OOS Stability Validation (equal_weight vs ic_weighted)\n") + lines.append( + f"Project: **{cfg.project.name}** · source: **{cfg.data.source}** · " + f"ran in **{result.elapsed_seconds:.1f}s**\n" + ) + lines.append( + "\n> **This is NOT a return claim.** A small-sample stability check on one " + "index over two years: it compares the same three factors under " + "equal-weight vs walk-forward IC-weighted combination across a train/test " + "split. Subperiod metrics carry wide uncertainty.\n" + ) + + lines.append("\n## Split boundaries\n") + lines.append( + f"- split_date: **{_date_str(result.split_date)}** (train = " + f"[data.start, split), test = [split, data.end])\n" + f"- train: **{_date_str(result.train_start)} → {_date_str(result.train_end)}** " + f"({result.n_train_days} trading days)\n" + f"- test: **{_date_str(result.test_start)} → {_date_str(result.test_end)}** " + f"({result.n_test_days} trading days)\n" + f"- semantics: evaluation is WALK-FORWARD (rolling subperiod) — weights at " + f"any date d use only observations REALIZED by d (t + horizon <= d), so no " + f"test-period forward return reaches a train-period computation (locked by " + f"tests). The split is an accounting boundary for the statistics below.\n" + f"- performance slicing is HOLDING-WINDOW aware: a rebalance row's return " + f"covers [that rebalance, the next one], so train rows must have their " + f"holding END on/before the split and test rows their holding START on/" + f"after it. IC stats are sliced by the realization date (t + horizon) the " + f"same way.\n" + ) + if result.boundary_dates: + bd = ", ".join(_date_str(d) for d in result.boundary_dates) + lines.append( + f"- straddling rebalance(s) excluded from BOTH subperiods (holding " + f"window crosses the split): {bd}\n" + ) + + lines.append("\n## Config echo\n") + lines.append( + f"- universe: type=`{cfg.universe.type}`, index_code=`{cfg.universe.index_code}`, " + f"top_n=`{cfg.portfolio.top_n}`\n" + f"- factors: `{list(result.factor_names)}` · " + f"neutralize=`{cfg.processing.neutralize.enabled}`\n" + f"- ic_weighted params: {result.alpha_summary}\n" + ) + + lines.append("\n## Subperiod performance (same data, same rules, two alphas)\n") + lines.append(_oos_perf_table(result.performance)) + + lines.append("\n## IC stability (raw factors + both combo scores)\n") + lines.append( + "_hit rate = share of positive daily ICs; sign consistency = train and " + "test mean ICs share one nonzero sign._\n\n" + ) + lines.append(_oos_ic_table(result.ic_stats, result.sign_consistency)) + + lines.append("\n## Weight stability (ic_weighted)\n") + lines.append( + f"- scored dates: **{result.n_scored}** · equal-weight fallbacks: " + f"**{result.n_fallback}**\n" + ) + for reason, count in (result.fallback_reasons or {}).items(): + lines.append(f" - fallback reason ×{count}: {reason}\n") + flips = ", ".join(f"`{k}` {v}" for k, v in (result.sign_flips or {}).items()) + lines.append( + f"- sign flips between consecutive TRAINED rebalance weights " + f"(fallback rows excluded): {flips or '_n/a_'}\n\n" + ) + w = result.weights_at_rebalances + if w is not None and not w.empty: + factor_cols = [c for c in w.columns if c != "fallback"] + lines.append("Effective weights at each settled rebalance date " + "(ic_weighted leg):\n\n") + header = "| Date | period | " + " | ".join(f"`{c}`" for c in factor_cols) + header += " | fallback |\n|" + "---|" * (len(factor_cols) + 3) + "\n" + body = "" + # period labels match the PERFORMANCE slicing: a straddling rebalance + # (holding window crosses the split) is labelled boundary, not train. + boundary = {pd.Timestamp(d) for d in (result.boundary_dates or ())} + for date, row in w.iterrows(): + ts = pd.Timestamp(date) + if ts in boundary: + period = "boundary" + else: + period = "test" if ts >= result.split_date else "train" + cells = " | ".join(_fmt(float(row[c])) for c in factor_cols) + body += ( + f"| {_date_str(date)} | {period} | {cells} | " + f"{'YES' if row['fallback'] else 'no'} |\n" + ) + lines.append(header + body) + + lines.append("\n## DOWNGRADES / caveats (INV-007 — must be disclosed)\n") + for item in result.downgrades: + lines.append(f"- {item}\n") + + lines.append("\n## Artifacts\n") + lines.append(f"- report: `{result.report_path}`\n- log: `{result.log_path}`\n") + return "".join(lines) + + +def write_oos_stability_summary(result) -> Path: + """Render and write the OOS stability report; return the path (SEC-003).""" + target = result.report_path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(render_oos_stability(result), encoding="utf-8") + return target + + # --------------------------------------------------------------------------- # # Repo-root delivery docs (framework spec §10). These describe the run; they # carry no secrets and are regenerable. diff --git a/tests/test_oos_stability.py b/tests/test_oos_stability.py new file mode 100644 index 0000000..14325df --- /dev/null +++ b/tests/test_oos_stability.py @@ -0,0 +1,334 @@ +"""P3-3: OOS stability validation (network-free). + +Locks the validation-layer contract: + * split boundary carries NO leakage: perturbing every forward return realized + AFTER the split cannot change any train-period date's weights (the + walk-forward property restated at the split boundary); + * subperiod statistics are computed strictly within their period; + * weight-stability diagnostics (sign flips on trained rows, fallback counts / + reasons) are correct on synthetic logs; + * the report disclosed the split dates, OOS metrics, weight stability, the + small-sample caveat, and leaks no secret; + * the OOS runner refuses the demo source and a config without an ``oos`` + section; the split date must fall inside the data window. +""" + +from __future__ import annotations + +import math +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest +import yaml + +from alpha.ic_weight import RollingICWeightAlpha +from qt.config import ConfigError, load_config +from qt.oos_stability import ( + fallback_reason_counts, + ic_period_stats, + sign_consistent, + split_nav_by_holding, + subperiod_perf, + weight_sign_flips, +) + +_OOS_CONFIG = str( + Path(__file__).resolve().parents[1] / "config" / "phase3_real_oos_stability.yaml" +) + + +# --------------------------------------------------------------------------- # +# split-boundary no-leakage (the P3-3 red-line) +# --------------------------------------------------------------------------- # +def test_perturbing_post_split_returns_leaves_train_weights_unchanged(): + rng = np.random.default_rng(11) + dates = pd.bdate_range("2023-01-02", periods=120) + syms = [f"S{i}" for i in range(8)] + idx = pd.MultiIndex.from_product([dates, syms], names=["date", "symbol"]) + base = rng.normal(size=len(idx)) + factors = pd.DataFrame({"a": base, "b": rng.normal(size=len(idx))}, index=idx) + fwd = pd.Series(base, index=idx) + + split = dates[60] + h = 1 + model = RollingICWeightAlpha(window=30, min_periods=10, horizon=h) + model.fit(factors, fwd) + train_dates = [d for d in dates if d < split] + before = {d: model.weights_for(d) for d in train_dates} + + # poison EVERY forward return of factor dates on/after the split: none of + # them is realized before the split, so no train-period weight may move. + poisoned = fwd.copy() + mask = factors.index.get_level_values("date") >= split + poisoned[mask] = rng.normal(size=int(mask.sum())) * 1e3 + model2 = RollingICWeightAlpha(window=30, min_periods=10, horizon=h) + model2.fit(factors, poisoned) + for d in train_dates: + w_b, w_a = before[d], model2.weights_for(d) + if w_b is None: + assert w_a is None + else: + pd.testing.assert_series_equal(w_b, w_a) + + +# --------------------------------------------------------------------------- # +# subperiod statistics (pure functions) +# --------------------------------------------------------------------------- # +def _nav_table(): + dates = pd.to_datetime( + ["2023-01-31", "2023-02-28", "2023-03-31", "2023-04-28", "2023-05-31"] + ) + return pd.DataFrame( + { + "net_return": [0.01, -0.02, 0.03, 0.01, -0.01], + "turnover": [1.0, 0.4, 0.5, 0.3, 0.2], + "cost": [0.001] * 5, + "gross_return": [0.011, -0.019, 0.031, 0.011, -0.009], + "nav": (1 + pd.Series([0.01, -0.02, 0.03, 0.01, -0.01])).cumprod().values, + }, + index=pd.Index(dates, name="date"), + ) + + +def test_split_nav_by_holding_assigns_rows_by_holding_window(): + """A row's return covers [index[i], index[i+1]] — the split must respect it. + + With split 2023-03-01: the 2023-01-31 row (held to 2023-02-28) is fully + pre-split -> train; the 2023-02-28 row is held to 2023-03-31, STRADDLING the + split -> excluded from BOTH and disclosed; rows starting on/after the split + -> test. The last row's end is unknown (terminal candidate) but it starts + post-split here, so it is test. + """ + nav = _nav_table() + train, test, boundary = split_nav_by_holding(nav, pd.Timestamp("2023-03-01")) + assert list(train.index) == [pd.Timestamp("2023-01-31")] + assert list(test.index) == [ + pd.Timestamp("2023-03-31"), pd.Timestamp("2023-04-28"), + pd.Timestamp("2023-05-31"), + ] + assert boundary == [pd.Timestamp("2023-02-28")] # straddler disclosed + + +def test_split_nav_by_holding_unknown_end_prestart_is_boundary(): + # split AFTER the last row's start: the last row's holding end is unknown + # (terminal candidate), so it cannot be proven pre-split -> boundary. + nav = _nav_table() + train, test, boundary = split_nav_by_holding(nav, pd.Timestamp("2023-06-15")) + assert pd.Timestamp("2023-05-31") in boundary + assert len(test) == 0 + assert list(train.index) == [ + pd.Timestamp("2023-01-31"), pd.Timestamp("2023-02-28"), + pd.Timestamp("2023-03-31"), pd.Timestamp("2023-04-28"), + ] + + +def test_subperiod_perf_stats_within_slice_only(): + nav = _nav_table() + train, test, _ = split_nav_by_holding(nav, pd.Timestamp("2023-03-01")) + tr = subperiod_perf(train, periods_per_year=12) + te = subperiod_perf(test, periods_per_year=12) + assert tr["n_rebalances"] == 1 and te["n_rebalances"] == 3 + assert tr["avg_turnover"] == pytest.approx(1.0) + assert te["avg_turnover"] == pytest.approx((0.5 + 0.3 + 0.2) / 3) + for key in ("annual_return", "volatility", "sharpe", "max_drawdown"): + assert key in tr and key in te + + +def test_subperiod_perf_empty_slice_is_nan_not_crash(): + nav = _nav_table() + out = subperiod_perf(nav.iloc[0:0], periods_per_year=12) + assert out["n_rebalances"] == 0 + assert math.isnan(out["annual_return"]) + + +def test_ic_period_stats_splits_by_realization_date(): + """IC at factor date t realizes at t+h: train requires realization < split. + + 10 trading days, h=1, split = dates[5]: ICs of t in dates[0..3] realize at + dates[1..4] (< split) -> train; t = dates[4] realizes ON dates[5] (== split, + not strictly before) -> excluded (boundary); t >= dates[5] -> test. + """ + dates = pd.bdate_range("2023-01-02", periods=10) + ic = pd.Series([0.1, 0.2, -0.1, 0.3, 0.1, -0.2, -0.3, -0.1, 0.2, np.nan], + index=dates) + split = dates[5] + stats = ic_period_stats(ic, split, horizon=1) + tr, te = stats["train"], stats["test"] + assert tr["n"] == 4 # dates[0..3] only; dates[4] straddles -> excluded + assert te["n"] == 4 # dates[5..8]; dates[9] is NaN -> dropped + assert tr["ic_mean"] == pytest.approx(np.mean([0.1, 0.2, -0.1, 0.3])) + assert tr["hit_rate"] == pytest.approx(3 / 4) + assert te["hit_rate"] == pytest.approx(1 / 4) + assert math.isfinite(tr["ic_ir"]) and math.isfinite(te["ic_ir"]) + + +def test_sign_consistent_requires_same_nonzero_sign(): + assert sign_consistent({"train": {"ic_mean": 0.02}, "test": {"ic_mean": 0.01}}) + assert sign_consistent({"train": {"ic_mean": -0.02}, "test": {"ic_mean": -0.01}}) + assert not sign_consistent({"train": {"ic_mean": 0.02}, "test": {"ic_mean": -0.01}}) + assert not sign_consistent({"train": {"ic_mean": float("nan")}, + "test": {"ic_mean": 0.01}}) + + +# --------------------------------------------------------------------------- # +# weight stability diagnostics +# --------------------------------------------------------------------------- # +def _weights_log(): + dates = pd.to_datetime( + ["2023-01-31", "2023-02-28", "2023-03-31", "2023-04-28", "2023-05-31"] + ) + return pd.DataFrame( + { + "a": [0.33, 0.5, -0.4, 0.6, -0.2], # trained signs: +, -, +, - -> 3 flips + "b": [0.33, -0.5, -0.6, -0.4, -0.8], # trained signs: -, -, -, - -> 0 flips + "fallback": [True, False, False, False, False], + }, + index=pd.Index(dates, name="date"), + ) + + +def test_weight_sign_flips_counts_trained_rows_only(): + flips = weight_sign_flips(_weights_log()) + # the fallback row (all-positive equal weights) must NOT inject flips + assert flips == {"a": 3, "b": 0} + + +def test_fallback_reason_counts_aggregates(): + log = { + pd.Timestamp("2023-01-31"): "insufficient realized IC history", + pd.Timestamp("2023-02-28"): None, + pd.Timestamp("2023-03-31"): "insufficient realized IC history", + } + counts = fallback_reason_counts(log) + assert counts == {"insufficient realized IC history": 2} + + +# --------------------------------------------------------------------------- # +# config + guards +# --------------------------------------------------------------------------- # +def test_oos_config_validates_with_split_inside_window(): + cfg = load_config(_OOS_CONFIG) + assert cfg.oos is not None + split = pd.Timestamp(cfg.oos.split_date) + assert pd.Timestamp(cfg.data.start) < split < pd.Timestamp(cfg.data.end) + assert cfg.alpha.model == "ic_weighted" # carries the ic params for the run + + +def test_split_outside_window_is_a_config_error(tmp_path): + raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) + raw["oos"]["split_date"] = "2031-01-01" + p = tmp_path / "bad.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + with pytest.raises(ConfigError, match="split_date"): + load_config(str(p)) + + +def test_oos_runner_rejects_demo_source(example_config_path): + from qt.oos_stability import run_phase3_oos + + with pytest.raises(ValueError, match="tushare|REAL"): + run_phase3_oos(example_config_path) + + +def test_oos_runner_rejects_non_ic_weighted_alpha(tmp_path): + """alpha.model=equal_weight would silently produce a FAKE comparison + (equal_weight vs equal_weight labelled ic_weighted) — must be refused.""" + raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) + raw["alpha"] = {"model": "equal_weight", "params": {}} + p = tmp_path / "eq_alpha.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + from qt.oos_stability import run_phase3_oos + + with pytest.raises(ValueError, match="ic_weighted"): + run_phase3_oos(str(p)) + + +def test_oos_runner_requires_oos_section(tmp_path): + raw = yaml.safe_load(Path(_OOS_CONFIG).read_text(encoding="utf-8")) + raw.pop("oos") + p = tmp_path / "no_oos.yaml" + p.write_text(yaml.safe_dump(raw), encoding="utf-8") + from qt.oos_stability import run_phase3_oos + + with pytest.raises(ValueError, match="oos"): + run_phase3_oos(str(p)) + + +# --------------------------------------------------------------------------- # +# report rendering +# --------------------------------------------------------------------------- # +def _synthetic_oos_result(): + from qt.oos_stability import OOSResult + + cfg = load_config(_OOS_CONFIG) + period = {"annual_return": -0.05, "volatility": 0.15, "sharpe": -0.3, + "max_drawdown": -0.12, "avg_turnover": 0.8, "n_rebalances": 11} + ic_p = {"ic_mean": 0.01, "ic_ir": 0.05, "hit_rate": 0.55, "n": 110} + return OOSResult( + config=cfg, + elapsed_seconds=100.0, + split_date=pd.Timestamp("2023-07-01"), + train_start=pd.Timestamp("2022-07-01"), train_end=pd.Timestamp("2023-06-30"), + test_start=pd.Timestamp("2023-07-03"), test_end=pd.Timestamp("2024-06-28"), + n_train_days=240, n_test_days=238, + # one boundary date inside the weights table (2023-05-31) so the + # weights-row labelling is exercised, plus one outside it. + boundary_dates=(pd.Timestamp("2023-05-31"), pd.Timestamp("2023-06-30")), + factor_names=("momentum_20", "roe", "netprofit_yoy"), + performance={ + "equal_weight": {"train": dict(period), "test": dict(period)}, + "ic_weighted": {"train": dict(period), "test": dict(period)}, + }, + ic_stats={ + "momentum_20": {"train": dict(ic_p), "test": dict(ic_p)}, + "combo_equal_weight": {"train": dict(ic_p), "test": dict(ic_p)}, + "combo_ic_weighted": {"train": dict(ic_p), "test": dict(ic_p)}, + }, + sign_consistency={"momentum_20": True, "combo_equal_weight": True, + "combo_ic_weighted": False}, + weights_at_rebalances=_weights_log(), + sign_flips={"a": 3, "b": 0}, + n_scored=221, n_fallback=20, + fallback_reasons={"insufficient realized IC history": 20}, + alpha_summary={"model": "ic_weighted", "window": 60, "min_periods": 20, + "horizon": 1, "mode": "rolling"}, + downgrades=("DATA PATH = REAL tushare: ...",), + report_path=Path("artifacts/reports/phase3_oos_stability.md"), + log_path=Path("artifacts/logs/run_phase3_oos.log"), + ) + + +def test_render_oos_report_disclosed_boundaries_and_metrics(): + from qt.oos_stability import render_oos_stability + + md = render_oos_stability(_synthetic_oos_result()) + # split boundaries written into the report + assert "2023-07-01" in md and "2022-07-01" in md and "2024-06-28" in md + assert "train" in md.lower() and "test" in md.lower() + # walk-forward boundary semantics + holding-window slicing stated + assert "realized" in md.lower() + assert "holding" in md.lower() + assert "2023-06-30" in md # the straddling rebalance is disclosed + # a straddling rebalance row in the WEIGHTS table is labelled boundary, + # matching the performance slicing (not mislabelled train). + assert "| 2023-05-31 | boundary |" in md + # both models' subperiod performance + IC stability + weight stability + assert "equal_weight" in md and "ic_weighted" in md + assert "hit rate" in md.lower() + assert "sign consistency" in md.lower() or "sign-consistency" in md.lower() + assert "sign flip" in md.lower() + assert "fallback" in md.lower() + # the caveat: a stability CHECK, not a return claim + assert "not a" in md.lower() and "claim" in md.lower() + assert "small-sample" in md.lower() or "small sample" in md.lower() + + +def test_render_oos_report_leaks_no_secret(): + from qt.oos_stability import render_oos_stability + + result = _synthetic_oos_result() + md = render_oos_stability(result) + assert result.config.data.external_secret_file not in md + assert "token" not in md.lower()