diff --git a/AGENTS.md b/AGENTS.md index ec3100d..eda87bd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,7 +39,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 数据：tushare - **token**：`/home/shaofl/Projects/financial_projects/.config.json`（key `tushare.token`）。 ⚠️ **绝不打印、绝不写进 repo、绝不 commit。** 代码里从该文件读取，不硬编码。 -- **权限**：实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业(`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上，无需先退回日线。 +- **权限**：实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业历史(`index_member_all`/`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上，无需先退回日线。 - **MCP（可选开发工具）**：`financial_projects/.mcp.json` 有 tushare MCP，仅供开发期交互查数；**从 `financial_projects/` 启动 Codex 才加载**。 - **数据层 ETL 一律用 Python SDK（批量/增量），不要建在 MCP 上。** 注意 tushare 各接口有每分钟调用上限，批量拉取需限流+重试。 @@ -57,14 +57,33 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**；代码/注释/commit message 用**英文**。 -- **Git**：feature 分支 + PR。main 已有骨架；当前在 `data` 分支搭数据层。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **Git**：feature 分支 + PR。**PR #1（P0+P1）、#2（P2-1）、#3（P2-2）、#4（进度文档）均已 merge 到 `main`；PR #5（P2-3）OPEN,已按用户决定改为「默认 SW-L1 可配置」版(industry_level=L1),待用户验收/合并,勿 as-is 合并。** commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 - **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 - **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 - 文件小而专（<800 行），immutable 优先。 ## 当前进度 -- ✅ 7 层骨架 + 架构文档（已在 `main`） -- ✅ Phase 0 MVP（`data` 分支，未提交）：DemoFeed → PanelStore → StaticUniverse → momentum_20 → zscore → EqualWeightAlpha → TopN 等权 → 月度回测（成本/换手）→ IC/绩效报告。 -- ✅ 质量门：`python -m pytest` 93 passed；`python -m ruff check .` clean；`python -m qt.cli run-phase0 --config config/example.yaml` 可复现。 -- ⚠️ P0 显式降级：静态 universe 非 PIT、简版 IC/绩效未走 alphalens/quantstats、`min_listing_days` 配置未生效、持有期末缺价按 0% 结算、tushare 路径接入但未实网验证。 -- 下一步 P1：接真 tushare 日线/复权/PIT 成分/可交易过滤，然后再扩因子和中性化。 +- ✅ 7 层骨架 + 架构文档（`main`） +- ✅ **Phase 0 MVP**（PR #1）：DemoFeed → PanelStore → StaticUniverse → momentum_20 → zscore → EqualWeightAlpha → TopN 等权 → 月度回测（成本/换手）→ IC/绩效报告，单命令可复现。 +- ✅ **Phase 1 偏差边界**（PR #1，全部真数据实证）： + - 前复权（qfq；store 存 raw，内存复权 → batch≡incremental 安全） + - PIT 指数成分（`index_weight` as-of，survivorship-safe；370 天 pre-start 回看 + 90 天分页） + - 可交易过滤（停牌 / ST / 涨跌停；**涨跌停用未复权 raw close** 比 `stk_limit`） + - 财务 `ann_date` 披露日 as-of（绝不按 end_date；500 天 lookback carry forward） + - 行业 + 市值中性化（按 date 截面 OLS 残差；欠定/无自由度截面 → NaN） + - 路径感知降级披露（demo/static vs tushare/index/ann_date，绝不把 demo 当真实验证） +- ✅ 真数据实证（tushare，非 CI）：复权除权日 raw−5.74%→qfq+0.99% / CSI300 全年 24 快照 328 名换手 / ann_date Q1 延后至 04-20 / 中性化 corr −0.617→0。详见 `BIAS_AUDIT.md`、`artifacts/reports/phase1_summary.md`。 +- ✅ **Phase 2-1 真实数据可复现基准**（PR #2 已 merge）：新 run mode `run-phase2-baseline` + `config/phase2_real_baseline.yaml`（上证50 `000016.SH`，2023-07~2024-06）。复用 P0/P1 全套机器，不扩因子、不调参。一次真实跑约 11 min，输出 `artifacts/reports/phase2_real_baseline.md`（gitignored）：数据窗口 / PIT 成分摘要 / ann_date 覆盖率 / 可交易过滤 / 每期持仓 / 换手成本 / IC / 绩效 / 全部 P2 降级。 +- ✅ **Phase 2-2 执行真实性**（PR #3 已 merge）：拆分 selection（选谁）与 execution feasibility（能否成交）。 + - `runtime/fills.py::simulate_fills` 方向感知执行：涨停挡买 / 跌停挡卖 / 停牌·缺收盘双向挡；按 panel flag 实时判定，与选股 toggle 无关。 + - 现金一致 sell-then-buy：卖在前释放现金、买在后，现金不足按比例部分成交 → 无杠杆；被挡交易 carry forward；换手/成本只算实际成交；闲置现金按 driver 的 `cash_return` 计息。 + - `universe.min_listing_days` 真实路径已执行（`stock_basic.list_date` 富化，买入资格过滤，边界 age==min 放行，缺 list_date 保留并披露）；demo 无上市日 → 披露 no-op。 + - 回测 `feasibility_log()` 记录 blocked buys/sells/carried/executed turnover/invested；phase2 报告有 **Execution feasibility** 小节。 +- 🔧 **Phase 2-3 历史 PIT 行业**（PR #5,分支 `p2-pit-industry`,**默认 SW-L1 可配置版,待用户验收/合并**）：把行业中性化协变量从 `stock_basic.industry` 当前标签升级为按 trade_date as-of 的历史申万行业。 + - `tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 的 `in_date`/`out_date` 区间；`data/clean/pit_industry.py::asof_industry` 按 `[in_date, out_date)` 覆盖该日取行业，改分类日新行业生效，PIT-safe，起始日前成分 carry forward 到窗口开始。 + - **SW 层级可配置** `processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,中性化标准 + 小截面自由度更稳)。**关键实证**:旧 tag 年化 −17.6% / SW-L1 −10.2% / SW-L2 −9.3% → **L1≈L2**,−17.6→−10 大跳主因是 **tushare→SW 分类切换**(补 PIT 必然:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**。(曾误判粒度、默认 L2,经 L2 实测推翻,改默认 L1。) + - 不静默退回 current：无 SW 历史的票 → 行业 NaN，被 neutralize 按截面丢弃；每次运行 **实际 level + PIT 覆盖率**进 phase2 报告。 + - 真实 baseline(SW-L1,默认):`run-phase2-baseline` OK,symbols=68,settled=11,**PIT SW-L1 coverage 98.53%**(67/68),IC 0.0083,**annual −10.19%**。 +- ✅ 当前质量门：`pytest -p no:cacheprovider` **223 passed**；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`）OK；`run-phase0`（demo）OK。 +- ⚠️ 剩余 P2（已显式披露）：日线 only、简版 IC/绩效未走 alphalens/quantstats、demo 路径非真数据。 +- 路线图下一步：财务因子组合 / 分钟级 / alphalens·quantstats 接入（architecture.html §11）。 diff --git a/BIAS_AUDIT.md b/BIAS_AUDIT.md index bfc0acb..0da0350 100644 --- a/BIAS_AUDIT.md +++ b/BIAS_AUDIT.md @@ -57,4 +57,6 @@ - 状态: **行业 + 市值中性化已实现(P1)**。 - `factors.process.neutralize.neutralize_by_date`:每个 date 截面把因子对 `[log(market_cap), one-hot(industry)]` 做 OLS,取残差,移除规模与行业暴露。缺行业 / 市值,或**残差自由度 ≤ 0**(名称数 ≤ 1+行业数,饱和拟合会给出无意义的伪 0 残差)时返回 **NaN**,绝不静默乱算;`processing.neutralize` 开启但协变量缺失(如 demo 路径)直接报可读错误。 - 实证:12 只票横跨 4 行业(2024-09-30),corr(原始 momentum, log市值) = -0.617 → 中性化后 -0.000,各行业残差均值 ≈ 0,确认规模/行业暴露被移除。 -- **降级**:行业来自 `stock_basic.industry` 的**当前**行业标签,非按历史时点,故行业中性化带有轻微成分前视(市值 `daily_basic.total_mv` 为逐日真值)。PIT 行业历史是后续项,此降级在此显式披露(INV-007)。 +- **行业 PIT(UNI-010,P2-3,已实现)**:行业协变量从 `stock_basic.industry` 的**当前**标签,升级为按 trade_date **as-of** 的历史申万行业。`data.feed.tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 拿每股 SW 行业的 `in_date`/`out_date` 区间;`data.clean.pit_industry.asof_industry` 按`[in_date, out_date)` 覆盖该日的区间取行业(改分类日**新行业**生效,PIT-safe,绝不用未来行业)。起始日前已有的成分能 carry forward 到窗口开始。 +- **SW 层级可配置**:`processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,行业中性化业界标准,小截面自由度更稳)。**实测**:旧 tag 年化 −17.6%、SW-L1 −10.2%、SW-L2 −9.3% —— L1≈L2,−17.6→−10 的大跳是 **tushare→SW 分类切换**(补 PIT 的必然代价:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**;报告披露实际 level 与覆盖率。 +- **缺失处理(不静默退回 current)**:无 SW 历史的票 → 行业 **NaN**,被 neutralize 按截面丢弃;每次运行的 **PIT 行业覆盖率**在 `phase2_real_baseline.md` 披露。市值 `daily_basic.total_mv` 为逐日真值。 diff --git a/CLAUDE.md b/CLAUDE.md index 4d1ceff..d008dff 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,6 +80,10 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - `universe.min_listing_days` **真实路径已执行**（`stock_basic.list_date` 富化,买入资格过滤,边界 age==min 放行,缺 list_date 保留并披露）;demo 无上市日 → 披露 no-op。 - 回测 `feasibility_log()`：每调仓期 blocked buys/sells/carried/executed turnover/invested;phase2 报告新增 **Execution feasibility** 小节。 - **保不变量**:demo 无 flag → 全可成交 → P0/P1 数字不变;无未来函数、PIT/ann_date/real-demo 分离不变。 -- ✅ 质量门：`pytest` **204 passed**（P0=94 / P1=75 / P2-1=14 / P2-2=21）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`）+ `run-phase0`（demo）均 OK。 -- ⚠️ 剩余 P2（已显式披露）：行业标签用**当前值**非 PIT、日线 only、简版 IC/绩效未走 alphalens/quantstats、demo 路径非真数据。 -- 路线图下一步：财务因子组合 / 历史 PIT 行业 / 分钟级 / alphalens·quantstats 接入（architecture.html §11）。 +- ✅ **Phase 2-3 历史 PIT 行业**（PR #5,**默认 SW-L1 可配置版**,等验收）：把行业中性化协变量从 `stock_basic.industry` **当前**标签升级为按 trade_date **as-of** 的历史申万行业。 + - `tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 取每股 SW 行业 `in_date`/`out_date` 区间;`data/clean/pit_industry.py::asof_industry` 按 `[in_date,out_date)` 覆盖该日取行业(改分类日新行业生效,PIT-safe;起始日前成分 carry forward 到窗口开始)。 + - **SW 层级可配置** `processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**=31 宽板块,中性化标准 + 小截面自由度更稳)。**关键实证**:旧 tag 年化 −17.6% / SW-L1 −10.2% / SW-L2 −9.3% → **L1≈L2**,大跳的主因是 **tushare→SW 分类切换**(补 PIT 的必然:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**。(注:曾误以为是粒度、默认 L2,经 L2 实测推翻,改默认 L1。) + - **不静默退回 current**:无 SW 历史的票 → 行业 NaN,被 neutralize 按截面丢弃(neutralize 数学不变,本就丢 NaN 行);每次运行 **实际 level + PIT 覆盖率**进 phase2 报告。 +- ✅ 质量门：`pytest` **223 passed**（P0=95 / P1=77 / P2-1=15 / P2-2=22 / P2-3=14）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`）+ `run-phase0`（demo）均 OK。 +- ⚠️ 剩余 P2（已显式披露）：日线 only、简版 IC/绩效未走 alphalens/quantstats、demo 路径非真数据。 +- 路线图下一步：财务因子组合 / 分钟级 / alphalens·quantstats 接入（architecture.html §11）。 diff --git a/RUNBOOK.md b/RUNBOOK.md index bb5cb53..76d65ea 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -204,19 +204,28 @@ Implemented in P1 (real path): - **ann_date financial alignment** — figures used only after disclosure date. - **Industry + size neutralization** — per-date OLS residual. -Resolved in P2-2 (was deferred): +Resolved in P2-2 / P2-3 (was deferred): -- **Direction-aware limits/suspension** — now in the execution layer (up-limit +- **Direction-aware limits/suspension** (P2-2) — now in the execution layer (up-limit blocks buys, down-limit blocks sells, suspended/missing blocks both; blocked trades carry forward, executed-only turnover). No longer a crude both-direction selection drop. -- **min_listing_days** — enforced on the real path (`stock_basic.list_date`) as a +- **min_listing_days** (P2-2) — enforced on the real path (`stock_basic.list_date`) as a buy/selection filter; demo stays a disclosed no-op. +- **PIT industry** (P2-3) — the neutralization industry covariate is now point-in-time + SW (as-of trade date via `index_member_all` in/out dates, `data/clean/pit_industry.py`), + not the current `stock_basic.industry` tag. The SW level is configurable + (`processing.neutralize.industry_level`, L1/L2/L3, **default L1** = 31 broad sectors, the + standard for neutralization and DOF-safe on small cross-sections). Real runs: old tag + annual −17.6%, SW-L1 −10.2%, SW-L2 −9.3% — L1 ≈ L2, so the −17.6→−10 jump is the + tushare→SW taxonomy switch (inherent to going PIT: only SW carries in/out-date history), + NOT granularity. Names with no SW history get NaN (a disclosed coverage gap the + neutralizer drops) — never a silent current-tag fallback; the actual level + PIT coverage + are reported in `phase2_real_baseline.md`. Still downgraded / deferred (disclosed): - **Demo path** uses offline `DemoFeed` — NOT real data (no PIT/financial meaning). - **Static universe** option remains a PIT downgrade (use `type: index` for real). -- **Industry tag is current** (`stock_basic`), not point-in-time — mild downgrade. - **Daily bars only**; **simple IC / performance** (numpy/pandas, not alphalens-reloaded / quantstats). diff --git a/TEST_REPORT.md b/TEST_REPORT.md index 87cd106..b1a53a7 100644 --- a/TEST_REPORT.md +++ b/TEST_REPORT.md @@ -1,4 +1,4 @@ -# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 (bias-boundary → execution realism) +# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 (bias-boundary → execution realism → PIT industry) ## Commands @@ -17,14 +17,14 @@ Run from the repo root with the project python (env `quant_mf`): | Gate | Command | Result | |---|---|---| -| Unit + integration | `pytest -q` | **204 passed, 0 failed** | +| Unit + integration | `pytest -q` | **223 passed, 0 failed** | | Lint | `ruff check .` | **All checks passed** | | Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`) | exit `0`, prints `OK` | | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | -Counts below are the actual per-file `pytest` numbers (sum = 204). +Counts below are the actual per-file `pytest` numbers (sum = 223). -## Per-file breakdown — Phase 0 core (94) +## Per-file breakdown — Phase 0 core (95) | Test file | Tests | Area | |---|---|---| @@ -43,9 +43,9 @@ Counts below are the actual per-file `pytest` numbers (sum = 204). | `test_analytics_factor.py` | 5 | IC / quantile | | `test_analytics_performance.py` | 3 | performance metrics | | `test_phase0_pipeline.py` | 8 | end-to-end pipeline | -| `test_bias_audit_report.py` | 5 | bias audit doc (+1: P2-2 disclosures) | +| `test_bias_audit_report.py` | 6 | bias audit doc (+2: P2-2 + P2-3 disclosures) | -## Per-file breakdown — Phase 1 bias-boundary (75) +## Per-file breakdown — Phase 1 bias-boundary (79) | Test file | Tests | Red-line / feature | |---|---|---| @@ -65,23 +65,29 @@ Counts below are the actual per-file `pytest` numbers (sum = 204). | `test_neutralize.py` | 5 | industry+size residual orthogonality | | `test_processing_neutralize.py` | 2 | neutralize wiring (covariates required) | | `test_covariates_enrich.py` | 3 | industry + market_cap enrichment | -| `test_tushare_covariates.py` | 2 | stock_basic + daily_basic feed | -| `test_real_path_config.py` | 3 | demo vs real-path downgrade disclosure | +| `test_tushare_covariates.py` | 5 | stock_basic + daily_basic + index_member_all SW feed (level L1/L2/L3 select, bad level) | +| `test_real_path_config.py` | 4 | demo vs real-path downgrade disclosure (+ PIT industry) | -## Per-file breakdown — Phase 2-1 real-data baseline (14) +## Per-file breakdown — Phase 2-1 real-data baseline (15) | Test file | Tests | Feature | |---|---|---| -| `test_phase2_baseline.py` | 14 | collectors, demo/real guard, report-field contract, no-secret-leak, settled-vs-candidate dates, loaded-vs-in-window membership | +| `test_phase2_baseline.py` | 15 | collectors, demo/real guard, report-field contract, no-secret-leak, settled-vs-candidate dates, loaded-vs-in-window membership, list_date + PIT-industry coverage | -## Per-file breakdown — Phase 2-2 execution realism (21) +## Per-file breakdown — Phase 2-2 execution realism (22) | Test file | Tests | Red-line / feature | |---|---|---| | `test_fills.py` | 10 | direction-aware fill sim (`simulate_fills`) + panel→feasibility adapter; cash-coherent sell-then-buy, no leverage, executed-only turnover | -| `test_driver_feasibility.py` | 5 | end-to-end: down-limit carries, up-limit blocks buy, suspended no-trade, feasibility log == nav index | +| `test_driver_feasibility.py` | 6 | end-to-end: down-limit carries, up-limit blocks buy, suspended no-trade, **holdings == achieved (not desired)**, feasibility log == nav index | | `test_min_listing_days.py` | 6 | `min_listing_days` buy-eligibility boundaries (age <, ==, >; missing list_date kept; no-op cases) | -| **Total (P0 + P1 + P2-1 + P2-2)** | **204** | | + +## Per-file breakdown — Phase 2-3 PIT industry (12) + +| Test file | Tests | Red-line / feature | +|---|---|---| +| `test_pit_industry.py` | 12 | SW **as-of** industry (switch at reclassification, carry-forward pre-start, missing → NaN, latest-in_date on overlap) + `enrich_pit_industry` + pipeline wiring (per-date industry, configured SW level passed, no current-tag fallback) + `industry_level` config (default L1, accepts L1/L2/L3, rejects invalid, phase2 config = L1) | +| **Total (P0 + P1 + P2-1 + P2-2 + P2-3)** | **223** | | ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) @@ -103,5 +109,18 @@ Counts below are the actual per-file `pytest` numbers (sum = 204). - `universe.min_listing_days` is enforced on the real path (list_date from `stock_basic`) as a buy/selection filter; a missing list_date is kept and disclosed; the demo path stays a disclosed no-op. +- **P2-3 PIT industry (locked by tests):** the neutralization industry covariate is + point-in-time SW (as-of trade date via `index_member_all` in/out dates), not the + current `stock_basic.industry` tag. The SW level is configurable + (`processing.neutralize.industry_level`, **default L1** = 31 broad sectors, the standard + for neutralization and DOF-safe on small cross-sections). Real runs: old tag annual + −17.6%, SW-L1 −10.2%, SW-L2 −9.3% — **L1 ≈ L2**, so the −17.6→−10 jump is the + **tushare→SW taxonomy switch** (inherent to going PIT — only SW carries in/out-date + history; the old tag cannot be PIT-aligned), NOT a granularity choice. Names with no SW + history get NaN (the neutralizer drops them) — never a silent current-tag fallback; the + actual level + PIT coverage are disclosed in the phase2 report. The neutralize math is + unchanged. - A duplicate test-function name across two files was found and renamed during P2-2 - (it had been silently shadowing one test in the full-suite run). + (it had been silently shadowing one test in the full-suite run). A second, harmless + duplicate (`test_enrich_does_not_mutate_input` in two files) was verified NOT to drop + a test (per-file sum == full-run total = 217). diff --git a/config/phase2_real_baseline.yaml b/config/phase2_real_baseline.yaml index f7fb4e0..b7b0afc 100644 --- a/config/phase2_real_baseline.yaml +++ b/config/phase2_real_baseline.yaml @@ -56,6 +56,7 @@ processing: enabled: true industry_col: industry size_col: market_cap + industry_level: L1 # PIT SW level (L1/L2/L3); L1 = 31 broad sectors (standard, DOF-safe) alpha: model: equal_weight diff --git a/data/clean/covariates.py b/data/clean/covariates.py index 63c2387..6b7754c 100644 --- a/data/clean/covariates.py +++ b/data/clean/covariates.py @@ -39,6 +39,22 @@ def enrich_covariates( return out +def enrich_pit_industry( + panel: pd.DataFrame, industry: pd.Series +) -> pd.DataFrame: + """Return a NEW panel with a POINT-IN-TIME ``industry`` column (UNI-010). + + ``industry`` is a ``(date, symbol)``-indexed Series of as-of SW-L1 industry + names (from :func:`data.clean.pit_industry.asof_industry`) — it varies by date, + unlike the current-tag broadcast. Rows with no as-of industry stay ``NaN`` (a + disclosed gap the neutralizer drops). Pure: never mutates the input panel. + """ + validate_panel(panel) + out = panel.copy() + out["industry"] = industry.reindex(out.index) + return out + + def enrich_listing( panel: pd.DataFrame, listing_dates: dict[str, pd.Timestamp] ) -> pd.DataFrame: diff --git a/data/clean/pit_industry.py b/data/clean/pit_industry.py new file mode 100644 index 0000000..e3f57f7 --- /dev/null +++ b/data/clean/pit_industry.py @@ -0,0 +1,76 @@ +"""Point-in-time as-of alignment of SW industry by membership intervals (P2-3). + +The correctness boundary: industry neutralization must use the industry a stock +belonged to AS OF each trade_date, not its CURRENT (latest) tag. `stock_basic.industry` +gives only the current label; broadcasting it to historical dates leaks a future +reclassification into the past (a mild membership look-ahead in the neutralization +covariate). + +`asof_industry` consumes per-symbol SW membership intervals (at the configured SW +level — L1/L2/L3, default L1) ``{symbol: [(industry_name, in_date, out_date), ...]}`` +(from :meth:`data.feed.tushare_covariates.TushareCovariatesFeed.pit_sw_intervals`, +which reads tushare ``index_member_all``) and returns, for each ``(trade_date, +symbol)``, the industry whose interval ``[in_date, out_date)`` covers that date — +the most recent ``in_date`` on the (rare) overlap. ``out_date`` ``None``/``NaT`` is +an open (still-active) membership. A date before any membership, or a symbol with +no intervals, yields ``NaN`` — a disclosed data gap that the neutralizer drops +(never a silent fallback to the current tag). +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + + +def asof_industry(index: pd.MultiIndex, intervals: dict) -> pd.Series: + """Return a ``(date, symbol)`` Series of the as-of SW industry name. + + Args: + index: target MultiIndex(date, symbol) to align onto. + intervals: ``{symbol: [(industry_name, in_date, out_date), ...]}`` where + ``in_date``/``out_date`` are Timestamps (``out_date`` ``None``/``NaT`` + means still active). Membership covers ``in_date <= date < out_date``, + so on a reclassification date the NEW industry wins (PIT-safe). + + Each row carries the covering interval's industry (latest ``in_date`` on + overlap), or ``NaN`` if no interval covers that date / the symbol is unknown. + """ + norm = _normalize_intervals(intervals) + dates = index.get_level_values("date") + symbols = index.get_level_values("symbol").astype(str) + + values: list = [] + for d, s in zip(dates, symbols): + rows = norm.get(s) + best_in = None + best_name = np.nan + if rows: + for name, in_date, out_date in rows: + if in_date <= d and (out_date is None or d < out_date): + if best_in is None or in_date > best_in: + best_in = in_date + best_name = name + values.append(best_name) + return pd.Series(values, index=index, name="industry") + + +def _normalize_intervals(intervals: dict) -> dict: + """Coerce interval dates to Timestamps; missing in_date -> Timestamp.min (open start).""" + norm: dict[str, list] = {} + for sym, ivs in (intervals or {}).items(): + rows = [] + for name, in_date, out_date in ivs: + in_ts = ( + pd.Timestamp(in_date) + if in_date is not None and not pd.isna(in_date) + else pd.Timestamp.min + ) + out_ts = ( + pd.Timestamp(out_date) + if out_date is not None and not pd.isna(out_date) + else None + ) + rows.append((name, in_ts, out_ts)) + norm[str(sym)] = rows + return norm diff --git a/data/feed/tushare_covariates.py b/data/feed/tushare_covariates.py index 2258468..ae95dd0 100644 --- a/data/feed/tushare_covariates.py +++ b/data/feed/tushare_covariates.py @@ -1,16 +1,24 @@ -"""TushareCovariatesFeed: industry + market cap for neutralization. +"""TushareCovariatesFeed: covariates for neutralization (+ listing dates). -Provides the two cross-sectional covariates the neutralizer needs: +Provides the cross-sectional inputs the pipeline needs: - * ``industry(symbols)`` -> {symbol: industry} (tushare ``stock_basic``). + * ``pit_sw_intervals(symbols, level)`` -> {symbol: [(industry_name, in_date, out_date)]} + (tushare ``index_member_all``): the POINT-IN-TIME SW industry membership history + at ``level`` (L1/L2/L3; default L1). This is the industry source the neutralizer + uses (P2-3): aligned as-of the trade date in + :func:`data.clean.pit_industry.asof_industry`, so a reclassification is + respected and no future industry leaks into the past. * ``market_cap(symbols, s, e)`` -> DataFrame[date, symbol, market_cap] (tushare ``daily_basic.total_mv``, in 10k CNY; only the log is used, so - units do not matter). - -Caveat (disclosed in the bias audit): ``stock_basic.industry`` is the CURRENT -industry tag, not a point-in-time history, so industry neutralization carries a -mild membership-style look-ahead. Market cap is genuinely per-date. Token is read -from the external config and never printed; the client is lazy. + units do not matter). Genuinely per-date. + * ``listing_dates(symbols)`` -> {symbol: list_date} (``stock_basic.list_date``, + for the ``min_listing_days`` buy filter, UNI-008). + * ``industry(symbols)`` -> {symbol: industry} (``stock_basic.industry``, + the CURRENT tag). Retained as an accessor but NO LONGER wired into + neutralization — the current tag would broadcast a future industry onto past + dates; the PIT intervals above replace it. + +Token is read from the external config and never printed; the client is lazy. """ from __future__ import annotations @@ -79,6 +87,47 @@ def listing_dates(self, symbols: list[str]) -> dict[str, pd.Timestamp]: ) return out + _SW_LEVEL_COLUMN = {"L1": "l1_name", "L2": "l2_name", "L3": "l3_name"} + + def pit_sw_intervals( + self, symbols: list[str], level: str = "L1" + ) -> dict[str, list[tuple]]: + """Return SW membership history per symbol at ``level`` (for PIT industry, UNI-010). + + ``{symbol: [(industry_name, in_date, out_date), ...]}`` from tushare + ``index_member_all`` (SW2021), where ``industry_name`` is the L1 / L2 / L3 + name per ``level`` (default ``L1`` — the 31 broad SW sectors, standard for + industry neutralization and DOF-safe on small cross-sections). ``in_date``/ + ``out_date`` are Timestamps; ``out_date`` is ``None`` for an active membership. + A symbol with no SW membership row is simply absent (the caller treats an + absent symbol as a disclosed industry data gap → NaN, never the current tag). + One call per symbol; the payload is a handful of intervals each. + """ + col = self._SW_LEVEL_COLUMN.get(str(level).upper()) + if col is None: + raise ValueError( + f"industry level must be one of {list(self._SW_LEVEL_COLUMN)}; got {level!r}." + ) + pro = self._client() + out: dict[str, list[tuple]] = {} + for sym in symbols: + df = self._call(pro.index_member_all, ts_code=sym) + if df is None or len(df) == 0 or col not in df.columns: + continue + rows: list[tuple] = [] + for r in df.itertuples(): + in_d = pd.to_datetime(str(r.in_date), format="%Y%m%d", errors="coerce") + out_raw = getattr(r, "out_date", None) + out_d = ( + pd.to_datetime(str(out_raw), format="%Y%m%d", errors="coerce") + if out_raw is not None and str(out_raw) not in ("", "None", "nan") + else None + ) + rows.append((getattr(r, col), in_d, out_d)) + if rows: + out[str(sym)] = rows + return out + def market_cap(self, symbols: list[str], start: str, end: str) -> pd.DataFrame: """Return DataFrame[date, symbol, market_cap] from daily_basic.total_mv.""" pro = self._client() diff --git a/qt/config.py b/qt/config.py index 2fd1bac..f9f82db 100644 --- a/qt/config.py +++ b/qt/config.py @@ -114,6 +114,13 @@ class NeutralizeCfg(_Strict): enabled: bool = False industry_col: str = "industry" size_col: str = "market_cap" + # SW industry level for the PIT industry covariate (P2-3). Default L1 = the 31 + # broad SW sectors, the standard granularity for industry neutralization and the + # safest on small cross-sections (more residual DOF than ~130 L2 sub-industries). + # NOTE: going PIT necessarily switches the taxonomy from the old (non-PIT-able) + # stock_basic.industry tag to SW — only SW carries in/out-date history — so the + # backtest result changes vs the old tag regardless of level (L1 ≈ L2 in tests). + industry_level: Literal["L1", "L2", "L3"] = "L1" class ProcessingCfg(_Strict): diff --git a/qt/phase2_baseline.py b/qt/phase2_baseline.py index e5a7596..334e765 100644 --- a/qt/phase2_baseline.py +++ b/qt/phase2_baseline.py @@ -80,6 +80,8 @@ class Phase2Result: # min_listing_days list_date coverage (known vs disclosed data gap, this run) list_date_known: int list_date_total: int + # PIT SW industry coverage for neutralization (NaN if neutralize off) + industry_pit_coverage: float # ann_date financial coverage (diagnostic) financial_field: str financial_coverage_overall: float @@ -405,6 +407,12 @@ def run_phase2_baseline(config_path: str) -> Phase2Result: list_date_known = 0 list_date_total = 0 + # PIT SW industry coverage (fraction of panel rows with an as-of industry). + if cfg.processing.neutralize.enabled and "industry" in panel.columns: + industry_pit_coverage = float(panel["industry"].notna().mean()) + else: + industry_pit_coverage = float("nan") + dates = panel.index.get_level_values("date") result = Phase2Result( config=cfg, @@ -419,6 +427,7 @@ def run_phase2_baseline(config_path: str) -> Phase2Result: ), list_date_known=list_date_known, list_date_total=list_date_total, + industry_pit_coverage=industry_pit_coverage, financial_field=financial_field, financial_coverage_overall=coverage_overall, financial_coverage_by_rebalance=coverage_by_rebalance, diff --git a/qt/pipeline.py b/qt/pipeline.py index 8bce489..dbb5155 100644 --- a/qt/pipeline.py +++ b/qt/pipeline.py @@ -37,7 +37,8 @@ from analytics.factor import compute_ic, forward_returns, ic_summary, quantile_returns from analytics.performance import performance_summary from data.clean.adjust import front_adjust -from data.clean.covariates import enrich_covariates, enrich_listing +from data.clean.covariates import enrich_covariates, enrich_listing, enrich_pit_industry +from data.clean.pit_industry import asof_industry from data.clean.pit_financials import asof_financials from data.clean.tradability import enrich_tradability from data.feed.base import DataFeed @@ -224,11 +225,23 @@ def _collect_downgrades(cfg: RootConfig) -> tuple[str, ...]: ) # neutralization - if cfg.processing.neutralize.enabled: + if cfg.processing.neutralize.enabled and real: + level = cfg.processing.neutralize.industry_level neutral = ( - "Factor is industry + market-cap neutralized; the industry tag is the " - "CURRENT one (stock_basic), a mild PIT downgrade (historical industry is " - "a later item)." + f"Factor is industry + market-cap neutralized; industry is **point-in-time** " + f"SW-{level} (UNI-010): as-of the trade date via index_member_all in/out dates, " + f"NOT the current stock_basic tag. The SW level is configurable " + f"(processing.neutralize.industry_level, default L1 = 31 broad sectors, the " + f"standard for neutralization). Going PIT switches the taxonomy from the old " + f"(non-PIT-able) stock_basic.industry tag to SW, so the result changes vs the " + f"old tag regardless of level (L1 ≈ L2 in tests). Names with no SW history get " + f"NaN (a disclosed coverage gap the neutralizer drops) — never a silent " + f"current-tag fallback. Market cap is per-date (daily_basic.total_mv)." + ) + elif cfg.processing.neutralize.enabled: + neutral = ( + "Factor is industry + market-cap neutralized (real path only; this is not a " + "tushare run, so the covariates are unavailable)." ) else: neutral = "No neutralization in this run (raw cross-sectional factor)." @@ -639,12 +652,21 @@ def _maybe_enrich_covariates( feed = TushareCovariatesFeed( cfg.data.external_secret_file, token_key=cfg.data.tushare_token_key ) - industry = feed.industry(symbols) market_cap = feed.market_cap(symbols, cfg.data.start, cfg.data.end) - panel = enrich_covariates(panel, industry=industry, market_cap=market_cap) + panel = enrich_covariates(panel, market_cap=market_cap) + # PIT (as-of) SW industry replaces the current-tag broadcast (UNI-010): each + # name's industry varies by trade_date via index_member_all in/out dates at the + # configured SW level; a name with no SW history gets NaN — a disclosed gap the + # neutralizer drops — never a silent fallback to the current stock_basic tag. + level = cfg.processing.neutralize.industry_level + intervals = feed.pit_sw_intervals(symbols, level=level) + industry_series = asof_industry(panel.index, intervals) + panel = enrich_pit_industry(panel, industry_series) + coverage = float(industry_series.notna().mean()) if len(industry_series) else 0.0 logger.info( - "covariates: industry(%d symbols) + market_cap(%d rows) for neutralization", - len(industry), len(market_cap), + "covariates: PIT SW-%s industry (coverage %.1f%%, %d/%d symbols with history) " + "+ market_cap(%d rows) for neutralization", + level, coverage * 100.0, len(intervals), len(symbols), len(market_cap), ) return panel diff --git a/qt/reports.py b/qt/reports.py index 6372947..a41d3c6 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -306,6 +306,15 @@ def render_phase2_baseline(result: "Phase2Result") -> str: f"**{result.list_date_known}/{result.list_date_total}** known, " f"**{missing}** missing (kept as a disclosed data gap, never excluded)\n" ) + if math.isfinite(result.industry_pit_coverage): + level = result.config.processing.neutralize.industry_level + lines.append( + f"- neutralization industry is **point-in-time SW-{level}** (as-of trade date; " + f"level configurable, default L1); PIT coverage this run: " + f"**{_fmt(result.industry_pit_coverage, pct=True)}** of (date, symbol) rows. " + f"Names with no SW history get NaN (dropped by the neutralizer) — never a " + f"current-tag fallback.\n" + ) lines.append("\n## Financial ann_date coverage\n") lines.append( @@ -505,9 +514,20 @@ def render_bias_audit() -> str: "缺失(如 demo 路径)直接报可读错误。\n" "- 实证:12 只票横跨 4 行业(2024-09-30),corr(原始 momentum, log市值) " "= -0.617 → 中性化后 -0.000,各行业残差均值 ≈ 0,确认规模/行业暴露被移除。\n" - "- **降级**:行业来自 `stock_basic.industry` 的**当前**行业标签,非按历史时点," - "故行业中性化带有轻微成分前视(市值 `daily_basic.total_mv` 为逐日真值)。" - "PIT 行业历史是后续项,此降级在此显式披露(INV-007)。\n" + "- **行业 PIT(UNI-010,P2-3,已实现)**:行业协变量从 `stock_basic.industry` 的" + "**当前**标签,升级为按 trade_date **as-of** 的历史申万行业。`data.feed." + "tushare_covariates.pit_sw_intervals(symbols, level)` 读 `index_member_all` 拿每股 " + "SW 行业的 `in_date`/`out_date` 区间;`data.clean.pit_industry.asof_industry` 按" + "`[in_date, out_date)` 覆盖该日的区间取行业(改分类日**新行业**生效,PIT-safe,绝不用" + "未来行业)。起始日前已有的成分能 carry forward 到窗口开始。\n" + "- **SW 层级可配置**:`processing.neutralize.industry_level`(L1/L2/L3,**默认 L1**" + "=31 宽板块,行业中性化业界标准,小截面自由度更稳)。**实测**:旧 tag 年化 −17.6%、" + "SW-L1 −10.2%、SW-L2 −9.3% —— L1≈L2,−17.6→−10 的大跳是 **tushare→SW 分类切换**" + "(补 PIT 的必然代价:只有 SW 有 in/out 历史可 PIT 化,旧 tag 无法),**与粒度无关**;" + "报告披露实际 level 与覆盖率。\n" + "- **缺失处理(不静默退回 current)**:无 SW 历史的票 → 行业 **NaN**,被 neutralize " + "按截面丢弃;每次运行的 **PIT 行业覆盖率**在 `phase2_real_baseline.md` 披露。市值 " + "`daily_basic.total_mv` 为逐日真值。\n" ) diff --git a/tests/test_bias_audit_report.py b/tests/test_bias_audit_report.py index b914357..4e74182 100644 --- a/tests/test_bias_audit_report.py +++ b/tests/test_bias_audit_report.py @@ -55,6 +55,14 @@ def test_bias_audit_discloses_min_listing_days_enforced_real_noop_demo(): assert "no-op" in text and "降级" in text # demo fallback, still disclosed +def test_bias_audit_discloses_pit_industry(): + """Industry neutralization covariate is disclosed as point-in-time, not current (P2-3).""" + text = render_bias_audit() + assert "UNI-010" in text + assert "as-of" in text and "index_member_all" in text + assert "PIT 行业覆盖率" in text or "PIT" in text # coverage disclosure named + + def test_bias_audit_discloses_direction_aware_execution(): """Direction-aware execution feasibility (limits/suspension) is disclosed (P2-2).""" text = render_bias_audit() diff --git a/tests/test_phase2_baseline.py b/tests/test_phase2_baseline.py index ee705ac..4d8378a 100644 --- a/tests/test_phase2_baseline.py +++ b/tests/test_phase2_baseline.py @@ -282,6 +282,7 @@ def _synthetic_result() -> Phase2Result: universe_summary=summarize_universe(_pit_universe(), "index", "2024-01-01", "2024-03-01"), list_date_known=67, list_date_total=68, + industry_pit_coverage=0.985, financial_field="roe", financial_coverage_overall=0.5, financial_coverage_by_rebalance=cov, @@ -331,3 +332,10 @@ def test_render_discloses_list_date_coverage_this_run(): md = render_phase2_baseline(_synthetic_result()) assert "67/68" in md # known/total for THIS run assert "missing" in md.lower() # the data gap is disclosed, not just generic + + +def test_render_discloses_pit_industry_coverage(): + md = render_phase2_baseline(_synthetic_result()) + assert "point-in-time SW-L1" in md # phase2 config sets industry_level: L1 + assert "98.50%" in md # industry_pit_coverage=0.985 rendered as a pct + assert "current-tag fallback" in md # explicitly no silent fallback diff --git a/tests/test_pit_industry.py b/tests/test_pit_industry.py new file mode 100644 index 0000000..e007874 --- /dev/null +++ b/tests/test_pit_industry.py @@ -0,0 +1,194 @@ +"""Point-in-time SW industry as-of alignment (P2-3, pure + network-free). + +`asof_industry` turns per-symbol SW membership intervals at the configured level +(L1/L2/L3) (industry, in_date, out_date) into a (date, symbol) industry Series +where each row carries the +industry the symbol belonged to AS OF that trade_date — never a future +reclassification. This is the PIT replacement for the `stock_basic.industry` +CURRENT tag that was broadcast to every date. +""" + +from __future__ import annotations + +from pathlib import Path + +import pandas as pd +import pytest + +from data.clean.pit_industry import asof_industry +from qt.config import ConfigError, RootConfig, load_config + + +# --------------------------------------------------------------------------- # +# config: processing.neutralize.industry_level (L1/L2/L3, default L1) +# --------------------------------------------------------------------------- # +def _min_cfg(level=None): + base = { + "data": {"source": "demo", "start": "2024-01-01", "end": "2024-03-01"}, + "universe": {"type": "static", "symbols": ["000001.SZ"]}, + "factors": [{"name": "momentum_20"}], + "alpha": {"model": "equal_weight"}, + "portfolio": {"top_n": 1}, + "backtest": {}, + "cost": {}, + "output": {}, + } + if level is not None: + base["processing"] = {"neutralize": {"enabled": True, "industry_level": level}} + return base + + +def test_industry_level_defaults_to_l1(): + cfg = RootConfig(**_min_cfg()) + assert cfg.processing.neutralize.industry_level == "L1" + + +def test_industry_level_accepts_l1_l2_l3(): + for lvl in ("L1", "L2", "L3"): + cfg = RootConfig(**_min_cfg(lvl)) + assert cfg.processing.neutralize.industry_level == lvl + + +def test_industry_level_rejects_invalid(tmp_path): + import yaml + + p = tmp_path / "bad.yaml" + p.write_text(yaml.safe_dump(_min_cfg("L9")), encoding="utf-8") + with pytest.raises(ConfigError): + load_config(str(p)) + + +def test_phase2_config_sets_l1(): + cfg = load_config(str(Path(__file__).resolve().parents[1] / "config" / "phase2_real_baseline.yaml")) + assert cfg.processing.neutralize.industry_level == "L1" + + +def _index(dates, symbols): + return pd.MultiIndex.from_product( + [pd.to_datetime(dates), symbols], names=["date", "symbol"] + ) + + +def _ts(s): + return pd.Timestamp(s) if s is not None else None + + +def test_asof_switches_at_reclassification_date(): + # X is IndA until 2023-06-01, then IndB from 2023-06-01 (open). + intervals = { + "X": [ + ("IndA", _ts("2020-01-01"), _ts("2023-06-01")), + ("IndB", _ts("2023-06-01"), None), + ] + } + idx = _index(["2023-05-31", "2023-06-01", "2023-07-01"], ["X"]) + out = asof_industry(idx, intervals) + assert out.loc[(pd.Timestamp("2023-05-31"), "X")] == "IndA" # before switch + assert out.loc[(pd.Timestamp("2023-06-01"), "X")] == "IndB" # on switch -> new + assert out.loc[(pd.Timestamp("2023-07-01"), "X")] == "IndB" # after + + +def test_asof_carries_forward_pre_start_membership(): + # Y joined IndC in 2015 and never left -> covers a 2023 window from the start. + intervals = {"Y": [("IndC", _ts("2015-01-01"), None)]} + idx = _index(["2023-01-01", "2024-06-30"], ["Y"]) + out = asof_industry(idx, intervals) + assert (out == "IndC").all() + + +def test_asof_missing_symbol_is_nan(): + intervals = {"Y": [("IndC", _ts("2015-01-01"), None)]} + idx = _index(["2023-01-01"], ["Z"]) # Z absent from intervals + out = asof_industry(idx, intervals) + assert pd.isna(out.loc[(pd.Timestamp("2023-01-01"), "Z")]) + + +def test_asof_date_before_first_membership_is_nan(): + # W only enters IndD in 2024 -> a 2023 date has no as-of industry (no future use). + intervals = {"W": [("IndD", _ts("2024-01-01"), None)]} + idx = _index(["2023-12-31", "2024-01-01"], ["W"]) + out = asof_industry(idx, intervals) + assert pd.isna(out.loc[(pd.Timestamp("2023-12-31"), "W")]) + assert out.loc[(pd.Timestamp("2024-01-01"), "W")] == "IndD" + + +def test_asof_returns_industry_series_aligned_to_index(): + intervals = {"A": [("I1", _ts("2010-01-01"), None)], "B": [("I2", _ts("2010-01-01"), None)]} + idx = _index(["2023-01-01"], ["A", "B"]) + out = asof_industry(idx, intervals) + assert list(out.index) == list(idx) + assert out.loc[(pd.Timestamp("2023-01-01"), "A")] == "I1" + assert out.loc[(pd.Timestamp("2023-01-01"), "B")] == "I2" + + +def test_asof_picks_latest_in_date_on_overlap(): + # Defensive: if two intervals both cover the date, the most recent in_date wins. + intervals = { + "X": [ + ("Old", _ts("2010-01-01"), None), + ("New", _ts("2022-01-01"), None), + ] + } + idx = _index(["2023-01-01"], ["X"]) + out = asof_industry(idx, intervals) + assert out.loc[(pd.Timestamp("2023-01-01"), "X")] == "New" + + +# --------------------------------------------------------------------------- # +# enrich_pit_industry + pipeline wiring (PIT replaces the current-tag broadcast) +# --------------------------------------------------------------------------- # +def test_enrich_pit_industry_sets_per_date_column(demo_panel): + from data.clean.covariates import enrich_pit_industry + + idx = demo_panel.index + # a deterministic as-of Series over the panel index + series = pd.Series("Bank", index=idx, name="industry") + out = enrich_pit_industry(demo_panel, series) + assert "industry" in out.columns + assert (out["industry"] == "Bank").all() + assert "industry" not in demo_panel.columns # input not mutated + + +def test_pipeline_covariates_uses_pit_industry_varying_by_date(monkeypatch, demo_panel, example_config_path): + # _maybe_enrich_covariates must build a PIT (per-date) industry, NOT a constant + # current tag, and leave names with no SW history as NaN. + import logging + from pathlib import Path + + from qt.config import load_config + from qt import pipeline as P + + cfg = load_config(str(Path(example_config_path).parent / "example_tushare.yaml")) + symbols = list(demo_panel.index.get_level_values("symbol").unique()) + switch = pd.Timestamp("2024-01-20") + + seen = {} + + class _FakeCov: + def __init__(self, *a, **k): + pass + + def market_cap(self, syms, start, end): + return pd.DataFrame( + {"date": [], "symbol": [], "market_cap": []} + ) # industry-only test + + def pit_sw_intervals(self, syms, level="L2"): + seen["level"] = level # the pipeline must pass the configured SW level + return { + symbols[0]: [("IndA", pd.Timestamp("2020-01-01"), switch), + ("IndB", switch, None)], # switches mid-window + symbols[1]: [("IndC", pd.Timestamp("2010-01-01"), None)], # carry forward + # symbols[2..] absent -> NaN (disclosed gap, no current-tag fallback) + } + + monkeypatch.setattr(P, "TushareCovariatesFeed", _FakeCov) + out = P._maybe_enrich_covariates(cfg, demo_panel, symbols, logging.getLogger("t")) + assert seen["level"] == cfg.processing.neutralize.industry_level # default L1 + + ind = out["industry"] + before = ind.loc[(pd.Timestamp("2024-01-19"), symbols[0])] + after = ind.loc[(pd.Timestamp("2024-01-22"), symbols[0])] + assert before == "IndA" and after == "IndB" # PIT switch, varies by date + assert (ind.xs(symbols[1], level="symbol") == "IndC").all() # carried forward + assert pd.isna(ind.xs(symbols[2], level="symbol")).all() # missing -> NaN diff --git a/tests/test_real_path_config.py b/tests/test_real_path_config.py index 726c78d..39fad9a 100644 --- a/tests/test_real_path_config.py +++ b/tests/test_real_path_config.py @@ -27,6 +27,15 @@ def test_downgrades_mark_real_path(example_config_path): assert any("neutralized" in x for x in items) +def test_downgrades_industry_is_pit_not_current(example_config_path): + # P2-3: the real-path neutralization downgrade must disclose POINT-IN-TIME SW + # industry (UNI-010) and must NOT claim the industry is the CURRENT stock_basic tag. + items = _collect_downgrades(load_config(_real_path(example_config_path))) + text = " ".join(items) + assert "point-in-time" in text and "UNI-010" in text + assert "the industry tag is the CURRENT" not in text + + def test_downgrades_mark_demo_path(example_config_path): items = _collect_downgrades(load_config(example_config_path)) assert items[0].startswith("DATA PATH = DEMO") diff --git a/tests/test_tushare_covariates.py b/tests/test_tushare_covariates.py index 807dde7..ed27ab3 100644 --- a/tests/test_tushare_covariates.py +++ b/tests/test_tushare_covariates.py @@ -18,6 +18,19 @@ def daily_basic(self, ts_code, start_date, end_date, fields): # noqa: ARG002 {"ts_code": [ts_code], "trade_date": ["20240301"], "total_mv": [123.0]} ) + def index_member_all(self, ts_code): + # one active SW membership (out_date None) carrying all three levels. + return pd.DataFrame( + { + "l1_name": ["食品饮料"], + "l2_name": ["白酒Ⅱ"], + "l3_name": ["白酒Ⅲ"], + "in_date": ["20010731"], + "out_date": [None], + "ts_code": [ts_code], + } + ) + def _feed(monkeypatch): feed = TushareCovariatesFeed("x.json") @@ -34,3 +47,25 @@ def test_market_cap_maps_columns(monkeypatch): assert list(out.columns) == ["date", "symbol", "market_cap"] assert out.iloc[0]["market_cap"] == 123.0 assert str(out["date"].dtype).startswith("datetime64") + + +def test_pit_sw_intervals_default_level_is_l1(monkeypatch): + out = _feed(monkeypatch).pit_sw_intervals(["600519.SH"]) # default L1 + name, in_d, out_d = out["600519.SH"][0] + assert name == "食品饮料" # L1 broad sector (default) + assert in_d == pd.Timestamp("2001-07-31") + assert out_d is None # active membership + + +def test_pit_sw_intervals_selects_level(monkeypatch): + feed = _feed(monkeypatch) + assert feed.pit_sw_intervals(["600519.SH"], level="L1")["600519.SH"][0][0] == "食品饮料" + assert feed.pit_sw_intervals(["600519.SH"], level="L2")["600519.SH"][0][0] == "白酒Ⅱ" + assert feed.pit_sw_intervals(["600519.SH"], level="L3")["600519.SH"][0][0] == "白酒Ⅲ" + + +def test_pit_sw_intervals_bad_level_raises(monkeypatch): + import pytest + + with pytest.raises(ValueError, match="industry level"): + _feed(monkeypatch).pit_sw_intervals(["600519.SH"], level="L9")