diff --git a/.gitignore b/.gitignore index 9d967c5..1d4c3f6 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ venv/ .env # Data artifacts (quant: keep large data out of git) +artifacts/ *.parquet *.feather *.h5 diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..ec3100d --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,70 @@ +# Quantitative_Trading — A股截面多因子框架 + +## 项目定位 +- A股 **截面多因子选股**（cross-sectional multi-factor）框架。**不做择时**。 +- 路径：研究/回测优先 → 渐进到实盘。 +- 构建方式：成熟工具 + **自建因子层**为核心。 +- 完整架构设计见 [`tmp/framework/architecture.html`](tmp/framework/architecture.html)。本文件是精简操作版，**细节以该文档为准**。 + +## 架构：7 层 +``` +data → universe → factors(特征) → alpha(合成/预测) → portfolio(+risk约束) → runtime(backtest|live) → analytics +``` + +| 层 | 职责 | +|---|---| +| `data/` | 采集(feed) / 清洗复权·对齐披露日(clean) / 面板存储(store) | +| `universe/` | PIT 成分股 + 可交易过滤(停牌/涨跌停/ST) | +| `factors/` | 截面因子 计算(compute) / 预处理三件套(process) / 落盘(store) | +| `alpha/` | 多因子合成/预测（独立成层） | +| `portfolio/` | 组合构建 + 风控**事前约束**(construct.py / risk.py) | +| `runtime/` | 回测 与 实盘 统一：driver + execution，两套实现(backtest/live) | +| `analytics/` | alphalens(因子检验) + quantstats(组合绩效) | + +## 不可违反的设计不变量（写代码必须守） +1. **factors 永不碰未来收益**；只有 `alpha` 层才用未来收益拟合权重（防未来函数边界）。 +2. **回测即实盘**：`runtime` 的 backtest/live 是同接口两实现；`factors/alpha/portfolio` 层两边复用、一行不改。 +3. **分层解耦**：factor 不碰数据源；portfolio 不碰下单。坏味道：策略里直接 `pro.daily(...)`。 +4. 决策是**截面的**（每个调仓日横向排序选股）；时序只用于算因子。 + +## 致命陷阱（correctness 红线，详见 architecture.html §8） +未来函数 · PIT 成分股 · 可交易过滤 · 财务按 `ann_date` 披露日对齐 · 前复权 · batch≡incremental 一致性 · 行业+市值中性化 · 交易成本/换手 · 过拟合(样本内外+IC稳定性)。 + +## 环境 +- **框架/测试环境**：conda `quant_mf`（Py 3.12）。运行 Phase 0、pytest、ruff、CLI 时用绝对路径 python： + `/home/shaofl/Development/env_tools/envs/quant_mf/bin/python` +- **数据拉取环境**：conda `data_fetch`（Py 3.12）。仅用于独立数据抓取/交互查数；在非交互 shell 里**用绝对路径 python**，不靠 activate： + `/home/shaofl/Development/env_tools/envs/data_fetch/bin/python` + +## 数据：tushare +- **token**：`/home/shaofl/Projects/financial_projects/.config.json`（key `tushare.token`）。 + ⚠️ **绝不打印、绝不写进 repo、绝不 commit。** 代码里从该文件读取，不硬编码。 +- **权限**：实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业(`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上，无需先退回日线。 +- **MCP（可选开发工具）**：`financial_projects/.mcp.json` 有 tushare MCP，仅供开发期交互查数；**从 `financial_projects/` 启动 Codex 才加载**。 +- **数据层 ETL 一律用 Python SDK（批量/增量），不要建在 MCP 上。** 注意 tushare 各接口有每分钟调用上限，批量拉取需限流+重试。 + +## 技术选型 +| 用途 | 选型 | +|---|---| +| 数据处理 | pandas（+ polars 可选）· numpy | +| 存储 | parquet（分钟级按 symbol/year 分区）· DuckDB | +| 因子检验 | alphalens-reloaded | +| 绩效 | quantstats | +| 回归/合成 | statsmodels · scikit-learn | +| 组合优化(后期) | cvxpy · riskfolio-lib | +| 实盘下单(后期) | vnpy / miniqmt(QMT) —— **仅作下单通道，不当回测引擎** | +| 配置/测试 | pydantic-settings + YAML · pytest | + +## 开发约定 +- **交流中文**；代码/注释/commit message 用**英文**。 +- **Git**：feature 分支 + PR。main 已有骨架；当前在 `data` 分支搭数据层。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 +- **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 +- 文件小而专（<800 行），immutable 优先。 + +## 当前进度 +- ✅ 7 层骨架 + 架构文档（已在 `main`） +- ✅ Phase 0 MVP（`data` 分支，未提交）：DemoFeed → PanelStore → StaticUniverse → momentum_20 → zscore → EqualWeightAlpha → TopN 等权 → 月度回测（成本/换手）→ IC/绩效报告。 +- ✅ 质量门：`python -m pytest` 93 passed；`python -m ruff check .` clean；`python -m qt.cli run-phase0 --config config/example.yaml` 可复现。 +- ⚠️ P0 显式降级：静态 universe 非 PIT、简版 IC/绩效未走 alphalens/quantstats、`min_listing_days` 配置未生效、持有期末缺价按 0% 结算、tushare 路径接入但未实网验证。 +- 下一步 P1：接真 tushare 日线/复权/PIT 成分/可交易过滤，然后再扩因子和中性化。 diff --git a/BIAS_AUDIT.md b/BIAS_AUDIT.md new file mode 100644 index 0000000..e92655e --- /dev/null +++ b/BIAS_AUDIT.md @@ -0,0 +1,59 @@ +# Bias Audit (Phase 0) + +本文件记录 P0 框架对各类偏差/未来函数的处理状态与降级。每个小节标注当前状态(已处理 / 降级 / 待办)。 + +## 未来函数 / lookahead + +- 状态: **已处理(P0)**。 +- `momentum_20[t] = close[t] / close[t-window] - 1`,严格只用 t 及之前的收盘价(`groupby(symbol).shift(window)`)。 +- 事件顺序固定:在 t 收盘计算因子,t 收盘后调仓,从 t+1 持有。回测用**下一持有期**的收益结算,绝不使用因子已经看见的当日收益。 +- forward returns 只在 `analytics/` 计算,因子层永远拿不到未来收益(INV-001)。 + +## PIT 成分股 + +- 状态: **PIT 已实现(P1) / StaticUniverse 为离线降级**。 +- `PITIndexUniverse`(`universe.type=index`)用 tushare `index_weight` 的历史快照做 as-of 成分:`members(date)` 取 ≤date 的最近快照,绝不用未来快照(UNI-009)。被剔除的票在其在册期内仍是成员 —— 无幸存者偏差、无成分前视。 +- pipeline 构建 index universe 时会额外向回看 370 天成分快照,确保回测从两次成分调整中间开始时,起始日也能取到“开始日前最近快照”,而不是错误空仓。 +- 实证:沪深300 2024 全年 24 个快照、328 个不同成分(每快照 300),28 进28 出;`000069.SZ` 在 06-03 在册、06-28 已剔,各按其时代正确归属。 +- 数据坑:`index_weight` 单次约 6000 行上限,长窗口会**静默丢最早快照**;feed 已分 90 天窗口分页拉取规避。 +- `StaticUniverse`(`universe.type=static`,demo/离线用)成分与日期无关(UNI-003),是**降级**:存在幸存者 / 成分前视偏差,仅供无网络的 demo 跑通,并在 `phase0_summary.md` 的 DOWNGRADES 小节显式记录。 + +## 可交易过滤 + +- 状态: **停牌 / ST / 涨跌停已实现(P1)**。 +- `missing_close`(总是开):截面日 `close` 为 NaN 的标的不可交易(UNI-004)。 +- 统一在 `universe.filters.apply_tradable_filters` 按 `UniverseFilters` 开关执行;flag 由 `data.clean.tradability.enrich_tradability` 从 tushare `suspend_d` / `namechange` / `stk_limit` 富化到 panel(StaticUniverse 与 PITIndexUniverse 共用)。demo 无 flag 数据时各过滤自动 no-op。 +- **ST(UNI-006)**:`namechange` 名称区间含 'ST'/'*ST' 即标记,按 date 取生效名称(实证:`000005.SZ` 2024 全程 ST,正确剔除)。 +- **涨跌停(UNI-007)**:用**未复权 raw close** 与当日 raw `up_limit`/`down_limit` 比较,标记 `at_up_limit`/`at_down_limit`(qfq 复权价仅用于因子/回测收益;flag 富化在 front_adjust **之前**完成,故比较的是同口径 raw 价)。实证:`000005.SZ` 2024-02-01 触跌停。当前选股层对两个方向都剔除;**方向感知**(买入只看涨停、持有跌停不强卖)属执行层,后续细化。 +- **停牌(UNI-005)**:`suspend_d` 标记停牌日。**实测发现**:tushare 全天停牌当日**无 bar** → 已被 `missing_close` 剔除,故显式 suspended flag 与之重叠;其价值在盘中停牌(`suspend_timing`)或会给停牌日 bar 的数据源,属防御性。 +- 退市 / 无数据标的(如 `000003.SZ`)同样表现为不在 panel 而被剔除。PIT 历史成分见上节。 +- `universe.min_listing_days` 已在配置中(默认 60),但仍 **未执行**(no-op,降级):新上市标的不会被剔除。显式披露(INV-007),后续接上市日期后强制。 + +## ann_date 财务对齐 + +- 状态: **已实现(P1)**。 +- 财务因子(`roe` / `netprofit_yoy`)经 `data.clean.pit_financials.asof_financials` 按披露日 `ann_date` 做 backward as-of 对齐:每个 trade_date 只取 `ann_date <= trade_date` 的最近一期报告,**绝不按 `end_date`(报告期末)join**(DATA-012)。 +- 拉取窗口向回看约 16 个月(`start` 之前),确保回测 `start` 前已披露的上一期财报在集合内、能 as-of **carry forward** 到早期交易日,避免早期 NaN 缺口。 +- 实证:平安银行 2024 Q1(end_date 2024-03-31)披露日 ann_date 2024-04-20;as-of roe 在 04-19 仍是上一期年报值(10.2436),04-22 才切到 Q1(3.1176)——晚于报告期末约 3 周,证明无未来披露泄漏。 +- 财务因子仅在 tushare 数据路径可用;demo 无披露日,配置财务因子 + demo 源会报可读错误,**不伪造财务**。 + +## 复权 + +- 状态: **前复权已实现(P1)**。 +- panel 始终携带 `adj_factor` 列(DemoFeed 中恒为 1.0)。`data/clean/adjust.py` 的 `front_adjust` 用 `adj_factor` 做前复权(qfq),在 pipeline 读盘后、因子计算前于内存中应用(DATA-003)。 +- 约定:按 symbol 锚定窗口内最新日 (`qfq = raw × adj_factor / adj_factor[latest]`)。锚定项在任何价格比值中约掉,故所有收益率 / 因子值对锚定与扩窗都不变 —— PanelStore 保持 raw(+adj_factor),复权在内存做,batch≡incremental 一致。 +- 实证:平安银行 2024-06-14 除权,raw 当日 -5.74%(分红跳空),qfq +0.99%(真实涨跌);momentum_20 因此最多变动 6.77pp。demo(adj=1.0)下为恒等。 + +## 交易成本 + +- 状态: **已处理(P0)**。 +- 成本 = L1 换手 × `fee_rate`;`turnover = sum(|target_w - current_w|)`,在 symbol 并集上对齐计算。 +- 每个调仓期 `net_return = gross_return - cost`,成本拖累在`phase0_summary.md` 中汇总(BT-004)。slippage 参数已预留。 +- **结算价缺失约定(P0 降级)**:若持仓标的在持有期末(end)的 `close` 为 NaN(停牌 / 缺数据),回测以 0.0(持平)记其该期收益,而非剔除或用最近可得价结算。该约定在此显式披露(INV-007);P1 接入真实停牌/退市处理后改进结算逻辑。 + +## 中性化 + +- 状态: **行业 + 市值中性化已实现(P1)**。 +- `factors.process.neutralize.neutralize_by_date`:每个 date 截面把因子对 `[log(market_cap), one-hot(industry)]` 做 OLS,取残差,移除规模与行业暴露。缺行业 / 市值,或**残差自由度 ≤ 0**(名称数 ≤ 1+行业数,饱和拟合会给出无意义的伪 0 残差)时返回 **NaN**,绝不静默乱算;`processing.neutralize` 开启但协变量缺失(如 demo 路径)直接报可读错误。 +- 实证:12 只票横跨 4 行业(2024-09-30),corr(原始 momentum, log市值) = -0.617 → 中性化后 -0.000,各行业残差均值 ≈ 0,确认规模/行业暴露被移除。 +- **降级**:行业来自 `stock_basic.industry` 的**当前**行业标签,非按历史时点,故行业中性化带有轻微成分前视(市值 `daily_basic.total_mv` 为逐日真值)。PIT 行业历史是后续项,此降级在此显式披露(INV-007)。 diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..7c00719 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,78 @@ +# Quantitative_Trading — A股截面多因子框架 + +## 项目定位 +- A股 **截面多因子选股**（cross-sectional multi-factor）框架。**不做择时**。 +- 路径：研究/回测优先 → 渐进到实盘。 +- 构建方式：成熟工具 + **自建因子层**为核心。 +- 完整架构设计见 [`tmp/framework/architecture.html`](tmp/framework/architecture.html)。本文件是精简操作版，**细节以该文档为准**。 + +## 架构：7 层 +``` +data → universe → factors(特征) → alpha(合成/预测) → portfolio(+risk约束) → runtime(backtest|live) → analytics +``` + +| 层 | 职责 | +|---|---| +| `data/` | 采集(feed) / 清洗复权·对齐披露日(clean) / 面板存储(store) | +| `universe/` | PIT 成分股 + 可交易过滤(停牌/涨跌停/ST) | +| `factors/` | 截面因子 计算(compute) / 预处理三件套(process) / 落盘(store) | +| `alpha/` | 多因子合成/预测（独立成层） | +| `portfolio/` | 组合构建 + 风控**事前约束**(construct.py / risk.py) | +| `runtime/` | 回测 与 实盘 统一：driver + execution，两套实现(backtest/live) | +| `analytics/` | alphalens(因子检验) + quantstats(组合绩效) | + +## 不可违反的设计不变量（写代码必须守） +1. **factors 永不碰未来收益**；只有 `alpha` 层才用未来收益拟合权重（防未来函数边界）。 +2. **回测即实盘**：`runtime` 的 backtest/live 是同接口两实现；`factors/alpha/portfolio` 层两边复用、一行不改。 +3. **分层解耦**：factor 不碰数据源；portfolio 不碰下单。坏味道：策略里直接 `pro.daily(...)`。 +4. 决策是**截面的**（每个调仓日横向排序选股）；时序只用于算因子。 + +## 致命陷阱（correctness 红线，详见 architecture.html §8） +未来函数 · PIT 成分股 · 可交易过滤 · 财务按 `ann_date` 披露日对齐 · 前复权 · batch≡incremental 一致性 · 行业+市值中性化 · 交易成本/换手 · 过拟合(样本内外+IC稳定性)。 + +## 环境 +- **框架/测试环境**：conda `quant_mf`（Py 3.12）。运行 Phase 0、pytest、ruff、CLI 时用绝对路径 python： + `/home/shaofl/Development/env_tools/envs/quant_mf/bin/python` +- **数据拉取环境**：conda `data_fetch`（Py 3.12）。仅用于独立数据抓取/交互查数；在非交互 shell 里**用绝对路径 python**，不靠 activate： + `/home/shaofl/Development/env_tools/envs/data_fetch/bin/python` + +## 数据：tushare +- **token**：`/home/shaofl/Projects/financial_projects/.config.json`（key `tushare.token`）。 + ⚠️ **绝不打印、绝不写进 repo、绝不 commit。** 代码里从该文件读取，不硬编码。 +- **权限**：实测充足——个股日线 / 分钟(`stk_mins`) / 复权(`adj_factor`) / 成分股(`index_weight`) / 申万行业(`index_classify`) / 财务含`ann_date`(`income`,`fina_indicator`) 全可取。分钟级可直接上，无需先退回日线。 +- **MCP（可选开发工具）**：`financial_projects/.mcp.json` 有 tushare MCP，仅供开发期交互查数；**从 `financial_projects/` 启动 claude 才加载**。 +- **数据层 ETL 一律用 Python SDK（批量/增量），不要建在 MCP 上。** 注意 tushare 各接口有每分钟调用上限，批量拉取需限流+重试。 + +## 技术选型 +| 用途 | 选型 | +|---|---| +| 数据处理 | pandas（+ polars 可选）· numpy | +| 存储 | parquet（分钟级按 symbol/year 分区）· DuckDB | +| 因子检验 | alphalens-reloaded | +| 绩效 | quantstats | +| 回归/合成 | statsmodels · scikit-learn | +| 组合优化(后期) | cvxpy · riskfolio-lib | +| 实盘下单(后期) | vnpy / miniqmt(QMT) —— **仅作下单通道，不当回测引擎** | +| 配置/测试 | pydantic-settings + YAML · pytest | + +## 开发约定 +- **交流中文**；代码/注释/commit message 用**英文**。 +- **Git**：feature 分支 + PR。main 已有骨架；`data` 分支已含 P0+P1（PR #1 `data→main`，OPEN）。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 +- **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 +- 文件小而专（<800 行），immutable 优先。 + +## 当前进度 +- ✅ 7 层骨架 + 架构文档（`main`） +- ✅ **Phase 0 MVP**（PR #1）：DemoFeed → PanelStore → StaticUniverse → momentum_20 → zscore → EqualWeightAlpha → TopN 等权 → 月度回测（成本/换手）→ IC/绩效报告，单命令可复现。 +- ✅ **Phase 1 偏差边界**（PR #1，全部真数据实证）： + - 前复权（qfq；store 存 raw，内存复权 → batch≡incremental 安全） + - PIT 指数成分（`index_weight` as-of，survivorship-safe；370 天 pre-start 回看 + 90 天分页） + - 可交易过滤（停牌 / ST / 涨跌停；**涨跌停用未复权 raw close** 比 `stk_limit`） + - 财务 `ann_date` 披露日 as-of（绝不按 end_date；500 天 lookback carry forward） + - 行业 + 市值中性化（按 date 截面 OLS 残差；欠定/无自由度截面 → NaN） + - 路径感知降级披露（demo/static vs tushare/index/ann_date，绝不把 demo 当真实验证） +- ✅ 质量门：`pytest` **168 passed**；`ruff` clean；`validate-config`（demo + `config/example_tushare.yaml`）+ `run-phase0`（demo）均 OK。 +- ✅ 真数据实证（tushare，非 CI）：复权除权日 raw−5.74%→qfq+0.99% / CSI300 全年 24 快照 328 名换手 / ann_date Q1 延后至 04-20 / 中性化 corr −0.617→0。详见 `BIAS_AUDIT.md`、`artifacts/reports/phase1_summary.md`。 +- ⚠️ 剩余 P2（已显式披露）：行业标签用**当前值**非 PIT、涨跌停未方向感知、`min_listing_days` no-op、日线 only、简版 IC/绩效未走 alphalens/quantstats、demo 路径非真数据。 +- 路线图下一步：财务因子组合 / 历史 PIT 行业 / 更细交易约束（architecture.html §11）。 diff --git a/RUNBOOK.md b/RUNBOOK.md new file mode 100644 index 0000000..84cf867 --- /dev/null +++ b/RUNBOOK.md @@ -0,0 +1,150 @@ +# RUNBOOK — Phase 1 (A-share cross-sectional multi-factor, bias-boundary) + +Phase 0 = a runnable demo MVP (offline `DemoFeed`). Phase 1 adds the real-data +bias boundaries: front-adjustment, PIT index membership, tradability filters, +ann_date financial alignment, and industry+size neutralization. The demo path is +unchanged; the real path is opt-in via `config/example_tushare.yaml`. + + +## Environment + +- Python (always use the absolute path; do NOT rely on `activate`): + ```bash + /home/shaofl/Development/env_tools/envs/quant_mf/bin/python + ``` +- Editable install is already done (`pip install -e . --no-deps`), so `import qt`, + `import data`, `import factors`, ... work from the repo root. +- No tushare token is needed for Phase 0: `run-phase0` uses the offline, + deterministic `DemoFeed` (no network, no credentials). The token lives ONLY in + the external `/home/shaofl/Projects/financial_projects/.config.json` and is + never read, printed, or logged by the demo pipeline. + +## Commands + +Run from the repo root +`/home/shaofl/Projects/financial_projects/stocks_market/Quantitative_Trading`. + +### 1. Validate a config (CLI-001) + +```bash +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli \ + validate-config --config config/example.yaml +``` + +Expected: prints `OK`, exit code `0`. A bad/missing config prints a readable +`ERROR: ...` (never a raw traceback) and exits `1`. + +### 2. Run the end-to-end pipeline (CLI-002) + +```bash +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli \ + run-phase0 --config config/example.yaml +``` + +Expected (numbers depend on demo data): + +``` +OK run-phase0: ic_mean=0.8681, annual_return=... +report: artifacts/reports/phase0_summary.md +``` + +This chains: demo feed -> `normalize_panel` -> `PanelStore.write/read` -> +`StaticUniverse` -> `MomentumFactor.compute` -> `ProcessingPipeline.transform` -> +`EqualWeightAlpha.fit(None).predict` (per rebalance date) -> `TopNEqualWeight.build` +-> `BacktestDriver.run` (`SimExecution`) -> analytics (IC, performance) -> report. + +### 3. Stage helper sub-commands (CLI-005, optional) + +Phase 0 keeps a single reproducible spine; these run that spine and report the +stage of interest (they do not persist partial cross-process state): + +```bash +... -m qt.cli fetch-data --config config/example.yaml +... -m qt.cli compute-factors --config config/example.yaml +... -m qt.cli run-backtest --config config/example.yaml +``` + +## Expected artifacts + +All writes are confined to the configured `output.*` directories (SEC-003). With +the example config: + +```text +artifacts/data/daily.parquet # canonical (date, symbol) market panel +artifacts/factors/factors.parquet # momentum_20 factor panel +artifacts/reports/phase0_summary.md # ANA-005 run summary (+ DOWNGRADES section) +artifacts/logs/run_phase0.log # run log (no secrets) +``` + +`artifacts/` is git-ignored (SEC-002); the reports are fully regenerable by +re-running `run-phase0` (INV-006). Re-running over existing files is safe +(re-entrant). + +## Phase 1 — real-data (tushare) path + +The real path is documented by `config/example_tushare.yaml`. It needs the token +in `/home/shaofl/Projects/financial_projects/.config.json` (read from there, never +printed/committed) and hits tushare, so it is NOT run in CI. An index run loads +~300 names × daily plus flags/covariates (rate-limited), so it is heavy. + +```bash +# validate the real-path config (no network) +... -m qt.cli validate-config --config config/example_tushare.yaml +# run the real path (network + token; heavy for an index universe) +... -m qt.cli run-phase0 --config config/example_tushare.yaml +``` + +What the real path turns on (each is a correctness red-line, disclosed in the +report's DATA PATH / DOWNGRADES section and `BIAS_AUDIT.md`): + +| Toggle | Effect | Module | +|---|---|---| +| `data.source: tushare` | real **front-adjusted (qfq)** prices | `data/clean/adjust.py` | +| `universe.type: index` | **PIT** index membership (as-of, survivorship-safe) | `universe/index_universe.py` | +| `filters.{suspended,st,limit_up_down}` | **tradability** filtering | `universe/filters.py` + `data/clean/tradability.py` | +| `factors: [{name: roe}]` | **ann_date** PIT financial factor | `data/clean/pit_financials.py` | +| `processing.neutralize.enabled` | **industry + size** neutralization | `factors/process/neutralize.py` | + +Financial factors / index universe / neutralization require the tushare path; on +the demo source they raise a readable error instead of fabricating data. + +Correctness details (locked by tests): + +- **Price-limit flags use RAW close** vs the raw `stk_limit` (limits are quoted in + unadjusted price); flags are enriched BEFORE front-adjust, so the qfq close is + used only for factors/returns, never for the limit comparison. +- **Financials look back ~16 months** before `start`, so the most recent report + disclosed before the backtest starts is fetched and as-of carried forward onto + the early trade dates (no NaN gap), still gated by `ann_date <= trade_date`. +- **Neutralization returns NaN** on a saturated cross-section (names ≤ 1 + #industries, + i.e. no residual degrees of freedom) rather than fabricated ~0 residuals. + +## Quality gate + +```bash +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m pytest -q +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m ruff check . +``` + +## Downgrades / status + +Each run's active path + downgrades are disclosed in its summary's DATA PATH / +DOWNGRADES section (INV-007) and explained in `BIAS_AUDIT.md`. + +Implemented in P1 (real path): + +- **Front-adjust (qfq)** — `adj_factor`-based; store stays raw, adjust in memory + (batch≡incremental safe). +- **PIT index membership** — resolves the static-universe survivorship downgrade. +- **Tradability filters** — suspended / ST / limit-up-down (+ always missing_close). +- **ann_date financial alignment** — figures used only after disclosure date. +- **Industry + size neutralization** — per-date OLS residual. + +Still downgraded / deferred (disclosed): + +- **Demo path** uses offline `DemoFeed` — NOT real data (no PIT/financial meaning). +- **Static universe** option remains a PIT downgrade (use `type: index` for real). +- **Industry tag is current** (`stock_basic`), not point-in-time — mild downgrade. +- **min_listing_days** configured but not enforced (no-op). +- **Daily bars only**; **simple IC/perf** (not alphalens/quantstats); + limit filter is not yet trade-direction-aware (P2). diff --git a/TEST_REPORT.md b/TEST_REPORT.md new file mode 100644 index 0000000..dc8963f --- /dev/null +++ b/TEST_REPORT.md @@ -0,0 +1,102 @@ +# TEST_REPORT — Phase 1 (bias-boundary) + +## Commands + +Run from the repo root with the project python (env `quant_mf`): + +```bash +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m pytest -q +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m ruff check . +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/example.yaml +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli run-phase0 --config config/example.yaml +``` + +## Results + +| Gate | Command | Result | +|---|---|---| +| Unit + integration | `pytest -q` | **168 passed, 0 failed** | +| Lint | `ruff check .` | **All checks passed** | +| Config validation | `validate-config` (demo + `example_tushare.yaml`) | exit `0`, prints `OK` | +| End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | + +Counts below are the actual `pytest --collect-only` numbers (sum = 168). + +## Per-file breakdown — Phase 0 core (93) + +| Test file | Tests | Area | +|---|---|---| +| `test_project_bootstrap.py` | 10 | bootstrap / imports / config | +| `test_config.py` | 5 | config schema | +| `test_data_schema.py` | 6 | panel schema | +| `test_panel_store.py` | 6 | parquet store | +| `test_data_feed.py` | 5 | demo + tushare feed boundary | +| `test_universe.py` | 4 | static universe | +| `test_factors_momentum.py` | 6 | momentum (no-lookahead) | +| `test_factor_processing.py` | 4 | drop_missing + z-score | +| `test_alpha_equal_weight.py` | 4 | equal-weight alpha | +| `test_portfolio_topn.py` | 8 | TopN construction | +| `test_sim_execution.py` | 8 | sim execution + cost | +| `test_backtest_driver.py` | 7 | backtest driver | +| `test_analytics_factor.py` | 5 | IC / quantile | +| `test_analytics_performance.py` | 3 | performance metrics | +| `test_phase0_pipeline.py` | 8 | end-to-end pipeline | +| `test_bias_audit_report.py` | 4 | bias audit doc | + +## Per-file breakdown — Phase 1 bias-boundary (75) + +| Test file | Tests | Red-line / feature | +|---|---|---| +| `test_adjust.py` | 7 | front-adjust (qfq); ex-dividend gap removed | +| `test_tushare_throttle.py` | 5 | shared rate-limit + retry | +| `test_index_universe.py` | 7 | PIT index membership (as-of, survivorship) | +| `test_index_feed.py` | 4 | index_weight feed + 90-day paging | +| `test_index_pipeline.py` | 1 | pre-start snapshot lookback (as-of edge) | +| `test_config_index.py` | 3 | `universe.type=index` config | +| `test_tradability_filters.py` | 7 | shared suspended/ST/limit filter | +| `test_tradability_enrich.py` | 6 | flag enrichment onto panel | +| `test_tushare_flags.py` | 3 | suspend_d/namechange/stk_limit feed | +| `test_pit_financials.py` | 7 | **ann_date as-of** (no disclosure leak) | +| `test_tushare_fina.py` | 2 | fina_indicator feed | +| `test_factors_financial.py` | 3 | financial factor (roe/netprofit_yoy) | +| `test_financial_pipeline.py` | 5 | factor dispatch + demo-source guard | +| `test_neutralize.py` | 5 | industry+size residual orthogonality | +| `test_processing_neutralize.py` | 2 | neutralize wiring (covariates required) | +| `test_covariates_enrich.py` | 3 | industry + market_cap enrichment | +| `test_tushare_covariates.py` | 2 | stock_basic + daily_basic feed | +| `test_real_path_config.py` | 3 | demo vs real-path downgrade disclosure | +| **Total (P0 + P1)** | **168** | | + +## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) + +Verified directly against tushare (results recorded in `BIAS_AUDIT.md` and +`artifacts/reports/phase1_summary.md`): + +- **front-adjust**: 平安银行 2024-06-14 ex-dividend — raw −5.74% vs qfq +0.99%; + momentum_20 shifts up to 6.77pp. +- **PIT index**: CSI300 2024 — 24 snapshots, 328 distinct names (28 in / 28 out); + `members(2024-06-15)` uses the 2024-06-03 snapshot; a dropped name stays in its era. +- **ann_date**: 平安银行 Q1 (end 2024-03-31, ann 2024-04-20) — as-of roe stays the + prior annual (10.24) until 04-19, switches to Q1 (3.12) on 04-22; no leak. +- **neutralization**: 12 names / 4 industries — corr(momentum, log_mcap) + −0.617 → −0.000; per-industry residual means ≈ 0. + +## Notes + +- No test hits the network or reads the tushare token (TEST-002, INV-004): the whole + suite runs on `DemoFeed` / fixtures / monkeypatched SDKs. +- Financial factors, the PIT index universe, tradability filters and neutralization + all require the real tushare path; on demo they raise a readable error rather than + fabricate (verified by `test_financial_pipeline.py`, `_build_universe`, the + neutralize guard). +- The demo portfolio's annualized return is intentionally extreme (demo price paths + include a 3x jump); P0/P1 do not optimize for realistic returns — pipeline + correctness (event order, costs, no-lookahead) is what is under test. +- P1 acceptance hardening (locked by tests): price-limit flags compare the RAW + (unadjusted) close to the raw `stk_limit`, enriched BEFORE front-adjust + (`test_tradability_enrich::test_limit_flag_uses_raw_close_and_survives_front_adjust`); + financials are fetched ~16 months before `start` so the prior disclosed report + carries forward (`test_pit_financials::test_asof_carries_forward_report_disclosed_before_window`, + `test_financial_pipeline::test_financial_fetch_uses_lookback_before_start`); + neutralization returns NaN on a saturated cross-section instead of fabricated ~0 + residuals (`test_neutralize::test_saturated_cross_section_returns_nan_not_zeros`). diff --git a/alpha/base.py b/alpha/base.py new file mode 100644 index 0000000..8ad5fe3 --- /dev/null +++ b/alpha/base.py @@ -0,0 +1,45 @@ +"""AlphaModel port: combine/predict a single score from a factor panel. + +The alpha layer is the ONLY place allowed to see forward returns (for fitting +weights), and even then it must never pass them down to the factor layer +(CLAUDE.md invariant #1, ALPHA-004). ``EqualWeightAlpha`` (P0) ignores +forward_returns entirely. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class AlphaModel(ABC): + """Abstract multi-factor combination / prediction model.""" + + @abstractmethod + def fit( + self, + factors: pd.DataFrame, + forward_returns: pd.Series | None = None, + ) -> "AlphaModel": + """Fit the model and return self (for chaining). + + Args: + factors: MultiIndex(date, symbol) factor panel (training window). + forward_returns: optional MultiIndex(date, symbol) future returns, + used ONLY here to learn weights. Must never be forwarded to the + factor layer. EqualWeightAlpha accepts ``None`` (ALPHA-003). + """ + raise NotImplementedError + + @abstractmethod + def predict(self, factors_today: pd.DataFrame) -> pd.Series: + """Predict a single score per symbol for one cross-section. + + Args: + factors_today: factor rows for one date (MultiIndex or symbol-indexed). + + Returns: + A pd.Series of scores indexed by symbol. + """ + raise NotImplementedError diff --git a/alpha/equal_weight.py b/alpha/equal_weight.py new file mode 100644 index 0000000..9f3dbe5 --- /dev/null +++ b/alpha/equal_weight.py @@ -0,0 +1,96 @@ +"""EqualWeightAlpha: the P0 baseline multi-factor combiner. + +Score = plain row-wise mean across the factor columns. No forward returns, no +learned weights (ALPHA-001/002/003). This is deliberately the simplest possible +alpha so the end-to-end pipeline can go green before any weighting model +(ICWeightAlpha, regression) is added. + +Design (documented behavior): +- Single factor column -> the score equals that column. +- Multiple columns -> the equal-weight (arithmetic) mean of the columns. +- NaN-by-row -> mean over the *available* (non-NaN) factors for that + symbol; if *all* factors are NaN for a symbol, its score is NaN. This is the + pandas ``DataFrame.mean(axis=1)`` default (``skipna=True``) and keeps a + partially-observed symbol scorable instead of dropping it. +- Output is a symbol-indexed ``pd.Series``. A single-date MultiIndex + cross-section is collapsed to a plain symbol index; an already symbol-indexed + frame is passed through. + +The model is stateless: ``fit`` only validates and returns ``self`` for +chaining. It never reads ``forward_returns`` and never forwards them anywhere +(CLAUDE.md invariant #1 / ALPHA-004 boundary). +""" + +from __future__ import annotations + +import pandas as pd + +from alpha.base import AlphaModel + +_SYMBOL_LEVEL = "symbol" + + +class EqualWeightAlpha(AlphaModel): + """Combine factors into one score via an equal-weight row mean.""" + + def fit( + self, + factors: pd.DataFrame, + forward_returns: pd.Series | None = None, + ) -> "EqualWeightAlpha": + """Validate inputs and return self. + + ``forward_returns`` is accepted for interface compatibility but is + ignored entirely: the equal-weight baseline needs no future data + (ALPHA-003) and must never leak it downstream (ALPHA-004). + """ + self._check_factor_frame(factors) + return self + + def predict(self, factors_today: pd.DataFrame) -> pd.Series: + """Return a symbol-indexed score = row mean across factor columns. + + Args: + factors_today: factor rows for one cross-section. Either a + symbol-indexed frame, or a MultiIndex(date, symbol) frame whose + rows all belong to a single date. + + Returns: + ``pd.Series`` of scores indexed by symbol (name ``"score"``). + """ + self._check_factor_frame(factors_today) + if factors_today.shape[1] == 0: + raise ValueError( + "EqualWeightAlpha.predict needs at least one factor column; " + "got an empty factor frame." + ) + + # skipna=True -> average over the available factors per row; a row that + # is all-NaN yields NaN (documented behavior). + scores = factors_today.mean(axis=1) + scores = self._to_symbol_index(scores) + return scores.rename("score") + + @staticmethod + def _check_factor_frame(factors: pd.DataFrame) -> None: + """Fail fast with a readable error on a non-DataFrame input.""" + if not isinstance(factors, pd.DataFrame): + raise TypeError( + "EqualWeightAlpha expects a pandas DataFrame of factors " + f"(columns = factor names); got {type(factors).__name__}." + ) + + @staticmethod + def _to_symbol_index(scores: pd.Series) -> pd.Series: + """Collapse a (date, symbol) cross-section to a plain symbol index. + + A single-date MultiIndex is reduced by dropping the date level; an + already symbol-indexed Series is returned unchanged. + """ + index = scores.index + if isinstance(index, pd.MultiIndex) and _SYMBOL_LEVEL in (index.names or []): + symbols = index.get_level_values(_SYMBOL_LEVEL) + out = pd.Series(scores.to_numpy(), index=symbols) + out.index.name = _SYMBOL_LEVEL + return out + return scores diff --git a/analytics/factor.py b/analytics/factor.py new file mode 100644 index 0000000..fbda238 --- /dev/null +++ b/analytics/factor.py @@ -0,0 +1,233 @@ +"""Factor checks: forward returns, IC, IC summary, quantile returns. + +This module is the ONLY place in the framework allowed to compute forward +(future-looking) returns. The factor layer must never receive them — see +CLAUDE.md invariant #1 and CONTRACTS.md §3. + +Implementation note (INV-007 downgrade): the IC / quantile logic here is a +simple numpy/pandas implementation, not alphalens-reloaded. It is deterministic, +dependency-light, and easy to audit. ``alphalens`` is available and may be wired +in later for the HTML report; any such swap must be recorded in the phase0 +report. The simple version is what runs in P0. + +All functions are pure: inputs are never mutated. +""" + +from __future__ import annotations + +import math + +import numpy as np +import pandas as pd + +from data.clean.schema import DATE_LEVEL, validate_panel + +_DEFAULT_PERIODS: tuple[int, ...] = (1, 5, 20) +_VALID_IC_METHODS = ("spearman", "pearson") + + +def forward_returns( + panel: pd.DataFrame, + periods: tuple[int, ...] = _DEFAULT_PERIODS, +) -> pd.DataFrame: + """Per-symbol forward returns from FUTURE close prices. + + For each ``n`` in ``periods``, column ``forward_return_{n}d`` holds, for each + (date, symbol): + + close[t + n] / close[t] - 1 + + computed within each symbol's own time series (no cross-symbol leakage). The + last ``n`` dates of every symbol are NaN (no future bar exists yet). + + Allowed here only — analytics is the forward-return boundary (INV-001). + + Parameters + ---------- + panel : canonical market panel, MultiIndex(date, symbol), must contain + ``close``. Validated via the shared schema helper. + periods : forward horizons in trading days; must all be positive ints. + + Returns + ------- + DataFrame aligned to ``panel.index`` with one column per requested period. + """ + validate_panel(panel) + if "close" not in panel.columns: + raise ValueError("forward_returns requires a 'close' column in the panel.") + if not periods: + raise ValueError("forward_returns: 'periods' must be a non-empty tuple of ints.") + bad = [p for p in periods if not isinstance(p, int) or p <= 0] + if bad: + raise ValueError(f"forward_returns: periods must be positive ints; got {bad}.") + + # A non-positive denominator (close <= 0, or NaN) makes ``future/close - 1`` + # produce +/-inf, which would silently pollute IC / quantile stats. Mask the + # denominator to NaN first so an undefined return is correctly NaN, never inf. + close = panel["close"] + safe_close = close.where(close > 0.0) + # group by symbol so shift(-n) never reaches across symbols + grouped = safe_close.groupby(level="symbol", sort=False, group_keys=False) + out: dict[str, pd.Series] = {} + for n in periods: + future = grouped.shift(-n) + out[f"forward_return_{n}d"] = future / safe_close - 1.0 + + result = pd.DataFrame(out, index=panel.index) + return result + + +def _cross_section_corr(factor: pd.Series, fwd_return: pd.Series, method: str) -> float: + """Correlate one date's cross-section, dropping non-finite pairs. NaN if < 2.""" + # Treat +/-inf like NaN so a polluted return can never bias the correlation. + pair = pd.DataFrame({"f": factor, "r": fwd_return}).replace( + [np.inf, -np.inf], np.nan + ).dropna() + if len(pair) < 2: + return float("nan") + if pair["f"].nunique() < 2 or pair["r"].nunique() < 2: + # zero variance on either side -> correlation undefined + return float("nan") + return float(pair["f"].corr(pair["r"], method=method)) + + +def compute_ic( + factor: pd.Series, + fwd_return: pd.Series, + method: str = "spearman", +) -> pd.Series: + """Per-date cross-sectional information coefficient (IC). + + For each date, correlate the factor cross-section against the forward-return + cross-section. ``method="spearman"`` gives rank IC (default); ``"pearson"`` + gives linear IC. NaN factor/return pairs are dropped within each date before + correlating; a date with fewer than 2 valid pairs (or zero variance) yields + NaN for that date. + + Parameters + ---------- + factor, fwd_return : MultiIndex(date, symbol) Series, aligned on the same + index. They are inner-joined on the index, so partial overlap is fine. + method : "spearman" or "pearson". + + Returns + ------- + Series indexed by date, one IC per date, sorted by date. + """ + if method not in _VALID_IC_METHODS: + raise ValueError( + f"compute_ic: method must be one of {_VALID_IC_METHODS}; got {method!r}." + ) + aligned = pd.DataFrame({"f": factor, "r": fwd_return}) + if not isinstance(aligned.index, pd.MultiIndex) or aligned.index.nlevels != 2: + raise ValueError( + "compute_ic: factor and fwd_return must share a MultiIndex(date, symbol)." + ) + + dates = aligned.index.get_level_values(DATE_LEVEL) + ic_by_date: dict[pd.Timestamp, float] = {} + for date, block in aligned.groupby(dates, sort=True): + ic_by_date[date] = _cross_section_corr(block["f"], block["r"], method) + + ic = pd.Series(ic_by_date, name="ic") + ic.index.name = DATE_LEVEL + return ic.sort_index() + + +def ic_summary(ic: pd.Series) -> dict[str, float]: + """Summarize an IC series. + + Returns ``{"ic_mean": , "ic_ir": }`` where the information + ratio uses the sample std (ddof=1). NaN ICs are ignored. If fewer than two + finite ICs exist, ``ic_ir`` is NaN (std undefined). + """ + clean = ic.dropna() + if clean.empty: + return {"ic_mean": float("nan"), "ic_ir": float("nan")} + mean = float(clean.mean()) + std = float(clean.std(ddof=1)) if len(clean) > 1 else float("nan") + ic_ir = mean / std if math.isfinite(std) and std != 0 else float("nan") + return {"ic_mean": mean, "ic_ir": float(ic_ir)} + + +def quantile_returns( + factor: pd.Series, + fwd_return: pd.Series, + quantiles: int = 5, +) -> pd.DataFrame: + """Mean forward return per (date, quantile bucket). + + On each date the cross-section is split into ``quantiles`` buckets by factor + rank (bucket 1 = lowest factor, bucket ``quantiles`` = highest). Each cell is + the mean forward return of the symbols in that bucket on that date. + + Shape contract: a DataFrame with one ROW per date and one COLUMN per bucket + (columns ``1..quantiles``, integer-labelled). Buckets with no members on a + date are NaN. NaN factor/return pairs are dropped before bucketing. + + Parameters + ---------- + factor, fwd_return : aligned MultiIndex(date, symbol) Series. + quantiles : number of buckets (>= 2). + + Returns + ------- + DataFrame, index = date, columns = bucket labels 1..quantiles. + """ + if quantiles < 2: + raise ValueError(f"quantile_returns: 'quantiles' must be >= 2; got {quantiles}.") + # Drop non-finite (+/-inf as well as NaN) before bucketing so a polluted + # return cannot turn a whole bucket mean into inf (shown as 'n/a'). + aligned = ( + pd.DataFrame({"f": factor, "r": fwd_return}) + .replace([np.inf, -np.inf], np.nan) + .dropna() + ) + if aligned.empty: + return pd.DataFrame() + + dates = aligned.index.get_level_values(DATE_LEVEL) + rows: dict[pd.Timestamp, pd.Series] = {} + bucket_labels = list(range(1, quantiles + 1)) + for date, block in aligned.groupby(dates, sort=True): + buckets = _assign_buckets(block["f"], quantiles) + means = block["r"].groupby(buckets).mean() + rows[date] = means.reindex(bucket_labels) + + out = pd.DataFrame(rows).T + out.index.name = DATE_LEVEL + out.columns = bucket_labels + return out + + +def _assign_buckets(factor_cs: pd.Series, quantiles: int) -> pd.Series: + """Assign 1..quantiles bucket labels to a single date's factor cross-section. + + Uses rank-based qcut so duplicate factor values are spread deterministically. + If the cross-section is too small/degenerate for ``quantiles`` distinct + edges, falls back to a uniform rank split so every symbol still gets a label. + """ + n = len(factor_cs) + if n == 0: + return pd.Series([], dtype="int64", index=factor_cs.index) + ranks = factor_cs.rank(method="first") + try: + labels = pd.qcut(ranks, q=quantiles, labels=bucket_range(quantiles)) + return labels.astype("int64") + except ValueError: + # too few points for that many edges -> uniform split by order statistic + order = (ranks - 1) * quantiles // n + 1 + return order.clip(upper=quantiles).astype("int64") + + +def bucket_range(quantiles: int) -> list[int]: + """Bucket labels 1..quantiles (lowest factor = 1).""" + return list(range(1, quantiles + 1)) + + +__all__ = [ + "forward_returns", + "compute_ic", + "ic_summary", + "quantile_returns", +] diff --git a/analytics/performance.py b/analytics/performance.py new file mode 100644 index 0000000..1ccdbe1 --- /dev/null +++ b/analytics/performance.py @@ -0,0 +1,148 @@ +"""Portfolio performance metrics from a nav (or returns) series. + +Implements the P0 metrics required by ANA-004: annualized return, max drawdown, +volatility, Sharpe. ``performance_summary`` bundles them into a plain dict for +the phase0 report. + +Implementation note (INV-007 downgrade): these are simple, dependency-light +numpy/pandas computations, NOT quantstats/empyrical. ``quantstats`` is available +and may be used later for the richer HTML tearsheet; any such use must be +recorded in the phase0 report. The simple version is what runs in P0. + +All functions are pure: inputs are never mutated. + +Conventions +----------- +- nav: a level series (e.g. starts at 1.0); returns are derived as pct_change. +- max_drawdown is returned as a NEGATIVE fraction (e.g. -0.5 for a 50% fall), + or 0.0 if the nav never falls below a prior peak. +- Annualization uses ``periods_per_year`` (default 252 trading days). +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +_DEFAULT_PERIODS_PER_YEAR: int = 252 + + +def _to_returns(series: pd.Series, is_nav: bool) -> pd.Series: + """Coerce input to a clean per-period returns series. + + If ``is_nav`` the input is a level series and returns are its pct_change; + otherwise it is already a returns series. Leading NaN (from pct_change) and + any other NaN are dropped so downstream stats are well defined. + """ + s = pd.Series(series).astype(float) + rets = s.pct_change() if is_nav else s + return rets.dropna() + + +def _nav_from_input(series: pd.Series, is_nav: bool) -> pd.Series: + """Return a nav (level) series from either a nav or a returns input.""" + s = pd.Series(series).astype(float) + if is_nav: + return s.dropna() + return (1.0 + s.fillna(0.0)).cumprod() + + +def max_drawdown(nav: pd.Series) -> float: + """Maximum drawdown of a nav (level) series, as a negative fraction. + + drawdown[t] = nav[t] / running_peak[t] - 1; the result is the minimum of + that series (<= 0). A monotonically non-decreasing nav returns 0.0. An empty + or single-point nav returns 0.0 (no drawdown observable). + """ + s = pd.Series(nav).astype(float).dropna() + if len(s) < 2: + return 0.0 + running_peak = s.cummax() + drawdown = s / running_peak - 1.0 + return float(drawdown.min()) + + +def annualized_return(nav: pd.Series, periods_per_year: int = _DEFAULT_PERIODS_PER_YEAR) -> float: + """Geometric annualized return of a nav (level) series. + + Uses total compounded growth over the number of return periods: + (nav_end / nav_start) ** (periods_per_year / n_periods) - 1 + Returns 0.0 if fewer than two points exist. + """ + s = pd.Series(nav).astype(float).dropna() + if len(s) < 2: + return 0.0 + n_periods = len(s) - 1 + total_growth = s.iloc[-1] / s.iloc[0] + if total_growth <= 0: + # nav hit zero/negative: annualization undefined, report total - 1 + return float(total_growth - 1.0) + return float(total_growth ** (periods_per_year / n_periods) - 1.0) + + +def volatility( + nav: pd.Series, + periods_per_year: int = _DEFAULT_PERIODS_PER_YEAR, + is_nav: bool = True, +) -> float: + """Annualized volatility (std of per-period returns, scaled by sqrt(ppy)). + + Uses sample std (ddof=1). Returns 0.0 if fewer than two returns exist. + """ + rets = _to_returns(nav, is_nav) + if len(rets) < 2: + return 0.0 + return float(rets.std(ddof=1) * np.sqrt(periods_per_year)) + + +def sharpe( + nav: pd.Series, + risk_free: float = 0.0, + periods_per_year: int = _DEFAULT_PERIODS_PER_YEAR, + is_nav: bool = True, +) -> float: + """Annualized Sharpe ratio. + + mean(excess per-period return) / std(per-period return) * sqrt(ppy), where + the per-period risk-free is ``risk_free / periods_per_year``. Returns 0.0 if + volatility is zero or fewer than two returns exist. + """ + rets = _to_returns(nav, is_nav) + if len(rets) < 2: + return 0.0 + rf_per_period = risk_free / periods_per_year + excess = rets - rf_per_period + std = excess.std(ddof=1) + if std == 0 or not np.isfinite(std): + return 0.0 + return float(excess.mean() / std * np.sqrt(periods_per_year)) + + +def performance_summary( + nav: pd.Series, + risk_free: float = 0.0, + periods_per_year: int = _DEFAULT_PERIODS_PER_YEAR, + is_nav: bool = True, +) -> dict[str, float]: + """Bundle the P0 performance metrics into a plain dict. + + Always contains at least: ``annual_return``, ``max_drawdown``, + ``volatility``, ``sharpe``. ``is_nav=True`` treats the input as a level + series; set ``is_nav=False`` to pass a per-period returns series instead. + """ + nav_series = _nav_from_input(nav, is_nav) + return { + "annual_return": annualized_return(nav_series, periods_per_year), + "max_drawdown": max_drawdown(nav_series), + "volatility": volatility(nav_series, periods_per_year, is_nav=True), + "sharpe": sharpe(nav_series, risk_free, periods_per_year, is_nav=True), + } + + +__all__ = [ + "max_drawdown", + "annualized_return", + "volatility", + "sharpe", + "performance_summary", +] diff --git a/config/example.yaml b/config/example.yaml new file mode 100644 index 0000000..e3549ce --- /dev/null +++ b/config/example.yaml @@ -0,0 +1,91 @@ +# V1 example configuration. +# This file is a template for config/example.yaml. + +project: + name: quantitative_trading_phase0 + timezone: Asia/Shanghai + +data: + source: demo + # source: tushare + freq: D + start: "2024-01-01" + end: "2024-12-31" + external_secret_file: "/home/shaofl/Projects/financial_projects/.config.json" + tushare_token_key: "tushare.token" + output_name: daily + +universe: + type: static + symbols: + - "000001.SZ" + - "000002.SZ" + - "000003.SZ" + - "000004.SZ" + - "000005.SZ" + min_listing_days: 60 + filters: + missing_close: true + suspended: false + st: false + limit_up_down: false + +factors: + - name: momentum_20 + enabled: true + params: + window: 20 + price_col: close + +processing: + drop_missing: true + standardize: + enabled: true + method: zscore + winsorize: + enabled: false + method: mad + n: 3.0 + neutralize: + enabled: false + industry_col: industry + size_col: market_cap + +alpha: + model: equal_weight + params: {} + +portfolio: + constructor: topn_equal_weight + top_n: 3 + long_only: true + max_weight: null + turnover_cap: null + +backtest: + initial_nav: 1.0 + rebalance: monthly + event_order: close_to_next_period + cash_return: 0.0 + +cost: + fee_rate: 0.001 + slippage_rate: 0.0 + turnover_formula: l1 + +analytics: + forward_return_periods: + - 1 + - 5 + - 20 + quantiles: 5 + benchmark: null + +output: + root_dir: artifacts + data_dir: artifacts/data + factor_dir: artifacts/factors + report_dir: artifacts/reports + log_dir: artifacts/logs + overwrite: true + diff --git a/config/example_tushare.yaml b/config/example_tushare.yaml new file mode 100644 index 0000000..9907bc3 --- /dev/null +++ b/config/example_tushare.yaml @@ -0,0 +1,97 @@ +# Real-data (tushare) P1 path — the "does not fool itself" configuration. +# +# This documents the REAL path (contrast with config/example.yaml = demo/offline): +# - source: tushare -> real front-adjusted (qfq) prices +# - universe.type: index -> point-in-time CSI300 membership (survivorship-safe) +# - filters all on -> suspended / ST / limit_up_down tradability filtering +# - processing.neutralize on -> industry + market-cap neutralized factor +# +# To exercise the ann_date PIT financial path instead, set +# factors: [{name: roe}] (or net_profit_yoy) +# which is aligned by disclosure date (ann_date <= trade_date), never report period. +# +# NOTE: an index run loads ~300 names x daily (+ flags/covariates), so it is heavy +# and hits tushare; it is provided as documentation of the real path, not for CI. + +project: + name: quantitative_trading_phase1_tushare + timezone: Asia/Shanghai + +data: + source: tushare + freq: D + start: "2024-01-01" + end: "2024-12-31" + external_secret_file: "/home/shaofl/Projects/financial_projects/.config.json" + tushare_token_key: "tushare.token" + output_name: daily + +universe: + type: index + index_code: "000300.SH" + symbols: [] + min_listing_days: 60 + filters: + missing_close: true + suspended: true + st: true + limit_up_down: true + +factors: + - name: momentum_20 + enabled: true + params: + window: 20 + price_col: close + +processing: + drop_missing: true + standardize: + enabled: true + method: zscore + winsorize: + enabled: false + method: mad + n: 3.0 + neutralize: + enabled: true + industry_col: industry + size_col: market_cap + +alpha: + model: equal_weight + params: {} + +portfolio: + constructor: topn_equal_weight + top_n: 30 + long_only: true + max_weight: null + turnover_cap: null + +backtest: + initial_nav: 1.0 + rebalance: monthly + event_order: close_to_next_period + cash_return: 0.0 + +cost: + fee_rate: 0.001 + slippage_rate: 0.0 + turnover_formula: l1 + +analytics: + forward_return_periods: + - 1 + - 5 + - 20 + quantiles: 5 + benchmark: null + +output: + root_dir: artifacts + data_dir: artifacts/data + factor_dir: artifacts/factors + report_dir: artifacts/reports + log_dir: artifacts/logs + overwrite: true diff --git a/data/clean/adjust.py b/data/clean/adjust.py new file mode 100644 index 0000000..f0e159c --- /dev/null +++ b/data/clean/adjust.py @@ -0,0 +1,67 @@ +"""Front-adjustment (前复权 / qfq) of a raw market panel using ``adj_factor``. + +tushare ``daily`` returns RAW (unadjusted) OHLC plus a separate cumulative +``adj_factor`` series. Raw prices jump artificially across ex-dividend / split +dates; that fake jump pollutes return-based factors (e.g. momentum would see a +spurious negative return on an ex-date). Front adjustment removes it. + +Convention (P1): + + qfq_price[t] = raw_price[t] * adj_factor[t] / adj_factor[latest_in_window] + +anchored, PER SYMBOL, to that symbol's most recent date present in the panel. + +Why this anchor is safe for this framework (the batch == incremental guarantee): + The framework's factors are RETURN based (momentum = close[t]/close[t-w] - 1). + The anchor ``adj_factor[latest]`` is a constant per-symbol multiplier that + CANCELS in any price ratio, so every return — and therefore every factor value + and the backtest's holding-period return — is INVARIANT to the anchor and to + how far the window is extended. Only the absolute adjusted price *level* shifts + when the window grows, and nothing downstream reads the absolute level. So we + keep the PanelStore raw (with ``adj_factor``) and front-adjust in memory after + reading: batch and incremental runs stay consistent (no silent re-anchoring of + persisted data). +""" + +from __future__ import annotations + +from collections.abc import Sequence + +import pandas as pd + +from data.clean.schema import validate_panel + +# Price columns scaled by the adjustment ratio. volume/amount are left RAW +# (adjusting volume is a separate convention we deliberately do not adopt in P0). +ADJUSTABLE_COLUMNS: tuple[str, ...] = ("open", "high", "low", "close", "pre_close") + + +def front_adjust( + panel: pd.DataFrame, + price_columns: Sequence[str] = ADJUSTABLE_COLUMNS, +) -> pd.DataFrame: + """Return a NEW panel with front-adjusted (qfq) prices. + + Requires an ``adj_factor`` column (the feed must preserve it, DATA-003). + Symbols are adjusted independently. A panel whose ``adj_factor`` is all 1.0 + (e.g. :class:`DemoFeed`) is returned unchanged (identity), so demo runs and + P0 tests are unaffected. Pure: never mutates the input. + """ + if "adj_factor" not in panel.columns: + raise ValueError( + "front_adjust requires an 'adj_factor' column on the panel; the feed " + "must preserve it (DATA-003)." + ) + validate_panel(panel) + + out = panel.copy() + adj = out["adj_factor"] + # Panel is sorted by (date, symbol); within a symbol group the rows run in + # ascending date order, so "last" is that symbol's most recent adj_factor. + anchor = adj.groupby(level="symbol").transform("last") + ratio = adj / anchor + + for col in price_columns: + if col in out.columns: + out[col] = out[col] * ratio + return out diff --git a/data/clean/covariates.py b/data/clean/covariates.py new file mode 100644 index 0000000..effc079 --- /dev/null +++ b/data/clean/covariates.py @@ -0,0 +1,39 @@ +"""Join industry + market-cap covariates onto the panel (for neutralization). + +Adds two optional columns the neutralizer consumes: + + * ``industry`` — per-symbol industry tag, broadcast to every date. + * ``market_cap`` — per (date, symbol) total market value. + +Only the provided covariates are added; pure (never mutates the input panel). +""" + +from __future__ import annotations + +import pandas as pd + +from data.clean.schema import validate_panel + + +def enrich_covariates( + panel: pd.DataFrame, + *, + industry: dict[str, str] | None = None, + market_cap: pd.DataFrame | None = None, +) -> pd.DataFrame: + """Return a NEW panel with ``industry`` / ``market_cap`` columns added.""" + validate_panel(panel) + out = panel.copy() + symbols = out.index.get_level_values("symbol") + + if industry is not None: + out["industry"] = [industry.get(str(s)) for s in symbols] + + if market_cap is not None and not market_cap.empty: + mc = market_cap.copy() + mc["date"] = pd.to_datetime(mc["date"]).dt.normalize() + mc["symbol"] = mc["symbol"].astype(str) + mc = mc.drop_duplicates(["date", "symbol"]).set_index(["date", "symbol"]) + out["market_cap"] = mc["market_cap"].reindex(out.index) + + return out diff --git a/data/clean/pit_financials.py b/data/clean/pit_financials.py new file mode 100644 index 0000000..00287b0 --- /dev/null +++ b/data/clean/pit_financials.py @@ -0,0 +1,72 @@ +"""Point-in-time as-of alignment of financial data by disclosure date (ann_date). + +The correctness red-line: a financial figure (e.g. Q1 ROE) must only become visible +AFTER it was disclosed (``ann_date``), NOT from the period end (``end_date``). For +example a Q1 report with ``end_date=2024-03-31`` is typically announced weeks later +(``ann_date=2024-04-20``); joining on ``end_date`` would leak the figure into trade +dates 04-01..04-19 that could not have known it. + +``asof_financials`` attaches, for each ``(trade_date, symbol)``, the values of the +LATEST report whose ``ann_date <= trade_date`` (a backward as-of join keyed on +``ann_date``). Reports not yet disclosed are invisible; early dates with no prior +disclosure get NaN. This is the only place financial features enter the panel, and +it is structurally incapable of look-ahead. +""" + +from __future__ import annotations + +from collections.abc import Sequence + +import pandas as pd + + +def asof_financials( + index: pd.MultiIndex, + fina: pd.DataFrame, + fields: Sequence[str], +) -> pd.DataFrame: + """Return a ``(date, symbol)`` frame of as-of financial ``fields``. + + Args: + index: target MultiIndex(date, symbol) to align onto. + fina: financial records with columns ``symbol``, ``ann_date`` (disclosure + date; str ``YYYYMMDD`` or datetime) and the requested ``fields``. + fields: financial column names to attach (e.g. ``["roe"]``). + + Each output row carries the field values from the latest report announced on + or before that ``trade_date``; rows before any disclosure are NaN. + """ + fields = list(fields) + missing = [f for f in fields if f not in fina.columns] + if missing: + raise ValueError(f"financial records are missing field(s): {missing}.") + + f = fina.copy() + f["symbol"] = f["symbol"].astype(str) + f["ann_date"] = pd.to_datetime(f["ann_date"].astype(str), format="%Y%m%d", errors="coerce") + f = f.dropna(subset=["ann_date"]) + # de-dup identical disclosures; keep the last record per (symbol, ann_date) + f = f.sort_values("ann_date").drop_duplicates(["symbol", "ann_date"], keep="last") + + keys = pd.DataFrame( + { + "date": index.get_level_values("date"), + "symbol": index.get_level_values("symbol").astype(str), + "_pos": range(len(index)), + } + ) + keys_sorted = keys.sort_values("date") + + merged = pd.merge_asof( + keys_sorted, + f[["symbol", "ann_date", *fields]], + left_on="date", + right_on="ann_date", + by="symbol", + direction="backward", # latest ann_date <= trade_date (never future) + ).sort_values("_pos") + + out = pd.DataFrame(index=index) + for field in fields: + out[field] = merged[field].to_numpy() + return out diff --git a/data/clean/schema.py b/data/clean/schema.py new file mode 100644 index 0000000..deafd4e --- /dev/null +++ b/data/clean/schema.py @@ -0,0 +1,164 @@ +"""Panel schema contract for the cross-sectional multi-factor framework. + +All core panels (market data, factors) share one canonical shape: + + index: MultiIndex(date, symbol) + date -> pandas.Timestamp, normalized to midnight (date only) + symbol -> str, tushare style, e.g. "000001.SZ" + sorted by (date, symbol), no duplicate (date, symbol) pairs. + +This module defines that shape and the helpers to enforce it. It is pure: +every function returns a new object and never mutates its input. + +IMPORTANT (design invariant): this module is panel *shape* only. It must NOT +compute forward returns or anything that touches the future — forward-return +computation lives in ``analytics`` (see CLAUDE.md invariant #1). +""" + +from __future__ import annotations + +import pandas as pd + +# Minimal required market-panel columns. +CORE_COLUMNS: list[str] = [ + "open", + "high", + "low", + "close", + "volume", + "amount", + "adj_factor", +] + +# Columns that may be present but are not required by the schema. +OPTIONAL_COLUMNS: list[str] = [ + "pre_close", + "trade_status", + "is_st", + "limit_up", + "limit_down", + "industry", + "market_cap", +] + +# Canonical index level names. +DATE_LEVEL = "date" +SYMBOL_LEVEL = "symbol" +INDEX_NAMES: list[str] = [DATE_LEVEL, SYMBOL_LEVEL] + + +def _extract_date_symbol(df: pd.DataFrame) -> tuple[pd.Series, pd.Series, pd.DataFrame]: + """Pull (date, symbol) out of a frame whether they are columns or the index. + + Returns (date_series, symbol_series, payload_frame_without_keys). + Raises a readable ValueError if neither layout is present. + """ + # Case 1: already a (date, symbol) MultiIndex (any order / names). + if isinstance(df.index, pd.MultiIndex) and df.index.nlevels == 2: + names = list(df.index.names) + if names == INDEX_NAMES or set(names) == set(INDEX_NAMES): + reset = df.reset_index() + date = reset[DATE_LEVEL] + symbol = reset[SYMBOL_LEVEL] + payload = reset.drop(columns=INDEX_NAMES) + return date, symbol, payload + # Unnamed 2-level index: assume (date, symbol) order. + if all(n is None for n in names): + date = df.index.get_level_values(0).to_series(index=range(len(df))) + symbol = df.index.get_level_values(1).to_series(index=range(len(df))) + payload = df.reset_index(drop=True) + return date, symbol, payload + + # Case 2: date and symbol live as ordinary columns. + if DATE_LEVEL in df.columns and SYMBOL_LEVEL in df.columns: + reset = df.reset_index(drop=True) + date = reset[DATE_LEVEL] + symbol = reset[SYMBOL_LEVEL] + payload = reset.drop(columns=INDEX_NAMES) + return date, symbol, payload + + raise ValueError( + "Panel must provide 'date' and 'symbol' either as a MultiIndex(date, symbol) " + "or as two columns named 'date' and 'symbol'. " + f"Got index names={list(df.index.names)} and columns={list(df.columns)}." + ) + + +def normalize_panel(df: pd.DataFrame) -> pd.DataFrame: + """Return a new panel in canonical shape. + + Accepts ``df`` with (date, symbol) either as columns or as a MultiIndex. + Produces a fresh DataFrame with: + - MultiIndex(date, symbol), date normalized to midnight Timestamp, symbol str + - rows sorted by (date, symbol) + Raises ValueError if any CORE_COLUMNS are missing, or on duplicate + (date, symbol) pairs. NaN cell values are allowed (missing prices are legal). + + The input is never mutated. + """ + date, symbol, payload = _extract_date_symbol(df) + + # Normalize key dtypes. .copy() keeps us off the caller's frame. + date = pd.to_datetime(date).dt.normalize() + date.name = DATE_LEVEL + symbol = symbol.astype(str) + symbol.name = SYMBOL_LEVEL + + payload = payload.copy() + missing = [c for c in CORE_COLUMNS if c not in payload.columns] + if missing: + raise ValueError( + f"Panel is missing required core columns: {missing}. " + f"CORE_COLUMNS = {CORE_COLUMNS}." + ) + + out = payload.copy() + out.index = pd.MultiIndex.from_arrays([date.to_numpy(), symbol.to_numpy()], names=INDEX_NAMES) + + if out.index.duplicated().any(): + dups = out.index[out.index.duplicated(keep=False)].unique().tolist() + raise ValueError( + "Panel has duplicate (date, symbol) index entries; each pair must be unique. " + f"Example duplicates: {dups[:5]}." + ) + + out = out.sort_index() + return out + + +def validate_panel(df: pd.DataFrame) -> None: + """Assert that ``df`` already satisfies the canonical panel contract. + + Raises a readable ValueError if any invariant is violated. Returns None on + success. Does not mutate or return a frame — use ``normalize_panel`` to fix + shape. + """ + if not isinstance(df.index, pd.MultiIndex) or df.index.nlevels != 2: + raise ValueError( + "Panel index must be a 2-level MultiIndex(date, symbol); " + f"got {type(df.index).__name__} with names={list(df.index.names)}." + ) + if list(df.index.names) != INDEX_NAMES: + raise ValueError( + f"Panel index level names must be {INDEX_NAMES}; got {list(df.index.names)}." + ) + + date_level = df.index.get_level_values(DATE_LEVEL) + if not pd.api.types.is_datetime64_any_dtype(date_level): + raise ValueError( + f"Panel 'date' level must be datetime (pandas.Timestamp); got dtype {date_level.dtype}." + ) + + symbol_level = df.index.get_level_values(SYMBOL_LEVEL) + if not all(isinstance(s, str) for s in symbol_level): + raise ValueError("Panel 'symbol' level must contain only str values.") + + missing = [c for c in CORE_COLUMNS if c not in df.columns] + if missing: + raise ValueError(f"Panel is missing required core columns: {missing}.") + + if df.index.duplicated().any(): + raise ValueError("Panel has duplicate (date, symbol) index entries.") + + if not df.index.is_monotonic_increasing: + raise ValueError("Panel index must be sorted by (date, symbol).") diff --git a/data/clean/tradability.py b/data/clean/tradability.py new file mode 100644 index 0000000..f3580c1 --- /dev/null +++ b/data/clean/tradability.py @@ -0,0 +1,74 @@ +"""Enrich a market panel with tradability flag columns. + +Joins the signals from :class:`~data.feed.tushare_flags.TushareFlagsFeed` onto the +canonical (date, symbol) panel as boolean columns that +:func:`universe.filters.apply_tradable_filters` consults: + + * ``suspended`` — the (date, symbol) was halted that day (suspend_d 'S'). + * ``is_st`` — the name effective on that date contains 'ST' / '*ST'. + * ``at_up_limit`` — close is at/above the day's upper price limit (can't buy). + * ``at_down_limit`` — close is at/below the day's lower price limit (can't sell). + +Only the flags whose source is supplied are added, so callers can enrich +incrementally and the offline demo path (no flag sources) is a no-op. Pure: never +mutates the input panel. +""" + +from __future__ import annotations + +import pandas as pd + +from data.clean.schema import validate_panel + +# close within this fraction of the limit counts as "at limit". +_LIMIT_TOL = 1e-6 + + +def enrich_tradability( + panel: pd.DataFrame, + *, + suspended: set[tuple] | None = None, + st_intervals: dict[str, list[tuple]] | None = None, + limits: pd.DataFrame | None = None, +) -> pd.DataFrame: + """Return a NEW panel with the supplied tradability flag columns added.""" + validate_panel(panel) + out = panel.copy() + dates = out.index.get_level_values("date") + symbols = out.index.get_level_values("symbol") + + if suspended is not None: + out["suspended"] = [(d, s) in suspended for d, s in zip(dates, symbols)] + + if st_intervals is not None: + out["is_st"] = [_is_st(st_intervals, s, d) for d, s in zip(dates, symbols)] + + if limits is not None and not limits.empty: + lim = limits.copy() + lim["date"] = pd.to_datetime(lim["date"]).dt.normalize() + lim["symbol"] = lim["symbol"].astype(str) + lim = lim.set_index(["date", "symbol"]) + up = lim["up_limit"].reindex(out.index) + down = lim["down_limit"].reindex(out.index) + out["at_up_limit"] = (out["close"] >= up - _LIMIT_TOL) & up.notna() + out["at_down_limit"] = (out["close"] <= down + _LIMIT_TOL) & down.notna() + + return out + + +def _is_st( + intervals_by_symbol: dict[str, list[tuple]], symbol: str, date: pd.Timestamp +) -> bool: + """Is ``symbol`` ST on ``date``? Uses the effective (latest-starting) name.""" + intervals = intervals_by_symbol.get(symbol) + if not intervals: + return False + covering = [ + (start, is_st) + for start, end, is_st in intervals + if date >= start and (end is None or date <= end) + ] + if not covering: + return False + covering.sort(key=lambda item: item[0]) # effective name = latest start + return bool(covering[-1][1]) diff --git a/data/feed/base.py b/data/feed/base.py new file mode 100644 index 0000000..819470f --- /dev/null +++ b/data/feed/base.py @@ -0,0 +1,45 @@ +"""DataFeed port: the only layer that touches a raw data source. + +A DataFeed returns a market Panel and nothing else. It must NOT compute factors, +build portfolios, run backtests, or print/log any secret (tushare token). +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class DataFeed(ABC): + """Abstract market-data source. + + Implementations: ``DemoFeed`` (deterministic fixture), ``TushareFeed`` + (real API). Downstream layers depend on this interface, never on a concrete + source (CLAUDE.md invariant #3). + """ + + @abstractmethod + def get_bars( + self, + symbols: list[str], + start: str, + end: str, + freq: str = "D", + ) -> pd.DataFrame: + """Return a normalized market Panel for ``symbols`` over [start, end]. + + Args: + symbols: tushare-style codes, e.g. ["000001.SZ"]. + start, end: "YYYY-MM-DD" inclusive date bounds. + freq: bar frequency; "D" for daily (P0). "1min" reserved (P1). + + Returns: + A panel in canonical shape (see ``data.clean.schema``): + MultiIndex(date, symbol) with at least CORE_COLUMNS. + + Must not: + - compute factors / portfolios / backtests, + - print or log the tushare token (SEC-001). + """ + raise NotImplementedError diff --git a/data/feed/demo_feed.py b/data/feed/demo_feed.py new file mode 100644 index 0000000..8edb471 --- /dev/null +++ b/data/feed/demo_feed.py @@ -0,0 +1,132 @@ +"""DemoFeed: a deterministic, network-free DataFeed (DATA-002, CFG-005). + +Produces a canonical market panel from a fixed, reproducible price path — no +randomness, no ``datetime.now``. It lets the whole pipeline (and tests) run with +zero network and zero credentials. The price patterns mirror the adversarial +fixture (rising / falling / flat) so downstream factor/portfolio behaviour is +predictable, but this module is self-contained: it must not import ``tests``. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from data.clean.schema import normalize_panel +from data.feed.base import DataFeed + +# Built-in demo universe and its deterministic close patterns. +_DEMO_SYMBOLS: tuple[str, ...] = ( + "000001.SZ", # strictly rising + "000002.SZ", # strictly falling + "000003.SZ", # flat + "000004.SZ", # mild rise + "000005.SZ", # flat then jump +) +_BASE_PRICE: float = 100.0 +_JUMP_DAY: int = 30 +_JUMP_MULTIPLIER: float = 3.0 +# Daily multiplicative decay for the falling symbol: strictly < 1 (so price +# strictly falls each day) yet strictly > 0 (so it never crosses zero, keeping +# forward returns finite over the full multi-hundred-day demo calendar). +_FALL_DECAY: float = 0.99 + + +def _close_path(symbol: str, n: int) -> np.ndarray: + """Deterministic close path for ``symbol`` over ``n`` trading days.""" + t = np.arange(n, dtype=float) + if symbol == "000001.SZ": + return _BASE_PRICE + t + if symbol == "000002.SZ": + # Strictly falling but ALWAYS positive over the full calendar: a small + # multiplicative daily decay (never crosses zero, so forward returns stay + # finite — no inf pollution). Keeps the qualitative "low momentum" shape. + return _BASE_PRICE * (_FALL_DECAY**t) + if symbol == "000003.SZ": + return np.full(n, _BASE_PRICE) + if symbol == "000004.SZ": + return _BASE_PRICE + 0.5 * t + if symbol == "000005.SZ": + close = np.full(n, _BASE_PRICE) + if n > _JUMP_DAY: + close[_JUMP_DAY:] = _BASE_PRICE * _JUMP_MULTIPLIER + return close + # Unknown symbol: a gentle deterministic ramp keyed off its hash, so any + # requested code still yields a stable, positive, reproducible series. + seed = abs(hash(symbol)) % 7 + 1 + return _BASE_PRICE + (t * seed) / 10.0 + + +class DemoFeed(DataFeed): + """Deterministic offline market-data source. + + Honors the requested ``symbols`` and the inclusive ``[start, end]`` window. + Output is a canonical panel (``normalize_panel``) with all CORE_COLUMNS and + ``adj_factor == 1.0``. No network, no randomness, no wall-clock. + """ + + def __init__(self, calendar_start: str = "2024-01-01", calendar_days: int = 250) -> None: + # A fixed business-day calendar the demo prices live on. Requests outside + # it simply return the overlapping slice (possibly empty). + self._calendar_start = calendar_start + self._calendar_days = int(calendar_days) + + def get_bars( + self, + symbols: list[str], + start: str, + end: str, + freq: str = "D", + ) -> pd.DataFrame: + """Return a normalized demo panel for ``symbols`` over [start, end].""" + if freq not in ("D", "1d", "daily"): + raise ValueError( + f"DemoFeed only supports daily bars (freq='D'); got freq={freq!r}." + ) + if not symbols: + raise ValueError("DemoFeed.get_bars requires a non-empty symbol list.") + + start_ts = pd.Timestamp(start).normalize() + end_ts = pd.Timestamp(end).normalize() + if start_ts > end_ts: + raise ValueError( + f"start ({start}) must be on or before end ({end}) for DemoFeed.get_bars." + ) + + calendar = pd.bdate_range(start=self._calendar_start, periods=self._calendar_days) + rows: list[dict] = [] + for symbol in symbols: + close = _close_path(symbol, len(calendar)) + for i, day in enumerate(calendar): + if day < start_ts or day > end_ts: + continue + c = float(close[i]) + rows.append( + { + "date": day, + "symbol": symbol, + "open": c, + "high": c * 1.01, + "low": c * 0.99, + "close": c, + "volume": 1_000_000.0 + i * 1_000.0, + "amount": (1_000_000.0 + i * 1_000.0) * c, + "adj_factor": 1.0, + } + ) + + raw = pd.DataFrame( + rows, + columns=[ + "date", + "symbol", + "open", + "high", + "low", + "close", + "volume", + "amount", + "adj_factor", + ], + ) + return normalize_panel(raw) diff --git a/data/feed/index_feed.py b/data/feed/index_feed.py new file mode 100644 index 0000000..46e5d57 --- /dev/null +++ b/data/feed/index_feed.py @@ -0,0 +1,119 @@ +"""IndexConstituentsFeed: point-in-time index membership from tushare. + +``index_weight`` returns periodic snapshots of an index's constituents +(``index_code, con_code, trade_date, weight``). This feed maps them onto the +canonical ``(date, symbol)`` shape (con_code -> symbol) so the PIT universe can +answer "who was in the index AS OF date d" using the latest snapshot <= d — with +no look-ahead and no survivorship bias (a name dropped later is still present in +the earlier snapshots). + +The feed only pulls and normalizes membership; it does not decide tradability or +touch portfolio logic. The token is read from the external config and never +printed/logged (same contract as :class:`TushareFeed`). +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pandas as pd + +from data.feed.throttle import request_with_retry +from data.feed.tushare_feed import _lookup_dotted + +# canonical constituents columns +CONSTITUENT_COLUMNS: tuple[str, ...] = ("date", "symbol", "weight") + + +class IndexConstituentsFeed: + """Pulls PIT index constituents from tushare ``index_weight``.""" + + def __init__( + self, + secret_file: str, + token_key: str = "tushare.token", + rate_limit: int | None = None, + max_retries: int = 3, + ) -> None: + self._secret_file = str(secret_file) + self._token_key = token_key + self._rate_limit = rate_limit + self._max_retries = max(1, int(max_retries)) + self._pro = None # lazily built + + # -- secret handling (token never logged) ------------------------------- # + def _read_token(self) -> str: + path = Path(self._secret_file) + if not path.exists(): + raise ValueError( + f"Secret config file not found: {self._secret_file}. " + f"Set data.external_secret_file to your .config.json path." + ) + try: + data = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValueError( + f"Secret config file is not valid JSON: {self._secret_file} ({exc.msg})." + ) from None + return _lookup_dotted(data, self._token_key) + + def _client(self): + if self._pro is None: + import tushare as ts + + self._pro = ts.pro_api(self._read_token()) # token handed straight in + return self._pro + + # tushare index_weight caps a single response at ~6000 rows; a ~300-name + # index therefore truncates beyond ~20 snapshots and SILENTLY drops the + # earliest dates. We page the window in chunks small enough to stay under the + # cap so no snapshot is lost. + _WINDOW_DAYS = 90 + + # -- API ---------------------------------------------------------------- # + def get_constituents(self, index_code: str, start: str, end: str) -> pd.DataFrame: + """Return constituent snapshots for ``index_code`` over [start, end]. + + Paged in <=90-day windows to dodge tushare's per-call row cap (otherwise a + full-year pull silently loses the earliest snapshots). Output columns: + ``date`` (Timestamp), ``symbol`` (str), ``weight`` (float), sorted by + (date, symbol), de-duplicated across window boundaries. Empty + (schema-shaped) frame if tushare returns nothing — not an error. + """ + pro = self._client() + start_ts, end_ts = pd.Timestamp(start), pd.Timestamp(end) + + frames: list[pd.DataFrame] = [] + win_start = start_ts + while win_start <= end_ts: + win_end = min(win_start + pd.Timedelta(days=self._WINDOW_DAYS - 1), end_ts) + raw = request_with_retry( + pro.index_weight, + max_retries=self._max_retries, + rate_limit=self._rate_limit, + index_code=index_code, + start_date=win_start.strftime("%Y%m%d"), + end_date=win_end.strftime("%Y%m%d"), + ) + if raw is not None and len(raw) > 0: + frames.append(raw) + win_start = win_end + pd.Timedelta(days=1) + + if not frames: + return self._empty() + + df = pd.concat(frames, ignore_index=True) + df = df.rename(columns={"con_code": "symbol", "trade_date": "date"}) + df["date"] = pd.to_datetime(df["date"].astype(str), format="%Y%m%d") + df["symbol"] = df["symbol"].astype(str) + df = df[list(CONSTITUENT_COLUMNS)].drop_duplicates(["date", "symbol"]) + return df.sort_values(["date", "symbol"]).reset_index(drop=True) + + @staticmethod + def _empty() -> pd.DataFrame: + return pd.DataFrame( + {"date": pd.Series([], dtype="datetime64[ns]"), + "symbol": pd.Series([], dtype=object), + "weight": pd.Series([], dtype=float)} + ) diff --git a/data/feed/secret.py b/data/feed/secret.py new file mode 100644 index 0000000..7f5f30d --- /dev/null +++ b/data/feed/secret.py @@ -0,0 +1,45 @@ +"""Read the tushare token from the external secret file — never echoed. + +Shared by the tushare-backed feeds so the dotted-key lookup + readable errors +live in one place. Error messages name the key path but NEVER include any value, +so a malformed config cannot leak the token. +""" + +from __future__ import annotations + +import json +from pathlib import Path + + +def lookup_dotted(data: dict, dotted_key: str) -> str: + """Resolve a dotted key (e.g. ``'tushare.token'``) in a nested dict.""" + node: object = data + for part in dotted_key.split("."): + if not isinstance(node, dict) or part not in node: + raise ValueError( + f"Secret config is missing key '{dotted_key}'. " + f"Expected a nested entry reachable via that dotted path." + ) + node = node[part] + if not isinstance(node, str) or not node: + raise ValueError( + f"Secret config key '{dotted_key}' must map to a non-empty string token." + ) + return node + + +def read_token(secret_file: str, token_key: str = "tushare.token") -> str: + """Load and return the token from ``secret_file`` (json, dotted ``token_key``).""" + path = Path(secret_file) + if not path.exists(): + raise ValueError( + f"Secret config file not found: {secret_file}. " + f"Set data.external_secret_file to your .config.json path." + ) + try: + data = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValueError( + f"Secret config file is not valid JSON: {secret_file} ({exc.msg})." + ) from None + return lookup_dotted(data, token_key) diff --git a/data/feed/throttle.py b/data/feed/throttle.py new file mode 100644 index 0000000..b6cee15 --- /dev/null +++ b/data/feed/throttle.py @@ -0,0 +1,45 @@ +"""Shared rate-limit + retry for tushare endpoints (SEC-004). + +A tushare endpoint is a plain callable (e.g. ``pro.daily``, ``pro.index_weight``). +``request_with_retry`` retries transient failures with exponential backoff, then +sleeps to respect a per-minute call cap. It never echoes the token: the failure +message carries only the exception TYPE, not its payload. +""" + +from __future__ import annotations + +import time +from collections.abc import Callable +from typing import Any + + +def request_with_retry( + fn: Callable[..., Any], + *, + max_retries: int = 3, + rate_limit: int | None = None, + **kwargs: Any, +) -> Any: + """Call ``fn(**kwargs)`` with retry-on-error + per-minute throttle. + + Retries transient exceptions with exponential backoff up to ``max_retries`` + attempts; on success sleeps ``60 / rate_limit`` seconds (no-op if unset). + Raises a readable ``RuntimeError`` (type only, never the token) if every + attempt fails. + """ + attempts = max(1, int(max_retries)) + last_exc: Exception | None = None + for attempt in range(attempts): + try: + result = fn(**kwargs) + except Exception as exc: # transient API / network error + last_exc = exc + time.sleep(min(2.0**attempt, 8.0)) + continue + if rate_limit: + time.sleep(60.0 / rate_limit) + return result + raise RuntimeError( + f"tushare call failed after {attempts} attempt(s): " + f"{type(last_exc).__name__}. Check connectivity / rate limit." + ) from last_exc diff --git a/data/feed/tushare_covariates.py b/data/feed/tushare_covariates.py new file mode 100644 index 0000000..39472d7 --- /dev/null +++ b/data/feed/tushare_covariates.py @@ -0,0 +1,89 @@ +"""TushareCovariatesFeed: industry + market cap for neutralization. + +Provides the two cross-sectional covariates the neutralizer needs: + + * ``industry(symbols)`` -> {symbol: industry} (tushare ``stock_basic``). + * ``market_cap(symbols, s, e)`` -> DataFrame[date, symbol, market_cap] + (tushare ``daily_basic.total_mv``, in 10k CNY; only the log is used, so + units do not matter). + +Caveat (disclosed in the bias audit): ``stock_basic.industry`` is the CURRENT +industry tag, not a point-in-time history, so industry neutralization carries a +mild membership-style look-ahead. Market cap is genuinely per-date. Token is read +from the external config and never printed; the client is lazy. +""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.secret import read_token +from data.feed.throttle import request_with_retry + + +class TushareCovariatesFeed: + """Loads industry (stock_basic) and market cap (daily_basic) from tushare.""" + + def __init__( + self, + secret_file: str, + token_key: str = "tushare.token", + rate_limit: int | None = None, + max_retries: int = 3, + ) -> None: + self._secret_file = str(secret_file) + self._token_key = token_key + self._rate_limit = rate_limit + self._max_retries = max(1, int(max_retries)) + self._pro = None + + def _client(self): + if self._pro is None: + import tushare as ts + + self._pro = ts.pro_api(read_token(self._secret_file, self._token_key)) + return self._pro + + def _call(self, fn, **kwargs): + return request_with_retry( + fn, max_retries=self._max_retries, rate_limit=self._rate_limit, **kwargs + ) + + def industry(self, symbols: list[str]) -> dict[str, str]: + """Return {symbol: industry} for ``symbols`` (current tag, not PIT).""" + pro = self._client() + df = self._call(pro.stock_basic, fields="ts_code,industry") + if df is None or len(df) == 0: + return {} + wanted = set(map(str, symbols)) + df = df[df["ts_code"].astype(str).isin(wanted)] + return {str(r.ts_code): r.industry for r in df.itertuples()} + + def market_cap(self, symbols: list[str], start: str, end: str) -> pd.DataFrame: + """Return DataFrame[date, symbol, market_cap] from daily_basic.total_mv.""" + pro = self._client() + s = pd.Timestamp(start).strftime("%Y%m%d") + e = pd.Timestamp(end).strftime("%Y%m%d") + frames: list[pd.DataFrame] = [] + for sym in symbols: + df = self._call( + pro.daily_basic, + ts_code=sym, + start_date=s, + end_date=e, + fields="ts_code,trade_date,total_mv", + ) + if df is not None and len(df) > 0: + frames.append(df) + if not frames: + return pd.DataFrame( + {"date": pd.Series([], dtype="datetime64[ns]"), + "symbol": pd.Series([], dtype=object), + "market_cap": pd.Series([], dtype=float)} + ) + out = pd.concat(frames, ignore_index=True).rename( + columns={"ts_code": "symbol", "total_mv": "market_cap"} + ) + out["date"] = pd.to_datetime(out["trade_date"].astype(str), format="%Y%m%d") + out["symbol"] = out["symbol"].astype(str) + return out[["date", "symbol", "market_cap"]] diff --git a/data/feed/tushare_feed.py b/data/feed/tushare_feed.py new file mode 100644 index 0000000..0d2f03a --- /dev/null +++ b/data/feed/tushare_feed.py @@ -0,0 +1,203 @@ +"""TushareFeed: the real-API DataFeed (DATA-001/004, SEC-001/004). + +Reads the tushare token from an EXTERNAL json config file (never hardcoded, never +committed) and maps tushare's daily fields onto the canonical panel schema. The +token is treated as a secret: it is never printed, logged, or exposed via repr. +The tushare client is built lazily so importing/constructing this class needs no +network and no credentials. + +Field mapping (tushare ``daily`` -> CORE_COLUMNS): + ts_code -> symbol + trade_date -> date + open/high/low/close -> open/high/low/close + vol -> volume + amount -> amount + adj_factor -> adj_factor (joined from the ``adj_factor`` endpoint; 1.0 if absent) +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pandas as pd + +from data.clean.schema import CORE_COLUMNS, normalize_panel +from data.feed.base import DataFeed +from data.feed.throttle import request_with_retry + +# tushare raw column -> canonical column. +_FIELD_MAP: dict[str, str] = { + "ts_code": "symbol", + "trade_date": "date", + "vol": "volume", +} + + +def _lookup_dotted(data: dict, dotted_key: str) -> str: + """Resolve a dotted key (e.g. 'tushare.token') in a nested dict. + + Raises a readable ValueError if any segment is missing — the message names the + missing key path but NEVER echoes any value (so no secret can leak). + """ + node: object = data + for part in dotted_key.split("."): + if not isinstance(node, dict) or part not in node: + raise ValueError( + f"Secret config is missing key '{dotted_key}'. " + f"Expected a nested entry reachable via that dotted path." + ) + node = node[part] + if not isinstance(node, str) or not node: + raise ValueError( + f"Secret config key '{dotted_key}' must map to a non-empty string token." + ) + return node + + +class TushareFeed(DataFeed): + """A DataFeed backed by the tushare pro API. + + The token lives in an external json file; this class reads it on first use + and hands it straight to the tushare client. It never stores the token in a + way that a repr/log could surface. + """ + + def __init__( + self, + secret_file: str, + token_key: str = "tushare.token", + rate_limit: int | None = None, + max_retries: int = 3, + ) -> None: + self._secret_file = str(secret_file) + self._token_key = token_key + # SEC-004: per-minute call cap (calls/min) + retry on transient errors. + self._rate_limit = rate_limit + self._max_retries = max(1, int(max_retries)) + self._pro = None # lazily built tushare pro client + + # -- secret handling ---------------------------------------------------- # + def _read_token(self) -> str: + """Read the token from the external json config (dotted-key lookup).""" + path = Path(self._secret_file) + if not path.exists(): + raise ValueError( + f"Secret config file not found: {self._secret_file}. " + f"Set data.external_secret_file to your .config.json path." + ) + try: + data = json.loads(path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ValueError( + f"Secret config file is not valid JSON: {self._secret_file} ({exc.msg})." + ) from None + return _lookup_dotted(data, self._token_key) + + def _client(self): + """Build (once) and return the tushare pro client. Lazy + no logging.""" + if self._pro is None: + import tushare as ts + + token = self._read_token() + # Do NOT log/print token. Hand it directly to the SDK. + self._pro = ts.pro_api(token) + return self._pro + + # -- rate limit + retry (SEC-004) --------------------------------------- # + def _call(self, fn, **kwargs): + """Invoke a tushare endpoint with the shared retry + per-minute throttle.""" + return request_with_retry( + fn, + max_retries=self._max_retries, + rate_limit=self._rate_limit, + **kwargs, + ) + + # -- DataFeed API ------------------------------------------------------- # + def get_bars( + self, + symbols: list[str], + start: str, + end: str, + freq: str = "D", + ) -> pd.DataFrame: + """Return a normalized market panel for ``symbols`` over [start, end].""" + if freq not in ("D", "1d", "daily"): + raise ValueError( + f"TushareFeed currently supports only daily bars (freq='D'); " + f"got freq={freq!r}. Minute support is reserved (DATA-011)." + ) + if not symbols: + raise ValueError("TushareFeed.get_bars requires a non-empty symbol list.") + + pro = self._client() + start_compact = pd.Timestamp(start).strftime("%Y%m%d") + end_compact = pd.Timestamp(end).strftime("%Y%m%d") + + frames: list[pd.DataFrame] = [] + for symbol in symbols: + raw = self._call( + pro.daily, + ts_code=symbol, + start_date=start_compact, + end_date=end_compact, + ) + if raw is None or len(raw) == 0: + continue + adj = self._fetch_adj_factor(pro, symbol, start_compact, end_compact) + frames.append(self._to_canonical(raw, adj)) + + if not frames: + # Empty result is not an error: build an empty canonical panel. + return self._empty_panel() + + combined = pd.concat(frames, ignore_index=True) + return normalize_panel(combined) + + # -- mapping helpers ---------------------------------------------------- # + def _fetch_adj_factor(self, pro, symbol: str, start_compact: str, end_compact: str): + """Fetch the adj_factor series for ``symbol``; tolerate absence.""" + getter = getattr(pro, "adj_factor", None) + if getter is None: + return None + adj = self._call( + getter, ts_code=symbol, start_date=start_compact, end_date=end_compact + ) + if adj is None or len(adj) == 0: + return None + return adj[["ts_code", "trade_date", "adj_factor"]].copy() + + def _to_canonical(self, raw: pd.DataFrame, adj: pd.DataFrame | None) -> pd.DataFrame: + """Rename tushare columns to canonical names and attach adj_factor.""" + df = raw.rename(columns=_FIELD_MAP).copy() + + if adj is not None: + adj = adj.rename(columns=_FIELD_MAP) + df = df.merge(adj, on=["symbol", "date"], how="left") + if "adj_factor" not in df.columns: + df["adj_factor"] = 1.0 + df["adj_factor"] = df["adj_factor"].fillna(1.0) + + # tushare trade_date is a compact 'YYYYMMDD' string; parse to datetime. + df["date"] = pd.to_datetime(df["date"].astype(str), format="%Y%m%d") + df["symbol"] = df["symbol"].astype(str) + + keep = ["date", "symbol", *CORE_COLUMNS] + present = [c for c in keep if c in df.columns] + return df[present] + + @staticmethod + def _empty_panel() -> pd.DataFrame: + """An empty but schema-shaped panel (used when tushare returns nothing).""" + index = pd.MultiIndex.from_arrays( + [pd.DatetimeIndex([]), pd.Index([], dtype=object)], + names=["date", "symbol"], + ) + return pd.DataFrame(columns=CORE_COLUMNS, index=index) + + def __repr__(self) -> str: # never leak the token + return ( + f"TushareFeed(secret_file={self._secret_file!r}, " + f"token_key={self._token_key!r}, rate_limit={self._rate_limit!r})" + ) diff --git a/data/feed/tushare_fina.py b/data/feed/tushare_fina.py new file mode 100644 index 0000000..d6d9807 --- /dev/null +++ b/data/feed/tushare_fina.py @@ -0,0 +1,102 @@ +"""TushareFinancialFeed: financial records with disclosure dates (ann_date). + +Pulls tushare ``fina_indicator`` and returns the raw records WITH both ``ann_date`` +(disclosure date) and ``end_date`` (report period). The point-in-time alignment +(``ann_date <= trade_date``) is done downstream in +:func:`data.clean.pit_financials.asof_financials`; this feed only fetches and +normalizes, it never joins to trade dates and never looks ahead. + +Token is read from the external config and never printed/logged; the client is +built lazily. +""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.secret import read_token +from data.feed.throttle import request_with_retry + +# financial fields supported as P1 factors. +DEFAULT_FIELDS: tuple[str, ...] = ("roe", "netprofit_yoy") + + +class TushareFinancialFeed: + """Loads ``fina_indicator`` records (with ann_date) from tushare.""" + + def __init__( + self, + secret_file: str, + token_key: str = "tushare.token", + rate_limit: int | None = None, + max_retries: int = 3, + ) -> None: + self._secret_file = str(secret_file) + self._token_key = token_key + self._rate_limit = rate_limit + self._max_retries = max(1, int(max_retries)) + self._pro = None + + def _client(self): + if self._pro is None: + import tushare as ts + + self._pro = ts.pro_api(read_token(self._secret_file, self._token_key)) + return self._pro + + def get_fina_indicator( + self, + symbols: list[str], + start: str, + end: str, + fields: list[str] | None = None, + ) -> pd.DataFrame: + """Return financial records whose REPORT PERIOD falls in [start, end]. + + tushare ``fina_indicator`` filters ``start_date``/``end_date`` by the + report period (``end_date``), NOT by the announcement date — every report + whose period ends in the window is returned, regardless of when it was + disclosed. Both ``ann_date`` (disclosure) and ``end_date`` (period) are + returned; the point-in-time ``ann_date <= trade_date`` alignment is done + downstream in :func:`data.clean.pit_financials.asof_financials`. This feed + never joins to trade dates and never looks ahead. + + Output columns: ``symbol``, ``ann_date`` (str YYYYMMDD), ``end_date``, and + the requested ``fields``. + """ + wanted = list(fields) if fields else list(DEFAULT_FIELDS) + col_spec = ",".join(["ts_code", "ann_date", "end_date", *wanted]) + pro = self._client() + s = pd.Timestamp(start).strftime("%Y%m%d") + e = pd.Timestamp(end).strftime("%Y%m%d") + + frames: list[pd.DataFrame] = [] + for sym in symbols: + df = request_with_retry( + pro.fina_indicator, + max_retries=self._max_retries, + rate_limit=self._rate_limit, + ts_code=sym, + start_date=s, + end_date=e, + fields=col_spec, + ) + if df is not None and len(df) > 0: + frames.append(df) + + if not frames: + return self._empty(wanted) + + out = pd.concat(frames, ignore_index=True).rename(columns={"ts_code": "symbol"}) + out["symbol"] = out["symbol"].astype(str) + keep = ["symbol", "ann_date", "end_date", *wanted] + return out[[c for c in keep if c in out.columns]] + + @staticmethod + def _empty(fields: list[str]) -> pd.DataFrame: + cols = {"symbol": pd.Series([], dtype=object), + "ann_date": pd.Series([], dtype=object), + "end_date": pd.Series([], dtype=object)} + for f in fields: + cols[f] = pd.Series([], dtype=float) + return pd.DataFrame(cols) diff --git a/data/feed/tushare_flags.py b/data/feed/tushare_flags.py new file mode 100644 index 0000000..b0189d4 --- /dev/null +++ b/data/feed/tushare_flags.py @@ -0,0 +1,119 @@ +"""TushareFlagsFeed: tradability signals (suspension, ST, price limits). + +Pulls the three tushare sources that drive the tradability filters and maps them +onto shapes the :mod:`data.clean.tradability` enrichment can join to the panel: + + * ``suspended(symbols, start, end)`` -> set of (date, symbol) suspended that day + (tushare ``suspend_d``, suspend_type 'S'). + * ``st_intervals(symbols)`` -> {symbol: [(start, end|None, is_st)]} + (tushare ``namechange``; a name containing 'ST' / '*ST' marks the interval). + * ``limits(symbols, start, end)`` -> DataFrame[date, symbol, up_limit, down_limit] + (tushare ``stk_limit``). + +Token is read from the external config and never printed/logged. The client is +built lazily, so constructing the feed needs no network/credentials. +""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.secret import read_token +from data.feed.throttle import request_with_retry + + +class TushareFlagsFeed: + """Loads suspension / ST / price-limit signals from tushare.""" + + def __init__( + self, + secret_file: str, + token_key: str = "tushare.token", + rate_limit: int | None = None, + max_retries: int = 3, + ) -> None: + self._secret_file = str(secret_file) + self._token_key = token_key + self._rate_limit = rate_limit + self._max_retries = max(1, int(max_retries)) + self._pro = None + + def _client(self): + if self._pro is None: + import tushare as ts + + self._pro = ts.pro_api(read_token(self._secret_file, self._token_key)) + return self._pro + + def _call(self, fn, **kwargs): + return request_with_retry( + fn, max_retries=self._max_retries, rate_limit=self._rate_limit, **kwargs + ) + + # -- suspensions (停牌) -------------------------------------------------- # + def suspended(self, symbols: list[str], start: str, end: str) -> set[tuple]: + """Return the set of (Timestamp date, symbol) suspended over [start, end].""" + pro = self._client() + s = pd.Timestamp(start).strftime("%Y%m%d") + e = pd.Timestamp(end).strftime("%Y%m%d") + out: set[tuple] = set() + for sym in symbols: + df = self._call( + pro.suspend_d, ts_code=sym, start_date=s, end_date=e, suspend_type="S" + ) + if df is None or len(df) == 0: + continue + for d in df["trade_date"].astype(str): + out.add((pd.to_datetime(d, format="%Y%m%d"), str(sym))) + return out + + # -- ST status (namechange) -------------------------------------------- # + def st_intervals(self, symbols: list[str]) -> dict[str, list[tuple]]: + """Return {symbol: [(start_ts, end_ts|None, is_st_bool), ...]} from names.""" + pro = self._client() + result: dict[str, list[tuple]] = {} + for sym in symbols: + df = self._call(pro.namechange, ts_code=sym) + if df is None or len(df) == 0: + continue + seen: set[tuple] = set() + intervals: list[tuple] = [] + for _, row in df.iterrows(): + start = pd.to_datetime(str(row["start_date"]), format="%Y%m%d") + end_raw = row["end_date"] + end = ( + None + if end_raw is None or pd.isna(end_raw) + else pd.to_datetime(str(end_raw), format="%Y%m%d") + ) + name = str(row["name"]) + key = (start, end, name) + if key in seen: + continue + seen.add(key) + intervals.append((start, end, "ST" in name.upper())) + result[str(sym)] = intervals + return result + + # -- price limits (涨跌停) ---------------------------------------------- # + def limits(self, symbols: list[str], start: str, end: str) -> pd.DataFrame: + """Return DataFrame[date, symbol, up_limit, down_limit] over [start, end].""" + pro = self._client() + s = pd.Timestamp(start).strftime("%Y%m%d") + e = pd.Timestamp(end).strftime("%Y%m%d") + frames: list[pd.DataFrame] = [] + for sym in symbols: + df = self._call(pro.stk_limit, ts_code=sym, start_date=s, end_date=e) + if df is not None and len(df) > 0: + frames.append(df) + if not frames: + return pd.DataFrame( + {"date": pd.Series([], dtype="datetime64[ns]"), + "symbol": pd.Series([], dtype=object), + "up_limit": pd.Series([], dtype=float), + "down_limit": pd.Series([], dtype=float)} + ) + df = pd.concat(frames, ignore_index=True).rename(columns={"ts_code": "symbol"}) + df["date"] = pd.to_datetime(df["trade_date"].astype(str), format="%Y%m%d") + df["symbol"] = df["symbol"].astype(str) + return df[["date", "symbol", "up_limit", "down_limit"]] diff --git a/data/store/panel_store.py b/data/store/panel_store.py new file mode 100644 index 0000000..7418bd5 --- /dev/null +++ b/data/store/panel_store.py @@ -0,0 +1,148 @@ +"""Parquet-backed store for canonical (date, symbol) panels (DATA-008/009/010). + +A :class:`PanelStore` persists a normalized panel to a single parquet file and +reads it back, optionally filtering by a **closed** date interval ``[start, end]`` +and/or a symbol subset. The round-trip is loss-free with respect to the panel +contract in :mod:`data.clean.schema`: the MultiIndex(date, symbol), column set and +order, sort order, dtypes, and values (NaN cells included) all survive. + +Design notes +------------ +- P0 layout: one file per logical panel name -> ``/.parquet``. +- Parquet cannot store a MultiIndex directly, so we ``reset_index`` before + writing and ``set_index`` + re-normalize after reading. Re-normalizing through + :func:`normalize_panel` guarantees the read panel obeys the same contract as a + freshly built one (sorted, datetime date level, str symbols). +- The store never mutates its inputs and never returns a view onto internal + state — every read produces a fresh frame. +- This layer is pure storage: it does not fetch data, compute factors, or touch + forward returns. +""" + +from __future__ import annotations + +from pathlib import Path + +import pandas as pd + +from data.clean.schema import ( + DATE_LEVEL, + INDEX_NAMES, + SYMBOL_LEVEL, + normalize_panel, + validate_panel, +) + + +class PanelStore: + """Persist and load canonical panels as parquet files under ``root``.""" + + def __init__(self, root: str) -> None: + """Create a store rooted at ``root`` (the base directory for parquet files). + + The directory is created lazily on the first :meth:`write`; constructing a + store never touches the filesystem, so it is safe in dry runs. + """ + self._root = Path(root) + + def path_for(self, name: str) -> Path: + """Return the parquet path for a logical panel ``name`` (``/.parquet``).""" + if not name or "/" in name or "\\" in name: + raise ValueError( + f"Panel name must be a simple non-empty token without path separators; got {name!r}." + ) + return self._root / f"{name}.parquet" + + def write(self, name: str, panel: pd.DataFrame, overwrite: bool = True) -> None: + """Persist ``panel`` (a canonical (date, symbol) panel) to parquet. + + ``panel`` is validated against the schema contract first, so a malformed + frame fails fast with a readable error instead of writing junk. The + MultiIndex is flattened to ``date``/``symbol`` columns for storage. + + ``overwrite`` mirrors ``output.overwrite``: when ``False`` and the target + file already exists, raise rather than clobber it. + """ + validate_panel(panel) + + target = self.path_for(name) + if target.exists() and not overwrite: + raise ValueError( + f"A stored panel '{name}' already exists at {target} and overwrite=False. " + "Pass overwrite=True to replace it, or choose a different name." + ) + + target.parent.mkdir(parents=True, exist_ok=True) + + # Flatten the MultiIndex into columns so parquet can round-trip it. + flat = panel.reset_index() + # Atomic-ish replace: write to a temp file then move into place, so a + # crash mid-write never leaves a half-written panel at the real path. + tmp = target.with_suffix(".parquet.tmp") + flat.to_parquet(tmp, engine="pyarrow", index=False) + tmp.replace(target) + + def read( + self, + name: str, + start: str | pd.Timestamp | None = None, + end: str | pd.Timestamp | None = None, + symbols: list[str] | None = None, + ) -> pd.DataFrame: + """Load the panel stored under ``name`` and return a normalized panel. + + Optional filters (applied before re-normalizing): + - ``start`` / ``end``: keep only rows whose date falls in the **closed** + interval ``[start, end]``. Either bound may be a ``"YYYY-MM-DD"`` string + or a ``pd.Timestamp``; either may be omitted to leave that side open. + - ``symbols``: keep only rows whose symbol is in this subset. + + The result is always a fresh, schema-valid panel (sorted, MultiIndex, + correct dtypes). Filtering to an empty selection is allowed and yields an + empty panel with the correct columns/index names. + """ + target = self.path_for(name) + if not target.exists(): + raise FileNotFoundError( + f"No stored panel '{name}' at {target}. Write it first with PanelStore.write()." + ) + + flat = pd.read_parquet(target, engine="pyarrow") + flat = self._apply_date_filter(flat, start, end) + flat = self._apply_symbol_filter(flat, symbols) + + # Rebuild the canonical shape. normalize_panel re-sorts, re-types the date + # level to datetime and the symbol level to str, and revalidates uniqueness. + return normalize_panel(flat) + + @staticmethod + def _apply_date_filter( + flat: pd.DataFrame, + start: str | pd.Timestamp | None, + end: str | pd.Timestamp | None, + ) -> pd.DataFrame: + """Return rows of ``flat`` whose ``date`` is in the closed [start, end] interval.""" + if start is None and end is None: + return flat + date_col = pd.to_datetime(flat[DATE_LEVEL]).dt.normalize() + mask = pd.Series(True, index=flat.index) + if start is not None: + mask &= date_col >= pd.Timestamp(start).normalize() + if end is not None: + mask &= date_col <= pd.Timestamp(end).normalize() + return flat.loc[mask] + + @staticmethod + def _apply_symbol_filter( + flat: pd.DataFrame, + symbols: list[str] | None, + ) -> pd.DataFrame: + """Return rows of ``flat`` whose ``symbol`` is in ``symbols`` (no-op if None).""" + if symbols is None: + return flat + wanted = {str(s) for s in symbols} + mask = flat[SYMBOL_LEVEL].astype(str).isin(wanted) + return flat.loc[mask] + + +__all__ = ["PanelStore", "INDEX_NAMES"] diff --git a/factors/base.py b/factors/base.py new file mode 100644 index 0000000..17533de --- /dev/null +++ b/factors/base.py @@ -0,0 +1,40 @@ +"""Factor port: compute a cross-sectional feature from market bars. + +NO-LOOKAHEAD RULE (CLAUDE.md invariant #1, INV-001): + A factor value at date t may use ONLY bars at dates <= t. It must never + read future prices and must never receive forward returns. Per the fixed + event order (see CONTRACTS.md), ``momentum_20[t] = close[t]/close[t-20] - 1`` + is acceptable because rebalancing happens AFTER the close of t and holding + starts the next trading day. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class Factor(ABC): + """Abstract cross-sectional factor. + + Subclasses set the class attribute ``name`` (used as the factor-panel column) + and implement ``compute``. + """ + + name: str + + @abstractmethod + def compute(self, panel: pd.DataFrame) -> pd.Series: + """Compute the factor over a canonical market panel. + + Args: + panel: MultiIndex(date, symbol) market panel with CORE_COLUMNS. + + Returns: + A pd.Series indexed by MultiIndex(date, symbol), aligned to ``panel``, + with ``.name == self.name``. Early dates with an insufficient window + yield NaN. Computation must be per-symbol (no cross-symbol leakage) + and must use only current/past bars (no lookahead). + """ + raise NotImplementedError diff --git a/factors/compute/financial.py b/factors/compute/financial.py new file mode 100644 index 0000000..f24f846 --- /dev/null +++ b/factors/compute/financial.py @@ -0,0 +1,40 @@ +"""FinancialFactor: a cross-sectional factor from a PIT-aligned financial column. + +The financial value on the panel is ALREADY point-in-time aligned by disclosure +date (:func:`data.clean.pit_financials.asof_financials` placed it there using +``ann_date <= trade_date``), so this factor does no temporal logic — it simply +surfaces that column as the factor series. The no-look-ahead guarantee lives in +the as-of alignment, upstream; the factor layer still never sees future data. +""" + +from __future__ import annotations + +import pandas as pd + +from factors.base import Factor + +# financial fields that may be requested as a factor (P1). +SUPPORTED_FIELDS: tuple[str, ...] = ("roe", "netprofit_yoy") + + +class FinancialFactor(Factor): + """Surface a PIT-aligned financial column (e.g. ``roe``) as a factor.""" + + def __init__(self, field: str = "roe") -> None: + if field not in SUPPORTED_FIELDS: + raise ValueError( + f"FinancialFactor field {field!r} not supported; " + f"choose one of {SUPPORTED_FIELDS}." + ) + self.name = field + self._field = field + + def compute(self, panel: pd.DataFrame) -> pd.Series: + """Return the as-of financial column as a MultiIndex(date, symbol) series.""" + if self._field not in panel.columns: + raise ValueError( + f"FinancialFactor('{self._field}') needs an as-of '{self._field}' " + f"column on the panel. Financial factors require the tushare data " + f"path (ann_date alignment); they are not available for demo data." + ) + return panel[self._field].rename(self.name) diff --git a/factors/compute/momentum.py b/factors/compute/momentum.py new file mode 100644 index 0000000..9d074eb --- /dev/null +++ b/factors/compute/momentum.py @@ -0,0 +1,84 @@ +"""The ``momentum_20`` cross-sectional factor (FAC-001..004). + +Definition (FIXED in CONTRACTS.md s6, fixed event order): + + momentum_20[t] = close[t] / close[t - window] - 1 # window default = 20 + + compute factor at the CLOSE of date t + rebalance AFTER the close of date t + hold from the NEXT trading day + +Using ``close[t]`` is NOT lookahead: the position formed from the factor at the +close of t is only held starting t+1, so it earns the t+1 forward return. A value +at date t therefore depends only on bars at dates <= t (INV-001 / CLAUDE.md +invariant #1). Early dates without a full ``window`` of history yield NaN. + +Computation is strictly per-symbol (grouped on the ``symbol`` index level) so one +symbol's prices can never leak into another's factor value. +""" + +from __future__ import annotations + +import pandas as pd + +from factors.base import Factor + + +class MomentumFactor(Factor): + """Trailing price-momentum over a fixed lookback window. + + Args: + window: Number of trading bars between the numerator and denominator + close. Default 20 (the canonical ``momentum_20``). + price_col: Panel column to use as the price. Default ``"close"``. + """ + + name: str = "momentum_20" + + def __init__(self, window: int = 20, price_col: str = "close") -> None: + if not isinstance(window, int) or window < 1: + raise ValueError( + f"momentum window must be a positive integer, got {window!r}." + ) + self._window = window + self._price_col = price_col + # Instance name tracks the actual window so a non-default window does NOT + # mislabel the factor column (the class attr stays the canonical default). + self.name = f"momentum_{window}" + + @property + def window(self) -> int: + return self._window + + @property + def price_col(self) -> str: + return self._price_col + + def compute(self, panel: pd.DataFrame) -> pd.Series: + """Compute per-symbol momentum aligned to the panel index. + + Returns a MultiIndex(date, symbol) Series named ``momentum_20``. The + input is never mutated. + """ + if self._price_col not in panel.columns: + raise ValueError( + f"momentum factor needs a '{self._price_col}' column; panel has " + f"{list(panel.columns)}." + ) + if not isinstance(panel.index, pd.MultiIndex) or list( + panel.index.names + ) != ["date", "symbol"]: + raise ValueError( + "momentum factor expects a MultiIndex(date, symbol) panel; got " + f"index names {list(panel.index.names)}." + ) + + price = panel[self._price_col] + # Group on the symbol index level so the ratio never crosses symbols. + # ``shift(window)`` looks strictly backward within each symbol, so the + # value at t uses only bars at t and t-window (both <= t): no lookahead. + prev = price.groupby(level="symbol").shift(self._window) + momentum = price / prev - 1.0 + + # Preserve the exact panel index/order; name it for the factor panel. + return momentum.reindex(panel.index).rename(self.name) diff --git a/factors/process/base.py b/factors/process/base.py new file mode 100644 index 0000000..3af7ff6 --- /dev/null +++ b/factors/process/base.py @@ -0,0 +1,31 @@ +"""FactorProcessor port: cross-sectional pre-processing of a factor panel. + +Processing is always *by date* (each cross-section independently): drop missing, +z-score standardize (P0); winsorize, neutralize (P1). Output keeps the canonical +MultiIndex(date, symbol) shape. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class FactorProcessor(ABC): + """Abstract cross-sectional factor processor.""" + + @abstractmethod + def transform(self, factors: pd.DataFrame) -> pd.DataFrame: + """Transform a factor panel cross-sectionally (per date). + + Args: + factors: MultiIndex(date, symbol) frame; columns are factor names. + + Returns: + A new MultiIndex(date, symbol) frame, same columns, each date's + cross-section processed independently. A single-name or zero-variance + cross-section must not raise (return NaN or 0 by documented rule). + The input is not mutated. + """ + raise NotImplementedError diff --git a/factors/process/neutralize.py b/factors/process/neutralize.py new file mode 100644 index 0000000..a5c9763 --- /dev/null +++ b/factors/process/neutralize.py @@ -0,0 +1,64 @@ +"""Cross-sectional industry + size neutralization (per-date OLS residual). + +For each date's cross-section, regress the factor on ``[log(market_cap), +one-hot(industry)]`` and keep the residual — the part of the factor NOT explained +by size or industry. This removes the (usually unwanted) systematic exposure of a +raw factor to firm size and sector, which otherwise dominates cross-sectional +ranks. + +Degenerate cross-sections (fewer than 3 valid names) and rows with a missing +factor / industry / non-positive market cap return NaN rather than a fabricated +value. The regression is solved with ``numpy.linalg.lstsq`` so collinear industry +dummies (which always sum to 1) are handled without raising. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +_MIN_NAMES = 3 # need at least this many valid names to fit a cross-section + + +def neutralize_by_date( + factor: pd.Series, industry: pd.Series, market_cap: pd.Series +) -> pd.Series: + """Return the per-date residual of ``factor`` on industry dummies + log size.""" + if not isinstance(factor.index, pd.MultiIndex): + raise ValueError( + "neutralize_by_date expects a MultiIndex(date, symbol) factor series." + ) + out = pd.Series(np.nan, index=factor.index, dtype=float) + for _, idx in factor.groupby(level="date").groups.items(): + out.loc[idx] = _residual_one_date(factor.loc[idx], industry, market_cap) + return out + + +def _residual_one_date( + y: pd.Series, industry: pd.Series, market_cap: pd.Series +) -> pd.Series: + """OLS residual for a single date's cross-section (index-aligned to ``y``).""" + df = pd.DataFrame( + { + "y": y, + "ind": industry.reindex(y.index), + "mc": market_cap.reindex(y.index), + } + ) + df["lmc"] = np.where(df["mc"] > 0, np.log(df["mc"]), np.nan) + valid = df.dropna(subset=["y", "ind", "lmc"]) + + out = pd.Series(np.nan, index=y.index, dtype=float) + # Need residual degrees of freedom > 0, else the fit is saturated and the + # "residuals" are fabricated ~0 (a perfect in-sample fit), not a real neutral + # factor. n_params = log(market cap) + one-hot(industry); require n > n_params. + n_params = 1 + valid["ind"].nunique() + if len(valid) < _MIN_NAMES or len(valid) <= n_params: + return out # too few names / no residual DOF -> NaN (no fabrication) + + dummies = pd.get_dummies(valid["ind"], drop_first=False).astype(float) + design = np.column_stack([valid["lmc"].to_numpy(dtype=float), dummies.to_numpy()]) + target = valid["y"].to_numpy(dtype=float) + beta, *_ = np.linalg.lstsq(design, target, rcond=None) + out.loc[valid.index] = target - design @ beta + return out diff --git a/factors/process/pipeline.py b/factors/process/pipeline.py new file mode 100644 index 0000000..93002c1 --- /dev/null +++ b/factors/process/pipeline.py @@ -0,0 +1,131 @@ +"""Cross-sectional factor preprocessing pipeline (Slice 6, PROC-001..004). + +``ProcessingPipeline`` is a concrete :class:`FactorProcessor`. It processes a +factor panel *by date* (each cross-section independently). P0 steps: + + drop_missing -> per-date, drop rows whose factor value is NaN + standardize -> per-date z-score (mean ~0, std ~1) + +Z-score uses the population standard deviation (``ddof=0``). A zero-variance or +single-name cross-section has no spread to scale by, so the documented rule is +that it standardizes to ``0.0`` (de-meaned, no scaling) rather than producing +NaN/inf -- it never raises. + +P1 steps (winsorize, neutralize) are present as optional no-op hooks so the +config surface (``ProcessingCfg``) is honoured, but they are intentionally not +implemented in P0. + +Design notes: + * Pure / immutable: ``transform`` never mutates its input; it returns a new + frame with the canonical ``MultiIndex(date, symbol)`` preserved. + * Decisions are cross-sectional: every column is standardized within each + date's cross-section, independent of other dates (no time-series leakage). +""" + +from __future__ import annotations + +import pandas as pd + +from factors.process.base import FactorProcessor +from factors.process.neutralize import neutralize_by_date + + +class ProcessingPipeline(FactorProcessor): + """Configurable by-date factor processor (drop_missing/winsorize/neutralize/zscore).""" + + def __init__( + self, + *, + drop_missing: bool = True, + standardize: bool = True, + winsorize: bool = False, + neutralize: bool = False, + industry: pd.Series | None = None, + market_cap: pd.Series | None = None, + ) -> None: + # Step toggles. winsorize stays a P1 no-op; neutralize is implemented (P1) + # and needs the industry + market_cap cross-sectional covariates. + self._drop_missing = drop_missing + self._standardize = standardize + self._winsorize = winsorize + self._neutralize = neutralize + self._industry = industry + self._market_cap = market_cap + + def transform(self, factors: pd.DataFrame) -> pd.DataFrame: + """Process every column cross-sectionally, per date. + + Args: + factors: ``MultiIndex(date, symbol)`` frame; columns are factor names. + + Returns: + A new ``MultiIndex(date, symbol)`` frame, same columns, each date's + cross-section processed independently. Never mutates the input. + """ + if not isinstance(factors.index, pd.MultiIndex): + raise ValueError( + "ProcessingPipeline.transform expects a MultiIndex(date, symbol) " + f"factor panel; got index type {type(factors.index).__name__}." + ) + if factors.empty: + return factors.copy() + + date_level = factors.index.names[0] + # Operate on a copy so the caller's frame is never mutated. + out = factors.copy() + + if self._drop_missing: + out = self._apply_drop_missing(out) + if self._winsorize: + out = self._apply_winsorize(out, date_level) + if self._neutralize: + out = self._apply_neutralize(out) + if self._standardize: + out = self._apply_zscore(out, date_level) + + return out.sort_index() + + @staticmethod + def _apply_drop_missing(frame: pd.DataFrame) -> pd.DataFrame: + """Drop rows with any NaN factor value (per-date is implicit per-row).""" + return frame.dropna(axis=0, how="any") + + @staticmethod + def _apply_zscore(frame: pd.DataFrame, date_level: str) -> pd.DataFrame: + """Z-score each column within each date's cross-section (ddof=0). + + Zero-variance / single-name cross-sections -> 0.0 (no spread to scale). + """ + grouped = frame.groupby(level=date_level) + # Per-date mean and population std, broadcast back to every row via + # ``transform`` (keeps the full MultiIndex aligned, no manual join). + mean = grouped.transform("mean") + std = grouped.transform("std", ddof=0) + demeaned = frame - mean + # Where std == 0 (constant column or single name), keep the de-meaned + # value (which is 0) instead of dividing -> avoids NaN/inf. + scaled = demeaned.divide(std).where(std != 0, other=demeaned) + return scaled[list(frame.columns)] + + @staticmethod + def _apply_winsorize(frame: pd.DataFrame, date_level: str) -> pd.DataFrame: + """P1 hook: clip extremes per date. No-op in P0 (returns unchanged).""" + return frame + + def _apply_neutralize(self, frame: pd.DataFrame) -> pd.DataFrame: + """Per-date industry + size residual for every factor column (P1). + + Requires the ``industry`` and ``market_cap`` covariates (set at + construction). If neutralization is enabled but either is missing, raise a + readable error rather than silently skipping or fabricating values. + """ + if self._industry is None or self._market_cap is None: + raise ValueError( + "processing.neutralize is enabled but industry/market_cap are not " + "available. They require the tushare data path (industry + market " + "cap enrichment); demo data has neither, so neutralization cannot run." + ) + out = frame.copy() + for column in out.columns: + out[column] = neutralize_by_date(out[column], self._industry, self._market_cap) + return out diff --git a/portfolio/base.py b/portfolio/base.py new file mode 100644 index 0000000..10cf112 --- /dev/null +++ b/portfolio/base.py @@ -0,0 +1,37 @@ +"""PortfolioConstructor port: turn scores into target weights. + +Hard boundary (CLAUDE.md invariant #3, INV-002): the portfolio layer receives +only scores / current weights / constraints. It must NOT touch a data source or +place orders. P0 builds a long-only TopN equal-weight portfolio. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class PortfolioConstructor(ABC): + """Abstract portfolio constructor.""" + + @abstractmethod + def build( + self, + scores: pd.Series, + current_weights: pd.Series | None = None, + ) -> pd.Series: + """Build target weights from alpha scores. + + Args: + scores: symbol-indexed alpha scores for one cross-section. NaN scores + are ignored (PF-003). + current_weights: optional symbol-indexed current weights (for + turnover-aware constructors, P1). P0 may ignore it. + + Returns: + A symbol-indexed pd.Series of target weights, summing to ~1.0 when + any candidate exists (PF-002), long-only (no negative weights, PF-009), + empty when there are no candidates (PF-004). The input is not mutated. + """ + raise NotImplementedError diff --git a/portfolio/construct.py b/portfolio/construct.py new file mode 100644 index 0000000..c6bd8de --- /dev/null +++ b/portfolio/construct.py @@ -0,0 +1,71 @@ +"""Portfolio constructors: turn alpha scores into target weights. + +P0 ships ``TopNEqualWeight`` — long-only, equal-weight the N highest-scoring +symbols. Hard boundary (CLAUDE.md invariant #3, INV-002): this layer receives +only scores / current weights / constraints; it never touches a data source or +places orders. The input Series is never mutated (immutable style). +""" + +from __future__ import annotations + +import pandas as pd + +from portfolio.base import PortfolioConstructor + + +class TopNEqualWeight(PortfolioConstructor): + """Equal-weight the ``top_n`` highest-scoring symbols (long-only, P0). + + Behaviour: + - drop NaN scores (PF-003); + - select the ``top_n`` highest remaining scores (PF-001); + - assign equal weight ``1/k`` so weights sum to 1 (PF-002), where ``k`` + is the number actually selected; + - if fewer candidates than ``top_n`` exist, equal-weight the actual + count (still sums to 1, PF-005); + - if no candidates exist, return an EMPTY Series (no crash, PF-004); + - long-only: never emit a negative weight (PF-009). + """ + + def __init__(self, top_n: int, long_only: bool = True) -> None: + if not isinstance(top_n, int) or isinstance(top_n, bool): + raise ValueError(f"top_n must be an int, got {type(top_n).__name__}") + if top_n <= 0: + raise ValueError(f"top_n must be a positive integer, got {top_n}") + self.top_n = top_n + self.long_only = long_only + + def build( + self, + scores: pd.Series, + current_weights: pd.Series | None = None, + ) -> pd.Series: + """Build long-only equal-weight target weights from alpha scores. + + Args: + scores: symbol-indexed alpha scores for one cross-section. NaN scores + are ignored. + current_weights: ignored in P0 (reserved for turnover-aware P1 + constructors); kept for interface compatibility. + + Returns: + A symbol-indexed pd.Series of target weights summing to ~1.0 when any + candidate exists, empty when there are none. The input is not mutated. + """ + # Drop NaN scores without mutating the input (PF-003). + candidates = scores.dropna() + + if candidates.empty: + # No candidates -> empty weights (PF-004). Preserve index name/dtype. + return pd.Series(dtype=float, index=candidates.index[:0], name="weight") + + # Select the top_n highest scores. ``nlargest`` clamps to the available + # count, so fewer-than-N candidates are handled naturally (PF-005). It is + # deterministic w.r.t. ties (keeps first occurrence) for reproducibility. + selected = candidates.nlargest(self.top_n) + + k = len(selected) + weight = 1.0 / k # equal weight; sums to exactly 1 for k positions (PF-002) + result = pd.Series(weight, index=selected.index, name="weight") + result.index.name = scores.index.name + return result diff --git a/portfolio/risk.py b/portfolio/risk.py new file mode 100644 index 0000000..a46d55b --- /dev/null +++ b/portfolio/risk.py @@ -0,0 +1,48 @@ +"""Thin P0 risk helpers for portfolio weights. + +These are minimal, pure functions consumed later by P1 constructors (max-weight +cap, turnover cap, industry constraints). Each returns a NEW Series and never +mutates its input (immutable style). Keep this module small. +""" + +from __future__ import annotations + +import pandas as pd + + +def enforce_long_only(weights: pd.Series) -> pd.Series: + """Return a copy of ``weights`` with all negative entries clipped to 0. + + P0 portfolios are long-only (PF-009). This does NOT renormalize; callers that + need weights to sum to 1 should renormalize afterwards. + """ + return weights.clip(lower=0.0) + + +def cap_weight(weights: pd.Series, max_weight: float) -> pd.Series: + """Cap each weight at ``max_weight`` and renormalize to sum to 1 (PF-006). + + Args: + weights: symbol-indexed weights (assumed non-negative). + max_weight: per-name upper bound, in (0, 1]. + + Returns: + A NEW symbol-indexed Series whose entries are <= ``max_weight`` and that + sums to ~1.0 when the input had positive mass. Returns the input + unchanged (as a copy) when it is empty or sums to 0. + + Notes: + A single capping + renormalize pass can push previously-uncapped names + back over the cap. This thin P0 helper does a single pass; the P1 + constructor that adopts it will iterate to a feasible solution. + """ + if max_weight <= 0.0: + raise ValueError(f"max_weight must be positive, got {max_weight}") + if weights.empty: + return weights.copy() + + capped = weights.clip(upper=max_weight) + total = capped.sum() + if total <= 0.0: + return capped + return capped / total diff --git a/pyproject.toml b/pyproject.toml new file mode 100644 index 0000000..da3dfd6 --- /dev/null +++ b/pyproject.toml @@ -0,0 +1,63 @@ +[build-system] +requires = ["setuptools>=68"] +build-backend = "setuptools.build_meta" + +[project] +name = "quantitative-trading" +version = "0.1.0" +description = "A-share cross-sectional multi-factor quant framework (Phase 0 MVP)" +requires-python = ">=3.12" +dependencies = [ + "pandas>=2.0", + "numpy", + "pyarrow", + "duckdb", + "pydantic>=2", + "pydantic-settings", + "pyyaml", + "tushare", + "statsmodels", + "scikit-learn", +] + +[project.optional-dependencies] +analytics = [ + "alphalens-reloaded", + "quantstats", + "matplotlib", +] +dev = [ + "pytest", + "pytest-cov", + "ruff", +] + +[project.scripts] +qt = "qt.cli:main" + +[tool.setuptools.packages.find] +include = [ + "data*", + "universe*", + "factors*", + "alpha*", + "portfolio*", + "runtime*", + "analytics*", + "qt*", +] +exclude = ["tests*"] + +[tool.pytest.ini_options] +testpaths = ["tests"] +addopts = "-q" + +[tool.ruff] +line-length = 100 +target-version = "py312" + +[tool.ruff.lint] +# E501 line-too-long is handled by formatter; keep a pragmatic ignore set. +ignore = [ + "E741", # ambiguous variable name (l/I/O); common in math code +] diff --git a/qt/__init__.py b/qt/__init__.py new file mode 100644 index 0000000..5385dfe --- /dev/null +++ b/qt/__init__.py @@ -0,0 +1,13 @@ +"""qt: command-line entry + config models for the Phase 0 framework. + +The CLI lives here (not in any business layer) so that ``data/``, ``factors/``, +``alpha/``, ``portfolio/`` and ``runtime/`` stay free of process orchestration. +""" + +from __future__ import annotations + +__version__ = "0.1.0" + +from qt.config import RootConfig, load_config + +__all__ = ["RootConfig", "load_config", "__version__"] diff --git a/qt/cli.py b/qt/cli.py new file mode 100644 index 0000000..a8ddbc2 --- /dev/null +++ b/qt/cli.py @@ -0,0 +1,123 @@ +"""Command-line entry point for the Phase 0 framework. + +Subcommands: + validate-config --config PATH Load + validate a YAML config. + run-phase0 --config PATH Run the full end-to-end pipeline + report. + fetch-data --config PATH Stage helper (runs the pipeline; see note). + compute-factors --config PATH Stage helper (runs the pipeline; see note). + run-backtest --config PATH Stage helper (runs the pipeline; see note). + +The CLI is intentionally thin: orchestration lives in :mod:`qt.pipeline`. Errors +are reported as readable one-line messages (CLI-003), never raw tracebacks. Run +as ``python -m qt.cli --config ``. + +Note on stage helpers: P0 keeps a single reproducible spine. The fetch-data / +compute-factors / run-backtest sub-commands run that spine and report the stage +of interest, rather than persisting partial cross-process state. ``run-phase0`` +is the canonical end-to-end command (CLI-002). +""" + +from __future__ import annotations + +import argparse +import sys + +from qt.config import ConfigError, load_config + + +def _cmd_validate_config(args: argparse.Namespace) -> int: + """Load and validate the config; print OK / error and return an exit code.""" + try: + load_config(args.config) + except ConfigError as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + print("OK") + return 0 + + +def _run_pipeline_cmd(config: str, stage: str) -> int: + """Shared runner for run-phase0 and the stage helper sub-commands.""" + # Imported lazily so ``validate-config`` stays light and import errors in a + # heavy slice never break config validation. + from qt.pipeline import run_phase0 + + try: + result = run_phase0(config) + except (ConfigError, ValueError, FileNotFoundError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + + if stage == "fetch-data": + print(f"OK fetch-data: {result.panel_rows} rows -> {result.data_path}") + elif stage == "compute-factors": + print(f"OK compute-factors: {result.factor_name} -> {result.factor_path}") + elif stage == "run-backtest": + annual = result.performance.get("annual_return", float("nan")) + print(f"OK run-backtest: annual_return={annual:.4f}, nav rows={len(result.nav_table)}") + else: + print( + f"OK run-phase0: ic_mean={result.ic_mean:.4f}, " + f"annual_return={result.performance.get('annual_return', float('nan')):.4f}\n" + f"report: {result.report_path}" + ) + return 0 + + +def _cmd_run_phase0(args: argparse.Namespace) -> int: + """Run the full end-to-end pipeline and write the phase0 report (CLI-002).""" + return _run_pipeline_cmd(args.config, "run-phase0") + + +def _cmd_fetch_data(args: argparse.Namespace) -> int: + """Stage helper: run the spine and report the data-fetch stage.""" + return _run_pipeline_cmd(args.config, "fetch-data") + + +def _cmd_compute_factors(args: argparse.Namespace) -> int: + """Stage helper: run the spine and report the factor-compute stage.""" + return _run_pipeline_cmd(args.config, "compute-factors") + + +def _cmd_run_backtest(args: argparse.Namespace) -> int: + """Stage helper: run the spine and report the backtest stage.""" + return _run_pipeline_cmd(args.config, "run-backtest") + + +def build_parser() -> argparse.ArgumentParser: + """Construct the argparse parser with all subcommands.""" + parser = argparse.ArgumentParser( + prog="qt", + description="A-share cross-sectional multi-factor framework (Phase 0).", + ) + sub = parser.add_subparsers(dest="command", required=True) + + p_validate = sub.add_parser("validate-config", help="Validate a YAML config file.") + p_validate.add_argument("--config", required=True, help="Path to the YAML config.") + p_validate.set_defaults(func=_cmd_validate_config) + + p_run = sub.add_parser("run-phase0", help="Run the end-to-end Phase 0 pipeline.") + p_run.add_argument("--config", required=True, help="Path to the YAML config.") + p_run.set_defaults(func=_cmd_run_phase0) + + for name, func, help_text in ( + ("fetch-data", _cmd_fetch_data, "Run the spine, report data fetch."), + ("compute-factors", _cmd_compute_factors, "Run the spine, report factor compute."), + ("run-backtest", _cmd_run_backtest, "Run the spine, report backtest."), + ): + p = sub.add_parser(name, help=help_text) + p.add_argument("--config", required=True, help="Path to the YAML config.") + p.set_defaults(func=func) + + return parser + + +def main(argv: list[str] | None = None) -> int: + """Parse args and dispatch. Returns a process exit code.""" + parser = build_parser() + args = parser.parse_args(argv) + return args.func(args) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/qt/config.py b/qt/config.py new file mode 100644 index 0000000..2fd1bac --- /dev/null +++ b/qt/config.py @@ -0,0 +1,257 @@ +"""Pydantic v2 config models for the Phase 0 framework. + +These mirror ``config/example.yaml`` (a.k.a. example_config_v1.yaml) exactly. +``load_config`` reads the YAML, validates it, and turns any pydantic validation +error into a user-readable message (CLI-003) — non-CS users must understand +what is wrong without reading a raw traceback. + +Design note: this is the single source of truth for config field names. +Downstream agents read fields off ``RootConfig`` and its sub-models; they do not +re-parse the YAML. +""" + +from __future__ import annotations + +from datetime import date as _date +from datetime import datetime +from pathlib import Path +from typing import Any, Literal + +import yaml +from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator, model_validator + + +class _Strict(BaseModel): + """Base model: forbid unknown keys so config typos surface early.""" + + model_config = ConfigDict(extra="forbid") + + +# --------------------------------------------------------------------------- # +# Sub-models (mirror the YAML block-by-block) +# --------------------------------------------------------------------------- # +class ProjectCfg(_Strict): + name: str + timezone: str = "Asia/Shanghai" + + +class DataCfg(_Strict): + source: Literal["demo", "tushare"] = "demo" + freq: str = "D" + start: str + end: str + external_secret_file: str | None = None + tushare_token_key: str = "tushare.token" + output_name: str = "daily" + + @field_validator("start", "end", mode="before") + @classmethod + def _coerce_date_to_str(cls, v: Any) -> Any: + # YAML may parse unquoted dates as date objects; keep them as ISO strings. + if isinstance(v, (_date, datetime)): + return v.strftime("%Y-%m-%d") + return v + + @model_validator(mode="after") + def _check_date_order(self) -> "DataCfg": + try: + start = datetime.strptime(self.start, "%Y-%m-%d") + end = datetime.strptime(self.end, "%Y-%m-%d") + except ValueError as exc: # pragma: no cover - exercised via load_config + raise ValueError( + f"data.start / data.end must be 'YYYY-MM-DD' dates; got " + f"start={self.start!r}, end={self.end!r} ({exc})." + ) from exc + if start > end: + raise ValueError( + f"data.start ({self.start}) must be on or before data.end ({self.end})." + ) + return self + + +class UniverseFilters(_Strict): + missing_close: bool = True + suspended: bool = False + st: bool = False + limit_up_down: bool = False + + +class UniverseCfg(_Strict): + type: Literal["static", "index"] = "static" + symbols: list[str] = Field(default_factory=list) + index_code: str | None = None # required when type == "index" (PIT membership) + min_listing_days: int = 60 + filters: UniverseFilters = Field(default_factory=UniverseFilters) + + @model_validator(mode="after") + def _check_type_requirements(self) -> "UniverseCfg": + if self.type == "index" and not self.index_code: + raise ValueError( + "universe.type is 'index' but universe.index_code is not set " + "(e.g. '000300.SH' for CSI300)." + ) + return self + + +class FactorCfg(_Strict): + name: str + enabled: bool = True + params: dict[str, Any] = Field(default_factory=dict) + + +class StandardizeCfg(_Strict): + enabled: bool = True + method: Literal["zscore"] = "zscore" + + +class WinsorizeCfg(_Strict): + enabled: bool = False + method: str = "mad" + n: float = 3.0 + + +class NeutralizeCfg(_Strict): + enabled: bool = False + industry_col: str = "industry" + size_col: str = "market_cap" + + +class ProcessingCfg(_Strict): + drop_missing: bool = True + standardize: StandardizeCfg = Field(default_factory=StandardizeCfg) + winsorize: WinsorizeCfg = Field(default_factory=WinsorizeCfg) + neutralize: NeutralizeCfg = Field(default_factory=NeutralizeCfg) + + +class AlphaCfg(_Strict): + model: str = "equal_weight" + params: dict[str, Any] = Field(default_factory=dict) + + +class PortfolioCfg(_Strict): + constructor: str = "topn_equal_weight" + top_n: int + long_only: bool = True + max_weight: float | None = None + turnover_cap: float | None = None + + @field_validator("top_n") + @classmethod + def _check_top_n(cls, v: int) -> int: + if v <= 0: + raise ValueError(f"portfolio.top_n must be a positive integer; got {v}.") + return v + + +class BacktestCfg(_Strict): + initial_nav: float = 1.0 + rebalance: Literal["monthly"] = "monthly" + event_order: str = "close_to_next_period" + cash_return: float = 0.0 + + +class CostCfg(_Strict): + fee_rate: float = 0.001 + slippage_rate: float = 0.0 + turnover_formula: Literal["l1"] = "l1" + + +class AnalyticsCfg(_Strict): + forward_return_periods: list[int] = Field(default_factory=lambda: [1, 5, 20]) + quantiles: int = 5 + benchmark: str | None = None + + +class OutputCfg(_Strict): + root_dir: str = "artifacts" + data_dir: str = "artifacts/data" + factor_dir: str = "artifacts/factors" + report_dir: str = "artifacts/reports" + log_dir: str = "artifacts/logs" + overwrite: bool = True + + +class RootConfig(_Strict): + """Top-level config composing every section. + + Required top-level sections (CFG-002): data, universe, factors, alpha, + portfolio, backtest, cost, output. ``project``, ``processing`` and + ``analytics`` have sensible defaults but are present in the template. + """ + + project: ProjectCfg = Field(default_factory=lambda: ProjectCfg(name="quantitative_trading")) + data: DataCfg + universe: UniverseCfg + factors: list[FactorCfg] + processing: ProcessingCfg = Field(default_factory=ProcessingCfg) + alpha: AlphaCfg + portfolio: PortfolioCfg + backtest: BacktestCfg + cost: CostCfg + analytics: AnalyticsCfg = Field(default_factory=AnalyticsCfg) + output: OutputCfg + + +# --------------------------------------------------------------------------- # +# Loader +# --------------------------------------------------------------------------- # +class ConfigError(ValueError): + """User-readable configuration error (CLI-003).""" + + +# Map machine field names to a friendly hint for required-field errors. +_REQUIRED_HINTS = { + "data": "the 'data' section (source/start/end)", + "universe": "the 'universe' section (type/symbols)", + "factors": "the 'factors' list (e.g. [{name: momentum_20}])", + "alpha": "the 'alpha' section (model)", + "portfolio": "the 'portfolio' section (constructor/top_n)", + "backtest": "the 'backtest' section (rebalance)", + "cost": "the 'cost' section (fee_rate)", + "output": "the 'output' section (root_dir)", + "start": "data.start (a 'YYYY-MM-DD' date)", + "end": "data.end (a 'YYYY-MM-DD' date)", + "top_n": "portfolio.top_n (a positive integer)", +} + + +def _format_validation_error(err: ValidationError) -> str: + """Turn a pydantic ValidationError into a readable, multi-line message.""" + lines: list[str] = ["Invalid configuration:"] + for e in err.errors(): + loc = ".".join(str(p) for p in e["loc"]) + leaf = str(e["loc"][-1]) if e["loc"] else "" + msg = e["msg"] + if e["type"] == "missing": + hint = _REQUIRED_HINTS.get(leaf, f"'{loc}'") + lines.append(f" - missing required field: {hint}") + else: + lines.append(f" - {loc}: {msg}") + return "\n".join(lines) + + +def load_config(path: str) -> RootConfig: + """Read a YAML config file and return a validated ``RootConfig``. + + Raises ``ConfigError`` with a user-readable message (never a raw pydantic + traceback) if the file is missing, unparseable, or invalid. + """ + p = Path(path) + if not p.exists(): + raise ConfigError(f"Config file not found: {path}") + try: + raw = yaml.safe_load(p.read_text(encoding="utf-8")) + except yaml.YAMLError as exc: + raise ConfigError(f"Config file is not valid YAML ({path}): {exc}") from exc + + if raw is None: + raise ConfigError(f"Config file is empty: {path}") + if not isinstance(raw, dict): + raise ConfigError( + f"Config root must be a mapping of sections; got {type(raw).__name__} in {path}." + ) + + try: + return RootConfig(**raw) + except ValidationError as exc: + raise ConfigError(_format_validation_error(exc)) from exc diff --git a/qt/pipeline.py b/qt/pipeline.py new file mode 100644 index 0000000..ed73efb --- /dev/null +++ b/qt/pipeline.py @@ -0,0 +1,646 @@ +"""Phase 0 end-to-end orchestration (the spine that wires every slice). + +This module owns the *only* place where the slices are composed into a single +reproducible run: + + demo feed -> normalize_panel -> PanelStore.write/read + -> StaticUniverse -> MomentumFactor.compute + -> ProcessingPipeline.transform + -> EqualWeightAlpha.fit(None).predict (per rebalance date) + -> TopNEqualWeight.build + -> BacktestDriver.run (SimExecution) + -> analytics (IC via forward_returns, performance summary) + -> markdown report writers. + +It is deliberately thin glue: every layer keeps its own boundary. The CLI +(:mod:`qt.cli`) only parses args and calls :func:`run_phase0`. + +Design invariants honoured here (CLAUDE.md / CONTRACTS.md): + * factors never see forward returns (analytics is the forward-return boundary); + * portfolio never touches a data source or places orders; + * backtest/live share the ``Execution`` port (we use ``SimExecution``); + * the event order is fixed: factor at close[t], hold from t+1 (the driver + settles each rebalance against the NEXT holding period's return); + * all writes land under the configured ``output`` dirs (SEC-003); + * the run is re-entrant: re-running over existing files must not fail (INV-006). +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass +from pathlib import Path + +import pandas as pd + +from alpha.equal_weight import EqualWeightAlpha +from analytics.factor import compute_ic, forward_returns, ic_summary, quantile_returns +from analytics.performance import performance_summary +from data.clean.adjust import front_adjust +from data.clean.covariates import enrich_covariates +from data.clean.pit_financials import asof_financials +from data.clean.tradability import enrich_tradability +from data.feed.base import DataFeed +from data.feed.demo_feed import DemoFeed +from data.feed.index_feed import IndexConstituentsFeed +from data.feed.tushare_covariates import TushareCovariatesFeed +from data.feed.tushare_feed import TushareFeed +from data.feed.tushare_fina import TushareFinancialFeed +from data.feed.tushare_flags import TushareFlagsFeed +from data.store.panel_store import PanelStore +from factors.compute.financial import SUPPORTED_FIELDS as SUPPORTED_FINANCIAL_FIELDS +from factors.compute.financial import FinancialFactor +from factors.compute.momentum import MomentumFactor +from factors.process.pipeline import ProcessingPipeline +from portfolio.construct import TopNEqualWeight +from qt.config import RootConfig, load_config +from qt.reports import write_phase0_summary +from runtime.backtest.driver import BacktestDriver +from runtime.backtest.sim_execution import SimExecution +from universe.base import Universe +from universe.index_universe import PITIndexUniverse +from universe.static import StaticUniverse + +_LOGGER_NAME = "qt.run_phase0" + +# Periods per year per rebalance cadence. The nav table has one row per rebalance +# (monthly -> 12 rows/year), so performance metrics must annualize against that +# cadence, NOT the daily 252 that ``performance_summary`` defaults to. +_PERIODS_PER_YEAR: dict[str, int] = {"monthly": 12} +_DEFAULT_PERIODS_PER_YEAR: int = 12 +_INDEX_UNIVERSE_LOOKBACK_DAYS: int = 370 +# Financials are fetched from before the backtest start so the latest report +# DISCLOSED before start (its period end is months earlier than its ann_date) is +# in the set and can be as-of carried forward onto early trade dates. ~16 months +# covers an annual report disclosed up to a year-plus before start. +_FINANCIAL_LOOKBACK_DAYS: int = 500 + + +def _periods_per_year(rebalance: str) -> int: + """Annualization factor for the configured rebalance cadence (P0: monthly=12).""" + return _PERIODS_PER_YEAR.get(rebalance, _DEFAULT_PERIODS_PER_YEAR) + + +@dataclass(frozen=True) +class Phase0Result: + """Immutable summary of one phase0 run (what the report/tests consume).""" + + config: RootConfig + panel_rows: int + panel_symbols: int + factor_name: str + ic_mean: float + ic_ir: float + quantile_returns: pd.DataFrame + nav_table: pd.DataFrame + performance: dict[str, float] + avg_turnover: float + cost_drag: float + downgrades: tuple[str, ...] + data_path: Path + factor_path: Path + report_path: Path + log_path: Path + + +class _FrameScores: + """Scores source backed by a precomputed (date, symbol) score panel. + + Bridges the processed factor panel + alpha model to the ``ScoresSource`` + port the :class:`BacktestDriver` depends on. It only *reads* scores already + computed from past/current factors — it never sees forward returns, so the + no-lookahead boundary is preserved. + """ + + def __init__(self, score_panel: pd.Series) -> None: + # score_panel: MultiIndex(date, symbol) -> score (one column collapsed). + self._scores = score_panel + + def get(self, date: pd.Timestamp, symbols: list[str]) -> pd.Series: + """Return symbol-indexed scores for ``date`` restricted to ``symbols``.""" + norm = pd.Timestamp(date).normalize() + date_level = self._scores.index.get_level_values("date") + cross = self._scores.loc[date_level == norm] + if cross.empty: + return pd.Series(index=list(symbols), dtype=float) + cross = pd.Series(cross.to_numpy(), index=cross.index.get_level_values("symbol")) + return cross.reindex(list(symbols)) + + +def _make_logger(log_path: Path) -> logging.Logger: + """A run-scoped logger that writes to ``log_path`` (and never logs secrets).""" + log_path.parent.mkdir(parents=True, exist_ok=True) + logger = logging.getLogger(_LOGGER_NAME) + logger.setLevel(logging.INFO) + # Re-entrant: drop handlers from a previous run so we don't double-write. + for handler in list(logger.handlers): + logger.removeHandler(handler) + handler.close() + file_handler = logging.FileHandler(log_path, mode="w", encoding="utf-8") + file_handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s")) + logger.addHandler(file_handler) + logger.propagate = False + return logger + + +def _build_scores( + processed: pd.DataFrame, alpha: EqualWeightAlpha +) -> pd.Series: + """Predict one score per (date, symbol) from the processed factor panel. + + The alpha is fit with ``forward_returns=None`` (equal-weight needs no future + data) and predicts per-date cross-sections, so a single (date, symbol) score + panel is produced for the backtest scores source. + """ + alpha.fit(processed, None) + dates = processed.index.get_level_values("date") + blocks: list[pd.Series] = [] + for date, block in processed.groupby(dates, sort=True): + scores = alpha.predict(block) + idx = pd.MultiIndex.from_product([[date], list(scores.index)], names=["date", "symbol"]) + blocks.append(pd.Series(scores.to_numpy(), index=idx)) + if not blocks: + return pd.Series(dtype=float, index=processed.index[:0]) + return pd.concat(blocks).rename("score") + + +def _collect_downgrades(cfg: RootConfig) -> tuple[str, ...]: + """Enumerate the downgrades that MUST be disclosed (INV-007), path-aware. + + The first item states the active DATA PATH explicitly so a reader can never + mistake a demo run for a real PIT / financial validation, or vice versa. + """ + real = cfg.data.source == "tushare" + factor_name = cfg.factors[0].name if cfg.factors else "?" + is_financial = factor_name in SUPPORTED_FINANCIAL_FIELDS + + if real: + parts = ["front-adjusted (qfq) prices"] + parts.append( + f"PIT index membership ({cfg.universe.index_code})" + if cfg.universe.type == "index" + else "STATIC universe (still a PIT downgrade — see below)" + ) + parts.append( + f"ann_date-aligned financial factor '{factor_name}'" + if is_financial + else f"price factor '{factor_name}'" + ) + if cfg.processing.neutralize.enabled: + parts.append("industry+size neutralized") + path = "DATA PATH = REAL tushare: " + "; ".join(parts) + "." + else: + path = ( + "DATA PATH = DEMO / offline (DemoFeed, deterministic, network-free): " + "this is NOT real market data; results must NOT be read as a real PIT, " + "ann_date, or financial validation (CFG-005)." + ) + + # membership + if cfg.universe.type == "index": + membership = ( + f"Index universe (PITIndexUniverse, {cfg.universe.index_code}): " + "point-in-time membership from tushare index_weight snapshots (latest " + "snapshot on-or-before each date; survivorship-safe). RESOLVES the " + "static-universe PIT downgrade (UNI-003/UNI-009)." + ) + else: + membership = ( + "Static universe (StaticUniverse): membership is date-independent, NOT " + "point-in-time index constituents (UNI-003 PIT downgrade). Survivorship / " + "look-ahead membership bias is present." + ) + + # financials / ann_date + if is_financial: + financials = ( + f"Financial factor '{factor_name}' is ann_date PIT-aligned (DATA-012): " + "a figure is used only after its disclosure date, never by report period." + ) + else: + financials = ( + "No financial factor in this run; ann_date alignment is implemented and " + "available but unused here (price factor only)." + ) + + # neutralization + if cfg.processing.neutralize.enabled: + neutral = ( + "Factor is industry + market-cap neutralized; the industry tag is the " + "CURRENT one (stock_basic), a mild PIT downgrade (historical industry is " + "a later item)." + ) + else: + neutral = "No neutralization in this run (raw cross-sectional factor)." + + items = [ + path, + membership, + financials, + neutral, + f"Daily ({cfg.data.freq}) bars only; minute-level link is deferred.", + "IC / quantile returns use a simple numpy/pandas implementation, NOT " + "alphalens-reloaded (simple-vs-alphalens fallback, INV-007).", + "Performance metrics use a simple numpy/pandas implementation, NOT " + "quantstats (simple-vs-quantstats fallback, INV-007).", + f"universe.min_listing_days is configured ({cfg.universe.min_listing_days}) " + "but NOT enforced (no-op); newly listed names are not excluded (INV-007).", + ] + return tuple(items) + + +def run_phase0(config_path: str) -> Phase0Result: + """Run the full Phase 0 pipeline from a YAML config and write the reports. + + Returns a :class:`Phase0Result`. Raises ``ConfigError`` (readable) on a bad + config and ``ValueError`` (readable) on an empty/degenerate run. All file + writes are confined to the configured ``output`` directories (SEC-003). + """ + cfg = load_config(config_path) + + log_path = Path(cfg.output.log_dir) / "run_phase0.log" + logger = _make_logger(log_path) + logger.info("phase0 start: project=%s source=%s", cfg.project.name, cfg.data.source) + + universe, symbols = _build_universe(cfg, logger) + panel = _load_panel(cfg, symbols, logger) + factor = _build_factor(cfg) + panel = _maybe_enrich_financials(cfg, panel, symbols, factor, logger) + panel = _maybe_enrich_covariates(cfg, panel, symbols, logger) + factor_panel = _compute_factor_panel(cfg, panel, factor, logger) + + processed = _process_factors(cfg, factor_panel, panel) + scores = _build_scores(processed, EqualWeightAlpha()) + + nav_table = _run_backtest(cfg, panel, scores, factor.name, universe, logger) + ic_mean, ic_ir, q_returns = _factor_analytics(cfg, panel, factor_panel, factor.name) + perf = ( + performance_summary( + nav_table["nav"], + periods_per_year=_periods_per_year(cfg.backtest.rebalance), + ) + if not nav_table.empty + else { + "annual_return": float("nan"), + "max_drawdown": float("nan"), + "volatility": float("nan"), + "sharpe": float("nan"), + } + ) + avg_turnover = float(nav_table["turnover"].mean()) if not nav_table.empty else 0.0 + cost_drag = float(nav_table["cost"].sum()) if not nav_table.empty else 0.0 + + downgrades = _collect_downgrades(cfg) + result = Phase0Result( + config=cfg, + panel_rows=len(panel), + panel_symbols=panel.index.get_level_values("symbol").nunique(), + factor_name=factor.name, + ic_mean=ic_mean, + ic_ir=ic_ir, + quantile_returns=q_returns, + nav_table=nav_table, + performance=perf, + avg_turnover=avg_turnover, + cost_drag=cost_drag, + downgrades=downgrades, + data_path=Path(cfg.output.data_dir) / f"{cfg.data.output_name}.parquet", + factor_path=Path(cfg.output.factor_dir) / "factors.parquet", + report_path=Path(cfg.output.report_dir) / "phase0_summary.md", + log_path=log_path, + ) + write_phase0_summary(result) + logger.info( + "phase0 done: ic_mean=%.4f ic_ir=%.4f annual_return=%.4f report=%s", + ic_mean, + ic_ir, + perf.get("annual_return", float("nan")), + result.report_path, + ) + return result + + +# --------------------------------------------------------------------------- # +# Step helpers (each small + single-purpose; orchestration stays linear above). +# --------------------------------------------------------------------------- # +def _build_feed(cfg: RootConfig) -> DataFeed: + """Construct the DataFeed for the configured source (``demo`` | ``tushare``). + + Dispatches on ``cfg.data.source`` so a ``tushare`` config is NOT silently + served demo data. The tushare branch requires ``external_secret_file`` (the + token never lives in the repo); constructing the feed performs no network / + token read — that is deferred to first ``get_bars`` (SEC-001/004). + """ + if cfg.data.source == "demo": + return DemoFeed(calendar_start=cfg.data.start) + if cfg.data.source == "tushare": + secret_file = cfg.data.external_secret_file + if not secret_file: + raise ValueError( + "data.source is 'tushare' but data.external_secret_file is not set. " + "Point it at your .config.json (the token is read from there, never " + "hardcoded)." + ) + return TushareFeed( + secret_file=secret_file, + token_key=cfg.data.tushare_token_key, + ) + raise ValueError( + f"Unsupported data.source {cfg.data.source!r}; expected 'demo' or 'tushare'." + ) + + +def _build_universe( + cfg: RootConfig, logger: logging.Logger +) -> tuple[Universe, list[str]]: + """Construct the universe and the symbol set whose market data must be loaded. + + ``static`` -> :class:`StaticUniverse` over the configured symbols (PIT downgrade). + ``index`` -> :class:`PITIndexUniverse` from real tushare ``index_weight`` + snapshots; the symbol set is the union of all historical constituents so the + backtest can settle names that later left the index (no survivorship bias). + """ + filters = cfg.universe.filters.model_dump() + if cfg.universe.type == "static": + symbols = list(cfg.universe.symbols) + if not symbols: + raise ValueError( + "universe.symbols is empty; configure at least one symbol (static)." + ) + return StaticUniverse(symbols, filters), symbols + if cfg.universe.type == "index": + if cfg.data.source != "tushare": + raise ValueError( + "universe.type='index' requires data.source='tushare' " + "(constituents come from tushare index_weight; demo has no index)." + ) + if not cfg.data.external_secret_file: + raise ValueError( + "universe.type='index' requires data.external_secret_file " + "(token for index_weight)." + ) + feed = IndexConstituentsFeed( + cfg.data.external_secret_file, token_key=cfg.data.tushare_token_key + ) + # As-of membership at cfg.data.start needs the latest snapshot *before* + # the backtest window too. Pulling only [start, end] can leave early + # rebalance dates empty when the run starts between two index snapshots. + cons_start = ( + pd.Timestamp(cfg.data.start) - pd.Timedelta(days=_INDEX_UNIVERSE_LOOKBACK_DAYS) + ).strftime("%Y-%m-%d") + cons = feed.get_constituents(cfg.universe.index_code, cons_start, cfg.data.end) + if cons.empty: + raise ValueError( + f"No constituents for index {cfg.universe.index_code} over " + f"[{cfg.data.start}, {cfg.data.end}]." + ) + symbols = sorted(cons["symbol"].unique().tolist()) + logger.info( + "universe: index %s, %d snapshots, %d distinct constituents", + cfg.universe.index_code, + cons["date"].nunique(), + len(symbols), + ) + return PITIndexUniverse(cons, filters), symbols + raise ValueError(f"Unsupported universe.type {cfg.universe.type!r}.") + + +def _load_panel( + cfg: RootConfig, symbols: list[str], logger: logging.Logger +) -> pd.DataFrame: + """Fetch the market panel for ``symbols``, persist it, and read it back.""" + if not symbols: + raise ValueError("No symbols to load; the universe produced an empty set.") + feed = _build_feed(cfg) + panel = feed.get_bars(symbols, cfg.data.start, cfg.data.end, freq=cfg.data.freq) + if panel.empty: + raise ValueError( + "No market data returned for the configured window " + f"[{cfg.data.start}, {cfg.data.end}] and symbols {symbols}." + ) + + store = PanelStore(cfg.output.data_dir) + store.write(cfg.data.output_name, panel, overwrite=cfg.output.overwrite) + panel = store.read(cfg.data.output_name) + # Tradability flags FIRST, on RAW prices: the price-limit flags must compare + # the UNADJUSTED close against the raw stk_limit (limits are quoted in raw + # price terms). front-adjust below only scales OHLC and leaves the boolean + # flags intact, so factors/backtest see qfq prices while limit flags stay right. + panel = _enrich_tradability(cfg, panel, symbols, logger) + # Store stays RAW (+ adj_factor, incremental-safe); front-adjust in memory so + # factors / backtest see continuous qfq prices. Identity for demo (adj=1.0). + panel = front_adjust(panel) + logger.info( + "data: %d rows, %d symbols (front-adjusted)", + len(panel), + panel.index.get_level_values("symbol").nunique(), + ) + return panel + + +def _enrich_tradability( + cfg: RootConfig, panel: pd.DataFrame, symbols: list[str], logger: logging.Logger +) -> pd.DataFrame: + """Add suspended / ST / price-limit flags when filters need them (tushare only). + + No-op for the offline demo source (no flag data) and when no flag-driven + filter is enabled, so the demo path is unchanged. Flags are joined as boolean + columns that :func:`universe.filters.apply_tradable_filters` consults. + """ + flt = cfg.universe.filters + if cfg.data.source != "tushare" or not (flt.suspended or flt.st or flt.limit_up_down): + return panel + feed = TushareFlagsFeed( + cfg.data.external_secret_file, token_key=cfg.data.tushare_token_key + ) + suspended = feed.suspended(symbols, cfg.data.start, cfg.data.end) if flt.suspended else None + st = feed.st_intervals(symbols) if flt.st else None + limits = feed.limits(symbols, cfg.data.start, cfg.data.end) if flt.limit_up_down else None + panel = enrich_tradability(panel, suspended=suspended, st_intervals=st, limits=limits) + logger.info( + "tradability: flags enriched (suspended=%s st=%s limit_up_down=%s)", + flt.suspended, flt.st, flt.limit_up_down, + ) + return panel + + +def _build_factor(cfg: RootConfig): + """Instantiate the (single, first-enabled) factor from config by name. + + ``momentum*`` -> :class:`MomentumFactor` (price-based, P0). A financial field + name (e.g. ``roe``, ``netprofit_yoy``) -> :class:`FinancialFactor`, which needs + the tushare data path (ann_date alignment) — not available for demo data. + """ + enabled = [f for f in cfg.factors if f.enabled] + if not enabled: + raise ValueError("No enabled factor in config.factors; phase0 needs a factor.") + spec = enabled[0] + params = dict(spec.params) + if spec.name in SUPPORTED_FINANCIAL_FIELDS: + return FinancialFactor(field=spec.name) + if spec.name.startswith("momentum"): + return MomentumFactor( + window=int(params.get("window", 20)), + price_col=str(params.get("price_col", "close")), + ) + raise ValueError( + f"Unknown factor {spec.name!r}; expected 'momentum*' or one of " + f"{SUPPORTED_FINANCIAL_FIELDS}." + ) + + +def _maybe_enrich_financials( + cfg: RootConfig, + panel: pd.DataFrame, + symbols: list[str], + factor, + logger: logging.Logger, +) -> pd.DataFrame: + """Attach the ann_date-aligned financial column when a financial factor is used. + + Financial factors require the tushare data path: a demo run has no disclosure + dates, so we fail with a readable error rather than fabricate financials. + """ + if not isinstance(factor, FinancialFactor): + return panel + if cfg.data.source != "tushare": + raise ValueError( + f"Factor '{factor.name}' is a financial factor and needs real financial " + f"data (data.source='tushare'); it cannot run on demo data." + ) + feed = TushareFinancialFeed( + cfg.data.external_secret_file, token_key=cfg.data.tushare_token_key + ) + # Look back before start so the prior already-disclosed report is fetched and + # can be as-of carried forward onto the early trade dates (no NaN gap). + fetch_start = ( + pd.Timestamp(cfg.data.start) - pd.Timedelta(days=_FINANCIAL_LOOKBACK_DAYS) + ).strftime("%Y-%m-%d") + fina = feed.get_fina_indicator(symbols, fetch_start, cfg.data.end, fields=[factor.name]) + enriched = panel.copy() + aligned = asof_financials(panel.index, fina, [factor.name]) + enriched[factor.name] = aligned[factor.name] + logger.info( + "financials: as-of aligned '%s' by ann_date (%d disclosed rows)", + factor.name, len(fina), + ) + return enriched + + +def _compute_factor_panel( + cfg: RootConfig, + panel: pd.DataFrame, + factor: MomentumFactor, + logger: logging.Logger, +) -> pd.DataFrame: + """Compute the factor series, frame it, and persist it via PanelStore-style write.""" + series = factor.compute(panel) + factor_panel = series.to_frame(name=factor.name) + _write_factor_panel(cfg, factor_panel) + logger.info("factor: %s computed (%d rows)", factor.name, len(factor_panel)) + return factor_panel + + +def _write_factor_panel(cfg: RootConfig, factor_panel: pd.DataFrame) -> None: + """Persist the factor panel to ``factors/factors.parquet`` (re-entrant).""" + target = Path(cfg.output.factor_dir) / "factors.parquet" + target.parent.mkdir(parents=True, exist_ok=True) + flat = factor_panel.reset_index() + tmp = target.with_suffix(".parquet.tmp") + flat.to_parquet(tmp, engine="pyarrow", index=False) + tmp.replace(target) + + +def _process_factors( + cfg: RootConfig, factor_panel: pd.DataFrame, panel: pd.DataFrame +) -> pd.DataFrame: + """Run drop_missing + (winsorize) + neutralize + z-score from config toggles. + + Neutralization pulls the ``industry`` / ``market_cap`` covariates off the panel + (placed there by :func:`_maybe_enrich_covariates`). When neutralize is enabled + but they are absent, ``ProcessingPipeline`` raises a readable error. + """ + industry = panel["industry"] if "industry" in panel.columns else None + market_cap = panel["market_cap"] if "market_cap" in panel.columns else None + pipeline = ProcessingPipeline( + drop_missing=cfg.processing.drop_missing, + standardize=cfg.processing.standardize.enabled, + winsorize=cfg.processing.winsorize.enabled, + neutralize=cfg.processing.neutralize.enabled, + industry=industry, + market_cap=market_cap, + ) + return pipeline.transform(factor_panel) + + +def _maybe_enrich_covariates( + cfg: RootConfig, panel: pd.DataFrame, symbols: list[str], logger: logging.Logger +) -> pd.DataFrame: + """Attach industry + market_cap when neutralization is enabled (tushare only).""" + if not cfg.processing.neutralize.enabled: + return panel + if cfg.data.source != "tushare": + raise ValueError( + "processing.neutralize is enabled but data.source is not 'tushare'; " + "industry + market_cap need real data (demo has neither)." + ) + feed = TushareCovariatesFeed( + cfg.data.external_secret_file, token_key=cfg.data.tushare_token_key + ) + industry = feed.industry(symbols) + market_cap = feed.market_cap(symbols, cfg.data.start, cfg.data.end) + panel = enrich_covariates(panel, industry=industry, market_cap=market_cap) + logger.info( + "covariates: industry(%d symbols) + market_cap(%d rows) for neutralization", + len(industry), len(market_cap), + ) + return panel + + +def _run_backtest( + cfg: RootConfig, + panel: pd.DataFrame, + scores: pd.Series, + factor_name: str, + universe: Universe, + logger: logging.Logger, +) -> pd.DataFrame: + """Wire the universe + scores + constructor + execution into the driver.""" + constructor = TopNEqualWeight(cfg.portfolio.top_n, long_only=cfg.portfolio.long_only) + execution = SimExecution(fee_rate=cfg.cost.fee_rate, cash_return=cfg.backtest.cash_return) + driver = BacktestDriver( + universe=universe, + scores=_FrameScores(scores), + constructor=constructor, + execution=execution, + prices=panel, + rebalance=cfg.backtest.rebalance, + fee_rate=cfg.cost.fee_rate, + initial_nav=cfg.backtest.initial_nav, + cash_return=cfg.backtest.cash_return, + ) + nav_table = driver.run() + logger.info("backtest: %d rebalance rows", len(nav_table)) + return nav_table + + +def _factor_analytics( + cfg: RootConfig, + panel: pd.DataFrame, + factor_panel: pd.DataFrame, + factor_name: str, +) -> tuple[float, float, pd.DataFrame]: + """IC summary + quantile returns from the RAW factor + forward returns. + + Analytics is the only place forward returns are computed (INV-001). We use + the first configured forward-return period for IC (the holding horizon proxy). + """ + periods = tuple(int(p) for p in cfg.analytics.forward_return_periods) + fwd = forward_returns(panel, periods=periods) + horizon = periods[0] + fwd_col = fwd[f"forward_return_{horizon}d"] + factor_series = factor_panel[factor_name] + ic = compute_ic(factor_series, fwd_col) + summary = ic_summary(ic) + q_returns = quantile_returns(factor_series, fwd_col, quantiles=cfg.analytics.quantiles) + return summary["ic_mean"], summary["ic_ir"], q_returns diff --git a/qt/reports.py b/qt/reports.py new file mode 100644 index 0000000..cc15671 --- /dev/null +++ b/qt/reports.py @@ -0,0 +1,242 @@ +"""Markdown report writers for the Phase 0 run (ANA-005, INV-006/007). + +``write_phase0_summary`` renders the end-to-end run result into +``artifacts/reports/phase0_summary.md``. It only WRITES under the configured +output dirs (SEC-003) and is fully regenerable from the same config (INV-006). + +The summary always contains an explicit DOWNGRADES section (INV-007) so the +static-universe PIT downgrade, the daily-data limitation, and any simple-vs- +alphalens / simple-vs-quantstats fallback are disclosed in the report itself. + +``write_bias_audit`` and ``write_runbook`` / ``write_test_report`` emit the +repo-root delivery docs from the framework spec §10. +""" + +from __future__ import annotations + +import math +from pathlib import Path +from typing import TYPE_CHECKING + +import pandas as pd + +if TYPE_CHECKING: # avoid a circular import at runtime (pipeline imports reports) + from qt.pipeline import Phase0Result + + +def _fmt(value: float, pct: bool = False) -> str: + """Format a float for the report (NaN/inf-safe; optional percent).""" + if value is None or not math.isfinite(value): # NaN or +/-inf + return "n/a" + return f"{value * 100:.2f}%" if pct else f"{value:.4f}" + + +def _quantile_table(q_returns: pd.DataFrame) -> str: + """Render mean-over-time quantile returns as a small markdown table.""" + if q_returns is None or q_returns.empty: + return "_(no quantile data)_" + means = q_returns.mean(axis=0) + header = "| Quantile | Mean forward return |\n|---|---|\n" + rows = "".join(f"| {int(q)} | {_fmt(float(v), pct=True)} |\n" for q, v in means.items()) + return header + rows + + +def render_phase0_summary(result: "Phase0Result") -> str: + """Build the phase0 summary markdown string (pure; no I/O).""" + cfg = result.config + perf = result.performance + lines: list[str] = [] + lines.append("# Phase 0 Summary\n") + lines.append(f"Project: **{cfg.project.name}**\n") + + lines.append("## Config echo\n") + lines.append( + f"- data: source=`{cfg.data.source}`, freq=`{cfg.data.freq}`, " + f"window=`[{cfg.data.start}, {cfg.data.end}]`\n" + f"- universe: type=`{cfg.universe.type}`, " + f"symbols={list(cfg.universe.symbols)}\n" + f"- factor: `{result.factor_name}`\n" + f"- alpha: `{cfg.alpha.model}`\n" + f"- portfolio: `{cfg.portfolio.constructor}`, top_n=`{cfg.portfolio.top_n}`\n" + f"- backtest: rebalance=`{cfg.backtest.rebalance}`, " + f"event_order=`{cfg.backtest.event_order}`\n" + f"- cost: fee_rate=`{cfg.cost.fee_rate}`, slippage=`{cfg.cost.slippage_rate}`\n" + ) + + lines.append("## Data shape\n") + lines.append( + f"- panel rows: **{result.panel_rows}**\n" + f"- symbols: **{result.panel_symbols}**\n" + ) + + lines.append("## Factor IC\n") + lines.append( + f"- IC mean: **{_fmt(result.ic_mean)}**\n" + f"- IC_IR (mean/std): **{_fmt(result.ic_ir)}**\n" + ) + + lines.append("## Quantile returns\n") + lines.append(_quantile_table(result.quantile_returns) + "\n") + + lines.append("## Portfolio performance\n") + lines.append( + f"- annual return: **{_fmt(perf.get('annual_return', float('nan')), pct=True)}**\n" + f"- max drawdown: **{_fmt(perf.get('max_drawdown', float('nan')), pct=True)}**\n" + f"- volatility: **{_fmt(perf.get('volatility', float('nan')), pct=True)}**\n" + f"- sharpe: **{_fmt(perf.get('sharpe', float('nan')))}**\n" + ) + + lines.append("## Turnover & cost\n") + lines.append( + f"- average turnover (per rebalance): **{_fmt(result.avg_turnover)}**\n" + f"- total cost drag: **{_fmt(result.cost_drag, pct=True)}**\n" + ) + + lines.append("## DOWNGRADES (INV-007 — must be disclosed)\n") + lines.append( + "This Phase 0 run intentionally uses simplified / downgraded components. " + "Each is recorded here so no downgrade is hidden:\n" + ) + for item in result.downgrades: + lines.append(f"- {item}\n") + + lines.append("\n## Artifacts\n") + lines.append( + f"- data: `{result.data_path}`\n" + f"- factors: `{result.factor_path}`\n" + f"- report: `{result.report_path}`\n" + f"- log: `{result.log_path}`\n" + ) + return "".join(lines) + + +def write_phase0_summary(result: "Phase0Result") -> Path: + """Render and write the phase0 summary; return the path written (SEC-003).""" + target = result.report_path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(render_phase0_summary(result), encoding="utf-8") + return target + + +# --------------------------------------------------------------------------- # +# Repo-root delivery docs (framework spec §10). These describe the run; they +# carry no secrets and are regenerable. +# --------------------------------------------------------------------------- # +_BIAS_AUDIT_SECTIONS = ( + "未来函数 / lookahead", + "PIT 成分股", + "可交易过滤", + "ann_date 财务对齐", + "复权", + "交易成本", +) + + +def render_bias_audit() -> str: + """Build the BIAS_AUDIT.md markdown (Slice 13; all required sections).""" + return ( + "# Bias Audit (Phase 0)\n\n" + "本文件记录 P0 框架对各类偏差/未来函数的处理状态与降级。每个小节标注当前" + "状态(已处理 / 降级 / 待办)。\n\n" + "## 未来函数 / lookahead\n\n" + "- 状态: **已处理(P0)**。\n" + "- `momentum_20[t] = close[t] / close[t-window] - 1`,严格只用 t 及之前" + "的收盘价(`groupby(symbol).shift(window)`)。\n" + "- 事件顺序固定:在 t 收盘计算因子,t 收盘后调仓,从 t+1 持有。回测用" + "**下一持有期**的收益结算,绝不使用因子已经看见的当日收益。\n" + "- forward returns 只在 `analytics/` 计算,因子层永远拿不到未来收益" + "(INV-001)。\n\n" + "## PIT 成分股\n\n" + "- 状态: **PIT 已实现(P1) / StaticUniverse 为离线降级**。\n" + "- `PITIndexUniverse`(`universe.type=index`)用 tushare `index_weight` 的" + "历史快照做 as-of 成分:`members(date)` 取 ≤date 的最近快照,绝不用未来快照" + "(UNI-009)。被剔除的票在其在册期内仍是成员 —— 无幸存者偏差、无成分前视。\n" + "- pipeline 构建 index universe 时会额外向回看 370 天成分快照,确保回测从两次" + "成分调整中间开始时,起始日也能取到“开始日前最近快照”,而不是错误空仓。\n" + "- 实证:沪深300 2024 全年 24 个快照、328 个不同成分(每快照 300),28 进" + "28 出;`000069.SZ` 在 06-03 在册、06-28 已剔,各按其时代正确归属。\n" + "- 数据坑:`index_weight` 单次约 6000 行上限,长窗口会**静默丢最早快照**;" + "feed 已分 90 天窗口分页拉取规避。\n" + "- `StaticUniverse`(`universe.type=static`,demo/离线用)成分与日期无关" + "(UNI-003),是**降级**:存在幸存者 / 成分前视偏差,仅供无网络的 demo 跑通," + "并在 `phase0_summary.md` 的 DOWNGRADES 小节显式记录。\n\n" + "## 可交易过滤\n\n" + "- 状态: **停牌 / ST / 涨跌停已实现(P1)**。\n" + "- `missing_close`(总是开):截面日 `close` 为 NaN 的标的不可交易(UNI-004)。\n" + "- 统一在 `universe.filters.apply_tradable_filters` 按 `UniverseFilters` 开关" + "执行;flag 由 `data.clean.tradability.enrich_tradability` 从 tushare " + "`suspend_d` / `namechange` / `stk_limit` 富化到 panel(StaticUniverse 与 " + "PITIndexUniverse 共用)。demo 无 flag 数据时各过滤自动 no-op。\n" + "- **ST(UNI-006)**:`namechange` 名称区间含 'ST'/'*ST' 即标记,按 date 取" + "生效名称(实证:`000005.SZ` 2024 全程 ST,正确剔除)。\n" + "- **涨跌停(UNI-007)**:用**未复权 raw close** 与当日 raw `up_limit`/" + "`down_limit` 比较,标记 `at_up_limit`/`at_down_limit`(qfq 复权价仅用于因子/" + "回测收益;flag 富化在 front_adjust **之前**完成,故比较的是同口径 raw 价)。" + "实证:`000005.SZ` 2024-02-01 触跌停。当前选股层对两个方向都剔除;**方向感知**" + "(买入只看涨停、持有跌停不强卖)属执行层,后续细化。\n" + "- **停牌(UNI-005)**:`suspend_d` 标记停牌日。**实测发现**:tushare 全天停牌" + "当日**无 bar** → 已被 `missing_close` 剔除,故显式 suspended flag 与之重叠;" + "其价值在盘中停牌(`suspend_timing`)或会给停牌日 bar 的数据源,属防御性。\n" + "- 退市 / 无数据标的(如 `000003.SZ`)同样表现为不在 panel 而被剔除。PIT 历史" + "成分见上节。\n" + "- `universe.min_listing_days` 已在配置中(默认 60),但仍 **未执行**(no-op," + "降级):新上市标的不会被剔除。显式披露(INV-007),后续接上市日期后强制。\n\n" + "## ann_date 财务对齐\n\n" + "- 状态: **已实现(P1)**。\n" + "- 财务因子(`roe` / `netprofit_yoy`)经 `data.clean.pit_financials.asof_financials` " + "按披露日 `ann_date` 做 backward as-of 对齐:每个 trade_date 只取 " + "`ann_date <= trade_date` 的最近一期报告,**绝不按 `end_date`(报告期末)join**" + "(DATA-012)。\n" + "- 拉取窗口向回看约 16 个月(`start` 之前),确保回测 `start` 前已披露的上一期" + "财报在集合内、能 as-of **carry forward** 到早期交易日,避免早期 NaN 缺口。\n" + "- 实证:平安银行 2024 Q1(end_date 2024-03-31)披露日 ann_date 2024-04-20;" + "as-of roe 在 04-19 仍是上一期年报值(10.2436),04-22 才切到 Q1(3.1176)——" + "晚于报告期末约 3 周,证明无未来披露泄漏。\n" + "- 财务因子仅在 tushare 数据路径可用;demo 无披露日,配置财务因子 + demo 源" + "会报可读错误,**不伪造财务**。\n\n" + "## 复权\n\n" + "- 状态: **前复权已实现(P1)**。\n" + "- panel 始终携带 `adj_factor` 列(DemoFeed 中恒为 1.0)。`data/clean/adjust.py` " + "的 `front_adjust` 用 `adj_factor` 做前复权(qfq),在 pipeline 读盘后、因子" + "计算前于内存中应用(DATA-003)。\n" + "- 约定:按 symbol 锚定窗口内最新日 " + "(`qfq = raw × adj_factor / adj_factor[latest]`)。锚定项在任何价格比值中" + "约掉,故所有收益率 / 因子值对锚定与扩窗都不变 —— PanelStore 保持 raw" + "(+adj_factor),复权在内存做,batch≡incremental 一致。\n" + "- 实证:平安银行 2024-06-14 除权,raw 当日 -5.74%(分红跳空),qfq +0.99%" + "(真实涨跌);momentum_20 因此最多变动 6.77pp。demo(adj=1.0)下为恒等。\n\n" + "## 交易成本\n\n" + "- 状态: **已处理(P0)**。\n" + "- 成本 = L1 换手 × `fee_rate`;`turnover = sum(|target_w - current_w|)`," + "在 symbol 并集上对齐计算。\n" + "- 每个调仓期 `net_return = gross_return - cost`,成本拖累在" + "`phase0_summary.md` 中汇总(BT-004)。slippage 参数已预留。\n" + "- **结算价缺失约定(P0 降级)**:若持仓标的在持有期末(end)的 `close` " + "为 NaN(停牌 / 缺数据),回测以 0.0(持平)记其该期收益,而非剔除或用" + "最近可得价结算。该约定在此显式披露(INV-007);P1 接入真实停牌/退市" + "处理后改进结算逻辑。\n\n" + "## 中性化\n\n" + "- 状态: **行业 + 市值中性化已实现(P1)**。\n" + "- `factors.process.neutralize.neutralize_by_date`:每个 date 截面把因子对 " + "`[log(market_cap), one-hot(industry)]` 做 OLS,取残差,移除规模与行业暴露。" + "缺行业 / 市值,或**残差自由度 ≤ 0**(名称数 ≤ 1+行业数,饱和拟合会给出无意义" + "的伪 0 残差)时返回 **NaN**,绝不静默乱算;`processing.neutralize` 开启但协变量" + "缺失(如 demo 路径)直接报可读错误。\n" + "- 实证:12 只票横跨 4 行业(2024-09-30),corr(原始 momentum, log市值) " + "= -0.617 → 中性化后 -0.000,各行业残差均值 ≈ 0,确认规模/行业暴露被移除。\n" + "- **降级**:行业来自 `stock_basic.industry` 的**当前**行业标签,非按历史时点," + "故行业中性化带有轻微成分前视(市值 `daily_basic.total_mv` 为逐日真值)。" + "PIT 行业历史是后续项,此降级在此显式披露(INV-007)。\n" + ) + + +def write_bias_audit(repo_root: Path) -> Path: + """Write BIAS_AUDIT.md at the repo root and return its path.""" + target = Path(repo_root) / "BIAS_AUDIT.md" + target.write_text(render_bias_audit(), encoding="utf-8") + return target + + +def bias_audit_required_sections() -> tuple[str, ...]: + """The section titles the bias audit must contain (Slice 13 contract).""" + return _BIAS_AUDIT_SECTIONS diff --git a/runtime/backtest/driver.py b/runtime/backtest/driver.py new file mode 100644 index 0000000..357b4a4 --- /dev/null +++ b/runtime/backtest/driver.py @@ -0,0 +1,193 @@ +"""BacktestDriver: wires the strategy layers through the fixed event order. + +The driver owns no strategy logic of its own; it is pure orchestration over +injected collaborators (Ports): + + universe -> .members / .tradable(date, panel) + scores -> .get(date, symbols) -> symbol-indexed scores + constructor -> .build(scores, current_weights) -> target weights + execution -> BacktestExecution port (Execution + settle / last_cost / + last_turnover; the backtest-only sub-port) + prices -> canonical (date, symbol) panel with a ``close`` column + +Fixed event order (CONTRACTS §6): + + compute factor at close of t -> rebalance after close of t -> hold from t+1 + +So each rebalance is settled against the NEXT holding period's close-to-close +return (BT-003), never the same-day return the factor already saw. Empty +tradable universe -> the book is cash and earns ``cash_return`` (BT-007). + +Output (BT-005/006, framework_settings §7.9): a date-indexed DataFrame with +columns ``[nav, gross_return, cost, turnover, net_return]``. +""" + +from __future__ import annotations + +from typing import Protocol + +import pandas as pd + +from runtime.execution import BacktestExecution + + +class ScoresSource(Protocol): + """Minimal scores port the driver depends on.""" + + def get(self, date: pd.Timestamp, symbols: list[str]) -> pd.Series: ... + + +class UniversePort(Protocol): + """Minimal universe port the driver depends on.""" + + def tradable(self, date: pd.Timestamp, panel: pd.DataFrame) -> list[str]: ... + + +class ConstructorPort(Protocol): + """Minimal portfolio-constructor port the driver depends on.""" + + def build(self, scores: pd.Series, current_weights: pd.Series | None = ...) -> pd.Series: ... + + +_NAV_COLUMNS = ["nav", "gross_return", "cost", "turnover", "net_return"] + + +class BacktestDriver: + """Monthly-rebalanced, single-period-compounding backtest over a price panel.""" + + def __init__( + self, + *, + universe: UniversePort, + scores: ScoresSource, + constructor: ConstructorPort, + execution: BacktestExecution, + prices: pd.DataFrame, + rebalance: str = "monthly", + fee_rate: float = 0.0, + initial_nav: float = 1.0, + cash_return: float = 0.0, + ) -> None: + if rebalance != "monthly": + raise ValueError( + f"only 'monthly' rebalance is supported in P0, got {rebalance!r}" + ) + if "close" not in prices.columns: + raise ValueError("price panel must have a 'close' column") + self._universe = universe + self._scores = scores + self._constructor = constructor + self._execution = execution + self._prices = prices + self._rebalance = rebalance + self._fee_rate = float(fee_rate) + self._initial_nav = float(initial_nav) + self._cash_return = float(cash_return) + + # -- calendar --------------------------------------------------------- # + def _calendar(self) -> pd.DatetimeIndex: + """Sorted unique trading dates present in the price panel.""" + dates = self._prices.index.get_level_values("date").unique() + return pd.DatetimeIndex(sorted(dates)) + + def rebalance_dates(self) -> list[pd.Timestamp]: + """Last trading day of each month in the panel calendar (BT-001).""" + cal = self._calendar() + if len(cal) == 0: + return [] + frame = pd.DataFrame({"date": cal}) + frame["ym"] = frame["date"].dt.to_period("M") + last = frame.groupby("ym")["date"].max().sort_values() + return list(last) + + # -- pricing ---------------------------------------------------------- # + def _close_at(self, date: pd.Timestamp, symbol: str) -> float: + """Close of ``symbol`` on ``date`` (NaN if missing).""" + try: + return float(self._prices.loc[(date, symbol), "close"]) + except KeyError: + return float("nan") + + def _holding_returns( + self, start: pd.Timestamp, end: pd.Timestamp, symbols: list[str] + ) -> pd.Series: + """Close-to-close gross return per symbol over the FORWARD window. + + ``start`` is the rebalance (close of t); ``end`` is the next rebalance + date (or the final trading day). The position is held over (start, end], + so this is the next holding period's return — never a window ending on + ``start`` itself (BT-003). Symbols with a missing/zero start price get a + flat (0.0) return rather than NaN, so the book stays well-defined. + """ + out: dict[str, float] = {} + for sym in symbols: + start_px = self._close_at(start, sym) + end_px = self._close_at(end, sym) + if ( + start_px is None + or end_px is None + or pd.isna(start_px) + or pd.isna(end_px) + or start_px == 0.0 + ): + out[sym] = 0.0 + else: + out[sym] = end_px / start_px - 1.0 + return pd.Series(out, dtype=float) + + # -- run -------------------------------------------------------------- # + def run(self) -> pd.DataFrame: + """Run the backtest and return the NAV table (BT-005/006).""" + reb = self.rebalance_dates() + cal = self._calendar() + rows: list[dict] = [] + nav = self._initial_nav + for i, date in enumerate(reb): + end = reb[i + 1] if i + 1 < len(reb) else cal[-1] + # A rebalance on the final trading day has no forward holding window + # (end <= start) — settling it would emit a spurious zero-length + # period (close[d]/close[d]-1 == 0). Skip it (BT-003 honesty). + if end <= date: + continue + turnover, cost, gross, net = self._step(date, end) + nav = nav * (1.0 + net) + rows.append( + { + "date": date, + "nav": nav, + "gross_return": gross, + "cost": cost, + "turnover": turnover, + "net_return": net, + } + ) + out = pd.DataFrame(rows, columns=["date", *_NAV_COLUMNS]) + return out.set_index("date") + + def _step( + self, date: pd.Timestamp, end: pd.Timestamp + ) -> tuple[float, float, float, float]: + """One rebalance: universe -> scores -> build -> rebalance -> settle. + + Returns ``(turnover, cost, gross_return, net_return)`` for the period + held from ``date`` (exclusive) to ``end`` (inclusive). + """ + tradable = list(self._universe.tradable(date, self._prices)) + if not tradable: + # Cash book: no trade, no turnover/cost, earns cash_return (BT-007). + # The driver owns cash semantics so cash_return is honoured even when + # the execution adapter's own cash_return differs. + self._execution.rebalance_to(pd.Series(dtype=float), date) + cost = self._execution.last_cost + gross = self._cash_return + return (self._execution.last_turnover, cost, gross, gross - cost) + scores = self._scores.get(date, tradable) + current = self._execution.positions() + target = self._constructor.build(scores, current) + self._execution.rebalance_to(target, date) + holding = self._holding_returns(date, end, list(target.index)) + net = self._execution.settle(holding) + turnover = self._execution.last_turnover + cost = self._execution.last_cost + gross = net + cost + return (turnover, cost, gross, net) diff --git a/runtime/backtest/sim_execution.py b/runtime/backtest/sim_execution.py new file mode 100644 index 0000000..835d73f --- /dev/null +++ b/runtime/backtest/sim_execution.py @@ -0,0 +1,131 @@ +"""SimExecution: the backtest adapter for the Execution port. + +This is the simulated counterpart to a live broker (CLAUDE.md invariant #2, +INV-003): identical ``rebalance_to`` / ``positions`` / ``last_return`` surface, +so the strategy code above it is unchanged between backtest and live. + +Math pinned by CONTRACTS / BT-004: + + turnover = sum(|target_w - current_w|) # full L1, aligned over the union + cost = turnover * fee_rate + net_return (of a holding period) = gross_portfolio_return - cost + +The gross holding return is injected (``settle``) rather than read from a data +source here, which keeps the return/cost arithmetic unit-testable in isolation +and keeps this adapter free of any market-data dependency. +""" + +from __future__ import annotations + +import pandas as pd + +from runtime.execution import BacktestExecution + + +class SimExecution(BacktestExecution): + """Simulated execution: tracks weights, turnover, trading cost, net return. + + Subclasses :class:`BacktestExecution` (itself an ``Execution``) so the + strategy code above it is identical between backtest and live (INV-003), + while the backtest-only ``settle`` / ``last_cost`` / ``last_turnover`` hooks + live on the sub-port the driver depends on. Positions start empty; each + ``rebalance_to`` records the turnover/cost of moving to the new target and + replaces the book. + """ + + def __init__(self, fee_rate: float = 0.0, cash_return: float = 0.0) -> None: + if fee_rate < 0: + raise ValueError(f"fee_rate must be >= 0, got {fee_rate!r}") + self._fee_rate = float(fee_rate) + self._cash_return = float(cash_return) + self._positions: pd.Series = pd.Series(dtype=float) + self._last_turnover: float = 0.0 + self._last_cost: float = 0.0 + self._last_return: float = 0.0 + + # -- properties ------------------------------------------------------- # + @property + def fee_rate(self) -> float: + return self._fee_rate + + @property + def cash_return(self) -> float: + return self._cash_return + + @property + def last_turnover(self) -> float: + """L1 turnover recorded by the most recent ``rebalance_to``.""" + return self._last_turnover + + @property + def last_cost(self) -> float: + """Trading cost (turnover * fee_rate) of the most recent rebalance.""" + return self._last_cost + + # -- Execution port --------------------------------------------------- # + def rebalance_to(self, target_weights: pd.Series, date: pd.Timestamp) -> None: + """Move the book toward ``target_weights`` as of the close of ``date``. + + Computes full-L1 turnover against the current book over the union of + symbols, stores the resulting cost, and replaces the current positions + with a clean copy of the target. Does not mutate the input. The new book + is held from the next trading day (fixed event order). + """ + target = self._clean_weights(target_weights) + current = self._positions + union = current.index.union(target.index) + aligned_target = target.reindex(union, fill_value=0.0) + aligned_current = current.reindex(union, fill_value=0.0) + turnover = float((aligned_target - aligned_current).abs().sum()) + self._last_turnover = turnover + self._last_cost = turnover * self._fee_rate + # Settle to the (gross) cost-only return for this rebalance until the + # holding period is settled; keeps last_return defined right after a + # rebalance with no holding info yet. + self._last_return = -self._last_cost + self._positions = target.copy() + + def positions(self) -> pd.Series: + """Return current symbol-indexed weights (after the last rebalance).""" + return self._positions.copy() + + def last_return(self) -> float: + """Net (after-cost) portfolio return of the last settled holding period.""" + return self._last_return + + # -- BacktestExecution port (backtest-only, not on the live port) ------ # + def settle(self, holding_returns: pd.Series | None) -> float: + """Settle the current book against per-symbol gross holding returns. + + Args: + holding_returns: symbol-indexed gross simple returns for the period + the current book is held. ``None``/empty -> the book is cash, so + the gross return is ``cash_return``. + + Returns: + Net return = gross portfolio return - cost of forming this book. + Symbols in the book but missing from ``holding_returns`` contribute + zero (treated as flat). Stored as ``last_return``. + """ + book = self._positions + if book.empty: + gross = self._cash_return + else: + if holding_returns is None: + rets = pd.Series(dtype=float) + else: + rets = pd.Series(holding_returns, dtype=float) + aligned = rets.reindex(book.index).fillna(0.0) + gross = float((book * aligned).sum()) + net = gross - self._last_cost + self._last_return = net + return net + + # -- internals -------------------------------------------------------- # + @staticmethod + def _clean_weights(weights: pd.Series | None) -> pd.Series: + """Coerce to a clean float Series; empty/None -> empty book (cash).""" + if weights is None: + return pd.Series(dtype=float) + s = pd.Series(weights, dtype=float) + return s.dropna() diff --git a/runtime/execution.py b/runtime/execution.py new file mode 100644 index 0000000..b17c308 --- /dev/null +++ b/runtime/execution.py @@ -0,0 +1,75 @@ +"""Execution port shared by backtest and live runtimes. + +This is the seam that makes "backtest == live" (CLAUDE.md invariant #2, INV-003): +the same factor/alpha/portfolio code drives either a ``SimExecution`` (backtest) +or a live broker adapter, both implementing this single interface. Only the +adapter changes; the strategy code does not. +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class Execution(ABC): + """Abstract execution port (backtest sim or live broker).""" + + @abstractmethod + def rebalance_to(self, target_weights: pd.Series, date: pd.Timestamp) -> None: + """Move the book toward ``target_weights`` as of ``date``. + + Args: + target_weights: symbol-indexed target weights (sum ~1.0, long-only). + date: the rebalance date. Per the fixed event order, this is the + close of ``date``; the new position is held from the next trading + day. + + A backtest adapter records turnover and trading cost here; a live adapter + would emit orders. Returns None. + """ + raise NotImplementedError + + @abstractmethod + def positions(self) -> pd.Series: + """Return current symbol-indexed weights (after the last rebalance).""" + raise NotImplementedError + + @abstractmethod + def last_return(self) -> float: + """Return the net (after-cost) portfolio return of the last holding period.""" + raise NotImplementedError + + +class BacktestExecution(Execution): + """Backtest-only extension of the :class:`Execution` port. + + A historical simulator needs hooks a live broker does not: it *settles* a + book against per-symbol holding returns and exposes the per-rebalance cost / + turnover it just incurred. Keeping these on a sub-port (instead of the live + ``Execution`` port) preserves "backtest == live" (CLAUDE.md invariant #2 / + INV-003): the live port stays minimal, while the driver — which is a backtest + concern — depends on this richer surface. A live adapter implements only the + base ``Execution`` and is never asked to ``settle``. + """ + + @abstractmethod + def settle(self, holding_returns: pd.Series | None) -> float: + """Settle the current book against per-symbol gross holding returns. + + Returns the net (after-cost) return of the just-held period. + """ + raise NotImplementedError + + @property + @abstractmethod + def last_cost(self) -> float: + """Trading cost (turnover * fee_rate) of the most recent rebalance.""" + raise NotImplementedError + + @property + @abstractmethod + def last_turnover(self) -> float: + """L1 turnover recorded by the most recent rebalance.""" + raise NotImplementedError diff --git a/tests/conftest.py b/tests/conftest.py new file mode 100644 index 0000000..cdea928 --- /dev/null +++ b/tests/conftest.py @@ -0,0 +1,49 @@ +"""Shared pytest fixtures for the framework test-suite. + +Downstream agents: import these by name in your test functions, e.g. +``def test_x(demo_panel): ...``. Do not re-implement the demo data — use these. +""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from tests.fixtures.panel_factory import ( + SYMBOLS, + make_demo_panel, + make_factor_panel, + make_scores, +) + + +@pytest.fixture +def demo_panel() -> pd.DataFrame: + """A normalized market panel (5 symbols, 45 business days).""" + return make_demo_panel() + + +@pytest.fixture +def factor_panel() -> pd.DataFrame: + """A small two-column factor panel aligned to the demo panel index.""" + return make_factor_panel() + + +@pytest.fixture +def demo_symbols() -> list[str]: + """The 5 demo symbols, in order.""" + return list(SYMBOLS) + + +@pytest.fixture +def scores_factory(): + """Callable: ``scores_factory(date) -> pd.Series`` of symbol-indexed scores.""" + return make_scores + + +@pytest.fixture +def example_config_path() -> str: + """Repo-relative path to the example config (resolved from this file).""" + from pathlib import Path + + return str(Path(__file__).resolve().parents[1] / "config" / "example.yaml") diff --git a/tests/fixtures/__init__.py b/tests/fixtures/__init__.py new file mode 100644 index 0000000..d1b3395 --- /dev/null +++ b/tests/fixtures/__init__.py @@ -0,0 +1 @@ +"""Test fixtures package.""" diff --git a/tests/fixtures/panel_factory.py b/tests/fixtures/panel_factory.py new file mode 100644 index 0000000..1ce4b41 --- /dev/null +++ b/tests/fixtures/panel_factory.py @@ -0,0 +1,147 @@ +"""Deterministic test-data factory (backlog section 2). + +Builds a tiny but adversarial market panel that exposes the classic bugs: +rising / falling / flat trends, a mid-series NaN, and an extreme jump. Fully +deterministic: no randomness, no ``datetime.now`` — the same input always +yields byte-identical output, so tests are reproducible. + +Symbols and their patterns: + 000001.SZ close strictly rising -> high momentum + 000002.SZ close strictly falling -> low momentum + 000003.SZ close flat -> ~zero momentum + 000004.SZ one NaN close mid-series -> missing-data handling + 000005.SZ one extreme jump -> later winsorize target +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from data.clean.schema import normalize_panel + +SYMBOLS: list[str] = [ + "000001.SZ", + "000002.SZ", + "000003.SZ", + "000004.SZ", + "000005.SZ", +] +N_DAYS: int = 45 +START_DATE: str = "2024-01-01" +BASE_PRICE: float = 100.0 + +# Index (0-based) where the special events happen. +NAN_DAY: int = 22 # 000004.SZ close is NaN here +JUMP_DAY: int = 30 # 000005.SZ close jumps here +JUMP_MULTIPLIER: float = 3.0 + + +def _trading_dates() -> pd.DatetimeIndex: + """45 business days starting at START_DATE (deterministic, no weekends).""" + return pd.bdate_range(start=START_DATE, periods=N_DAYS) + + +def _close_series_for(symbol: str) -> np.ndarray: + """Deterministic close path per symbol pattern.""" + t = np.arange(N_DAYS, dtype=float) + if symbol == "000001.SZ": + # Strictly rising: +1 per day. + close = BASE_PRICE + t + elif symbol == "000002.SZ": + # Strictly falling: -1 per day (stays positive over 45 days). + close = BASE_PRICE + 50.0 - t + elif symbol == "000003.SZ": + # Flat. + close = np.full(N_DAYS, BASE_PRICE) + elif symbol == "000004.SZ": + # Mildly rising, with one NaN inserted mid-series. + close = BASE_PRICE + 0.5 * t + close[NAN_DAY] = np.nan + elif symbol == "000005.SZ": + # Flat-ish, then a single extreme upward jump that persists. + close = np.full(N_DAYS, BASE_PRICE) + close[JUMP_DAY:] = BASE_PRICE * JUMP_MULTIPLIER + else: # pragma: no cover - defensive + close = np.full(N_DAYS, BASE_PRICE) + return close + + +def make_demo_panel() -> pd.DataFrame: + """Return a NORMALIZED demo market panel (via schema.normalize_panel). + + Shape: MultiIndex(date, symbol), CORE_COLUMNS present, adj_factor == 1.0, + positive volume/amount, open/high/low derived sanely from close. The single + NaN close (000004.SZ @ NAN_DAY) is preserved (NaN cells are legal). + """ + dates = _trading_dates() + rows: list[dict] = [] + for symbol in SYMBOLS: + close = _close_series_for(symbol) + for i, d in enumerate(dates): + c = close[i] + if np.isnan(c): + # Missing close -> whole OHLC row is missing, volume 0. + open_ = high = low = np.nan + volume = 0.0 + amount = 0.0 + else: + open_ = c # simple, deterministic OHLC around close + high = c * 1.01 + low = c * 0.99 + volume = 1_000_000.0 + i * 1_000.0 # positive, controllable + amount = volume * c + rows.append( + { + "date": d, + "symbol": symbol, + "open": open_, + "high": high, + "low": low, + "close": c, + "volume": volume, + "amount": amount, + "adj_factor": 1.0, + } + ) + raw = pd.DataFrame(rows) + return normalize_panel(raw) + + +def make_factor_panel() -> pd.DataFrame: + """Return a small deterministic factor panel for processing/alpha tests. + + Two columns (``momentum_20``, ``volatility_20``) over the same (date, symbol) + index as the demo panel. Values are deterministic placeholders, not the real + factor computation (that belongs to the factors agent); 000004.SZ keeps a NaN + where its close was missing, so NaN-handling paths get exercised. + """ + panel = make_demo_panel() + idx = panel.index + close = panel["close"] + # Deterministic, monotone-ish stand-ins derived from price so downstream + # tests have realistic-shaped numbers without depending on the real factor. + mom = (close / BASE_PRICE - 1.0).rename("momentum_20") + vol = (close.abs() / BASE_PRICE * 0.1).rename("volatility_20") + out = pd.DataFrame({"momentum_20": mom, "volatility_20": vol}, index=idx) + return out + + +def make_scores(date: str | pd.Timestamp) -> pd.Series: + """Return a deterministic symbol-indexed score Series for one date. + + Useful for portfolio tests. Scores rank symbols 000001..000005 ascending + (000005 highest), with 000004 set to NaN to exercise NaN handling. + """ + ts = pd.Timestamp(date).normalize() + values = { + "000001.SZ": 0.1, + "000002.SZ": 0.2, + "000003.SZ": 0.3, + "000004.SZ": np.nan, + "000005.SZ": 0.5, + } + s = pd.Series(values, name="score") + s.index.name = "symbol" + s.attrs["date"] = ts + return s diff --git a/tests/test_adjust.py b/tests/test_adjust.py new file mode 100644 index 0000000..eca00fc --- /dev/null +++ b/tests/test_adjust.py @@ -0,0 +1,90 @@ +"""Tests for front-adjustment (qfq) — the core correctness of P1 real data.""" + +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest + +from data.clean.adjust import front_adjust +from data.clean.schema import normalize_panel + + +def _raw_panel(closes, adj_factors, symbol="000001.SZ"): + """Build a normalized single-symbol raw panel from close + adj_factor lists.""" + n = len(closes) + dates = pd.bdate_range("2024-01-01", periods=n) + close = pd.Series(closes, dtype=float) # real prices are float, not int + df = pd.DataFrame( + { + "date": dates, + "symbol": symbol, + "open": close, + "high": close, + "low": close, + "close": close, + "volume": 1000.0, + "amount": 1000.0, + "adj_factor": pd.Series(adj_factors, dtype=float), + } + ) + return normalize_panel(df) + + +def test_front_adjust_removes_ex_dividend_gap(): + # raw close drops 10 -> 9 across an ex-date (a pure dividend, no real move). + # adj_factor jumps so raw*adj is continuous; older dates have a smaller factor. + raw = _raw_panel([10, 10, 9, 9, 9], [0.9, 0.9, 1.0, 1.0, 1.0]) + qfq = front_adjust(raw) + close = qfq["close"].to_numpy() + # the artificial -10% gap is removed -> qfq close is continuous + assert np.allclose(close, [9, 9, 9, 9, 9]) + # raw saw a fake -10% return across the ex-date ... + rc = raw["close"].to_numpy() + assert np.isclose(rc[2] / rc[1] - 1, -0.1) + # ... qfq smooths it to 0 (correct economic return) + assert np.isclose(close[2] / close[1] - 1, 0.0) + + +def test_front_adjust_anchors_latest_price_unchanged(): + raw = _raw_panel([10, 10, 9, 9, 9], [0.9, 0.9, 1.0, 1.0, 1.0]) + qfq = front_adjust(raw) + # most recent date: anchor ratio = 1, so qfq == raw there + assert np.isclose(qfq["close"].iloc[-1], raw["close"].iloc[-1]) + + +def test_front_adjust_returns_invariant_to_anchor(): + # same economics, adj_factor scaled by 10x -> returns must be identical + q1 = front_adjust(_raw_panel([10, 10, 9, 9, 9], [0.9, 0.9, 1.0, 1.0, 1.0])) + q2 = front_adjust(_raw_panel([10, 10, 9, 9, 9], [9.0, 9.0, 10.0, 10.0, 10.0])) + r1 = q1["close"].pct_change().to_numpy()[1:] + r2 = q2["close"].pct_change().to_numpy()[1:] + assert np.allclose(r1, r2) + + +def test_front_adjust_identity_when_adj_factor_one(): + raw = _raw_panel([10, 11, 12, 13, 14], [1.0] * 5) + pd.testing.assert_frame_equal(front_adjust(raw), raw) + + +def test_front_adjust_requires_adj_factor(): + raw = _raw_panel([10, 11, 12], [1.0] * 3).drop(columns=["adj_factor"]) + with pytest.raises(ValueError, match="adj_factor"): + front_adjust(raw) + + +def test_front_adjust_is_per_symbol(): + a = _raw_panel([10, 10, 9], [0.9, 0.9, 1.0], "000001.SZ") + b = _raw_panel([20, 22, 24], [1.0, 1.0, 1.0], "000002.SZ") + raw = normalize_panel(pd.concat([a, b]).reset_index()) + qfq = front_adjust(raw) + # symbol B (factor 1.0 throughout) is untouched -> no cross-contamination + qb = qfq.xs("000002.SZ", level="symbol")["close"].to_numpy() + assert np.allclose(qb, [20, 22, 24]) + + +def test_front_adjust_does_not_mutate_input(): + raw = _raw_panel([10, 10, 9, 9, 9], [0.9, 0.9, 1.0, 1.0, 1.0]) + before = raw["close"].copy() + front_adjust(raw) + pd.testing.assert_series_equal(raw["close"], before) diff --git a/tests/test_alpha_equal_weight.py b/tests/test_alpha_equal_weight.py new file mode 100644 index 0000000..c1d1717 --- /dev/null +++ b/tests/test_alpha_equal_weight.py @@ -0,0 +1,118 @@ +"""Slice 7 (alpha) tests: EqualWeightAlpha — ALPHA-001/002/003. + +EqualWeightAlpha combines a factor panel into a single score per symbol by a +plain row-wise mean across the factor columns. It is the P0 baseline alpha: +no forward returns, no learned weights. These tests pin: + +- single factor column -> score equals that column (ALPHA-001); +- multiple columns -> equal-weight (row) average (ALPHA-001); +- NaN-by-row behavior -> mean over the available factors, NaN if all missing + (documented; matches pandas skipna mean); +- ``fit`` works with ``forward_returns=None`` (ALPHA-003); +- ``predict`` returns a symbol-indexed ``pd.Series`` (ALPHA-002). +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from alpha.base import AlphaModel +from alpha.equal_weight import EqualWeightAlpha +from tests.fixtures.panel_factory import SYMBOLS + + +def _one_date_factor_panel(values: dict[str, list[float]]) -> pd.DataFrame: + """Build a single-cross-section factor frame: MultiIndex(date, symbol). + + ``values`` maps a factor column name to its per-symbol values (ordered like + ``SYMBOLS``). The single date is fixed and deterministic. + """ + date = pd.Timestamp("2024-02-01") + index = pd.MultiIndex.from_product( + [[date], SYMBOLS], names=["date", "symbol"] + ) + return pd.DataFrame({col: vals for col, vals in values.items()}, index=index) + + +def test_equal_weight_single_factor_returns_factor() -> None: + """One factor column -> the score equals that column, per symbol.""" + factors = _one_date_factor_panel( + {"momentum_20": [0.5, -0.2, 0.0, 0.1, 0.9]} + ) + model = EqualWeightAlpha().fit(factors) + scores = model.predict(factors) + + assert isinstance(scores, pd.Series) + assert list(scores.index) == SYMBOLS + expected = pd.Series( + [0.5, -0.2, 0.0, 0.1, 0.9], index=pd.Index(SYMBOLS, name="symbol") + ) + pd.testing.assert_series_equal( + scores, expected, check_names=False + ) + + +def test_equal_weight_multiple_factors_averages_columns() -> None: + """Multiple columns -> the score is the plain row mean across columns.""" + factors = _one_date_factor_panel( + { + "momentum_20": [0.4, 0.0, -0.2, 1.0, 0.6], + "volatility_20": [0.2, 0.4, 0.2, 0.0, 0.4], + } + ) + model = EqualWeightAlpha().fit(factors, forward_returns=None) + scores = model.predict(factors) + + expected_values = [ + (0.4 + 0.2) / 2, + (0.0 + 0.4) / 2, + (-0.2 + 0.2) / 2, + (1.0 + 0.0) / 2, + (0.6 + 0.4) / 2, + ] + expected = pd.Series( + expected_values, index=pd.Index(SYMBOLS, name="symbol") + ) + pd.testing.assert_series_equal(scores, expected, check_names=False) + + +def test_equal_weight_ignores_nan_by_row() -> None: + """NaN-by-row: mean over available factors; NaN only if all are missing. + + - 000001.SZ has one NaN of two factors -> score == the surviving factor. + - 000004.SZ has all factors NaN -> score is NaN. + """ + factors = _one_date_factor_panel( + { + "momentum_20": [np.nan, 0.2, 0.0, np.nan, 0.6], + "volatility_20": [0.8, 0.4, 0.2, np.nan, 0.4], + } + ) + model = EqualWeightAlpha().fit(factors) + scores = model.predict(factors) + + # Partial NaN -> averages only the present factor. + assert scores["000001.SZ"] == 0.8 + # Both factors present -> plain mean. + assert scores["000002.SZ"] == (0.2 + 0.4) / 2 + # All factors NaN for this symbol -> NaN score. + assert np.isnan(scores["000004.SZ"]) + + +def test_equal_weight_does_not_require_forward_returns() -> None: + """fit must work with no forward returns (P0 needs no future data).""" + factors = _one_date_factor_panel( + {"momentum_20": [0.1, 0.2, 0.3, 0.4, 0.5]} + ) + model = EqualWeightAlpha() + + # Both no-arg and explicit None must return self (for chaining). + assert model.fit(factors) is model + assert model.fit(factors, forward_returns=None) is model + assert isinstance(model, AlphaModel) + + # And predict still produces a symbol-indexed score Series. + scores = model.predict(factors) + assert isinstance(scores, pd.Series) + assert list(scores.index) == SYMBOLS diff --git a/tests/test_analytics_factor.py b/tests/test_analytics_factor.py new file mode 100644 index 0000000..ca4f329 --- /dev/null +++ b/tests/test_analytics_factor.py @@ -0,0 +1,159 @@ +"""Tests for analytics.factor (Slice 11, reqs ANA-001..003). + +Analytics is the ONLY layer allowed to compute forward returns. These tests +cover cross-sectional IC, NaN-pair handling, and quantile-return shape. They use +the shared demo fixtures (no invented data) and never touch real artifacts. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from analytics.factor import ( + compute_ic, + forward_returns, + ic_summary, + quantile_returns, +) +from data.clean.schema import INDEX_NAMES + + +def _toy_panel() -> pd.DataFrame: + """A tiny 2-date cross-section where rank-IC is exactly known. + + Two dates, four symbols. On each date the factor and the forward return are + PERFECTLY monotonically related, so the Spearman cross-sectional IC is +1. + """ + dates = pd.to_datetime(["2024-01-01", "2024-01-02"]) + symbols = ["A", "B", "C", "D"] + idx = pd.MultiIndex.from_product([dates, symbols], names=INDEX_NAMES) + # factor ascending A IC == 1 on each date + assert np.allclose(ic.to_numpy(), 1.0) + + summary = ic_summary(ic) + assert "ic_mean" in summary and "ic_ir" in summary + assert np.isclose(summary["ic_mean"], 1.0) + + +def test_ic_ignores_nan_pairs(): + """NaN factor/return pairs are dropped before correlating, per date. + + We blank out one symbol's factor on one date. The IC for that date must + still be computed from the remaining valid pairs (not NaN), and the other + date is unaffected. + """ + df = _toy_panel() + f = df["f"].copy() + r = df["r"].copy() + # NaN the factor of symbol D on date 1, and the return of symbol A on date 1 + d1 = pd.Timestamp("2024-01-01") + f.loc[(d1, "D")] = np.nan + r.loc[(d1, "A")] = np.nan + + ic = compute_ic(f, r, method="spearman") + # both dates still produce a finite IC + assert not ic.isna().any() + # remaining valid pairs on date 1 are still perfectly monotone -> IC 1.0 + assert np.isclose(ic.loc[d1], 1.0) + + +def test_quantile_returns_shape(): + """quantile_returns returns one mean forward return per (date, quantile). + + Shape contract: a DataFrame whose rows are dates, whose columns are the + quantile buckets 1..q, with mean forward return per cell. Five distinct + symbols across two dates split into 5 quantiles -> 5 columns. + """ + dates = pd.to_datetime(["2024-01-01", "2024-01-02"]) + symbols = ["A", "B", "C", "D", "E"] + idx = pd.MultiIndex.from_product([dates, symbols], names=INDEX_NAMES) + factor = pd.Series([1, 2, 3, 4, 5, 5, 4, 3, 2, 1], index=idx, name="f", dtype=float) + fwd = pd.Series( + [0.1, 0.2, 0.3, 0.4, 0.5, 0.5, 0.4, 0.3, 0.2, 0.1], + index=idx, + name="r", + dtype=float, + ) + + q = quantile_returns(factor, fwd, quantiles=5) + assert isinstance(q, pd.DataFrame) + # rows = dates, columns = 5 quantile buckets + assert q.shape == (2, 5) + assert set(pd.to_datetime(q.index)) == set(dates) + # each bucket holds exactly one symbol per date here -> values are the fwd ret + # top quantile on date 1 is symbol E (factor 5, fwd 0.5) + top_col = q.columns[-1] + assert np.isclose(q.loc[dates[0], top_col], 0.5) + + +def test_forward_returns_uses_future_close_per_symbol(): + """forward_returns is future-looking per symbol (allowed only in analytics). + + For a strictly +1/day symbol, the 1-day forward return at date t equals + close[t+1]/close[t]-1, and the LAST date must be NaN (no future bar). + """ + from tests.fixtures.panel_factory import make_demo_panel + + panel = make_demo_panel() + fwd = forward_returns(panel, periods=(1, 5, 20)) + + assert list(fwd.index.names) == INDEX_NAMES + assert "forward_return_1d" in fwd.columns + assert "forward_return_5d" in fwd.columns + assert "forward_return_20d" in fwd.columns + + rising = fwd.xs("000001.SZ", level="symbol")["forward_return_1d"].dropna() + closes = panel.xs("000001.SZ", level="symbol")["close"] + # rising +1/day: close goes 100,101,...; 1d fwd ret at t = (c[t+1]-c[t])/c[t] + expected_first = (closes.iloc[1] - closes.iloc[0]) / closes.iloc[0] + assert np.isclose(rising.iloc[0], expected_first) + + # last available date for this symbol has no t+1 close -> NaN + last_date = panel.index.get_level_values("date").max() + assert np.isnan(fwd.loc[(last_date, "000001.SZ"), "forward_return_1d"]) + + +def test_forward_returns_never_inf_on_demo_feed(): + """forward_returns contains no +/-inf over the full DemoFeed calendar (MEDIUM-1). + + A non-positive close once produced inf returns (the falling symbol crossed + zero on the long calendar), silently polluting IC / quantile stats. The + falling symbol now stays strictly positive and the divide masks any + non-positive denominator to NaN, so no inf may appear. + """ + from data.feed.demo_feed import DemoFeed + + # The example-config window is long enough that the old linear-decay falling + # symbol would have crossed zero — exactly where inf used to leak in. + panel = DemoFeed(calendar_start="2024-01-01").get_bars( + ["000001.SZ", "000002.SZ", "000003.SZ", "000004.SZ", "000005.SZ"], + "2024-01-01", + "2024-12-31", + ) + fwd = forward_returns(panel, periods=(1, 5, 20)) + values = fwd.to_numpy() + assert not np.isinf(values).any(), "forward_returns leaked +/-inf" + # The falling symbol stays strictly positive (the root-cause fix). + assert (panel.xs("000002.SZ", level="symbol")["close"] > 0).all() diff --git a/tests/test_analytics_performance.py b/tests/test_analytics_performance.py new file mode 100644 index 0000000..934603f --- /dev/null +++ b/tests/test_analytics_performance.py @@ -0,0 +1,73 @@ +"""Tests for analytics.performance (Slice 11, req ANA-004). + +Performance metrics from a nav series: annualized return, max drawdown, +volatility, Sharpe. A hand-computed drawdown series pins the math, and the +summary dict is checked for the required metric keys. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from analytics.performance import ( + annualized_return, + max_drawdown, + performance_summary, +) + + +def test_max_drawdown_known_series(): + """Max drawdown on a hand-computed nav series. + + nav = [1.0, 1.2, 0.6, 0.9, 1.5] + running peak = [1.0, 1.2, 1.2, 1.2, 1.5] + drawdown = [0, 0, -0.5, -0.25, 0] + The worst drawdown is -0.5 (a 50% fall from the 1.2 peak to 0.6). + """ + nav = pd.Series([1.0, 1.2, 0.6, 0.9, 1.5]) + mdd = max_drawdown(nav) + # documented sign convention: returned as a negative fraction + assert np.isclose(mdd, -0.5) + + +def test_performance_summary_contains_required_metrics(): + """performance_summary returns at least the required metric keys. + + Required per ANA-004: annual_return, max_drawdown, volatility, sharpe. + """ + # a gently rising nav over ~one year of business days + dates = pd.bdate_range("2024-01-01", periods=252) + nav = pd.Series(np.linspace(1.0, 1.2, len(dates)), index=dates) + + summary = performance_summary(nav) + assert isinstance(summary, dict) + for key in ("annual_return", "max_drawdown", "volatility", "sharpe"): + assert key in summary, f"missing required metric: {key}" + assert summary[key] is not None + assert np.isfinite(summary[key]) + + # a monotonically rising nav has no drawdown + assert np.isclose(summary["max_drawdown"], 0.0) + # and a positive annualized return + assert summary["annual_return"] > 0.0 + + +def test_monthly_nav_annualizes_at_12_not_252(): + """A 12-row monthly nav must annualize at 12/yr, not the daily 252 (HIGH-1). + + A monthly nav doubling over 12 months is ~+100%/yr. Annualizing the same nav + at 252/yr (the daily default) explodes it absurdly — exactly the headline bug + in the phase0 report. The cadence-correct call gives a finite, sane number. + """ + nav = pd.Series(np.linspace(1.0, 2.0, 13)) # 12 monthly returns, doubles + + monthly = annualized_return(nav, periods_per_year=12) + daily = annualized_return(nav, periods_per_year=252) + + assert np.isfinite(monthly) + # ~doubling over exactly one year of months -> ~+100%, comfortably < 5000%. + assert abs(monthly) < 50.0 + assert 0.5 < monthly < 1.5 + # The (wrong) daily annualization on the same nav explodes far past the band. + assert daily > 50.0 diff --git a/tests/test_backtest_driver.py b/tests/test_backtest_driver.py new file mode 100644 index 0000000..af4f4ea --- /dev/null +++ b/tests/test_backtest_driver.py @@ -0,0 +1,206 @@ +"""Tests for BacktestDriver (backlog Slice 10; reqs BT-001..007). + +The driver wires injected collaborators (universe / scores source / portfolio +constructor / execution / price panel) through the fixed event order: + + compute factor at close of t -> rebalance after close of t -> hold from t+1 + +so the driver MUST settle each rebalance against the NEXT holding period's return +(BT-003), never the same-day return the factor already saw. + +CRITICAL: no other slice is imported. All collaborators are small FAKES defined +here; only the foundation (runtime.execution, fixtures) is imported. +""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from runtime.execution import Execution +from runtime.backtest.driver import BacktestDriver +from runtime.backtest.sim_execution import SimExecution +from tests.fixtures.panel_factory import make_demo_panel + + +# --------------------------------------------------------------------------- # +# Small fakes (stand-ins for universe / scores / constructor collaborators). +# They expose only the Port methods the driver calls. Defined locally because +# the real slices are written in parallel and must not be imported. +# --------------------------------------------------------------------------- # +class FakeUniverse: + """Returns a fixed tradable list regardless of date/panel.""" + + def __init__(self, symbols: list[str]): + self._symbols = list(symbols) + + def members(self, date: pd.Timestamp) -> list[str]: + return list(self._symbols) + + def tradable(self, date: pd.Timestamp, panel: pd.DataFrame) -> list[str]: + return list(self._symbols) + + +class EmptyUniverse: + """Always empty tradable set (exercises the cash path, BT-007).""" + + def members(self, date: pd.Timestamp) -> list[str]: + return [] + + def tradable(self, date: pd.Timestamp, panel: pd.DataFrame) -> list[str]: + return [] + + +class FakeConstructor: + """Equal-weights whatever symbols are passed in via the scores index.""" + + def build(self, scores: pd.Series, current_weights=None) -> pd.Series: + valid = scores.dropna() + if valid.empty: + return pd.Series(dtype=float) + n = len(valid) + return pd.Series(1.0 / n, index=valid.index) + + +class FakeScores: + """Scores source: equal score for the tradable symbols on each date.""" + + def get(self, date: pd.Timestamp, symbols: list[str]) -> pd.Series: + return pd.Series(1.0, index=list(symbols)) + + +def _build_driver(universe, prices=None, fee_rate=0.0, cash_return=0.0): + panel = make_demo_panel() if prices is None else prices + return BacktestDriver( + universe=universe, + scores=FakeScores(), + constructor=FakeConstructor(), + execution=SimExecution(fee_rate=fee_rate), + prices=panel, + rebalance="monthly", + fee_rate=fee_rate, + initial_nav=1.0, + cash_return=cash_return, + ) + + +def test_rebalance_dates_are_monthly(): + """Rebalance dates are the last trading day of each month in the calendar.""" + panel = make_demo_panel() + driver = _build_driver(FakeUniverse(["000001.SZ"]), prices=panel) + dates = driver.rebalance_dates() + # 45 business days from 2024-01-01 reach 2024-03-01 -> 3 month-ends. + months = sorted({(d.year, d.month) for d in dates}) + assert months == [(2024, 1), (2024, 2), (2024, 3)] + # Each rebalance date must be the LAST trading day of its month in the panel. + cal = panel.index.get_level_values("date").unique().sort_values() + for d in dates: + same_month = [c for c in cal if c.year == d.year and c.month == d.month] + assert d == max(same_month) + # All rebalance dates are real trading days in the calendar. + assert set(dates).issubset(set(cal)) + + +def test_backtest_outputs_nav_columns(): + """run() returns a date-indexed frame with the contract columns (BT-005/006).""" + driver = _build_driver(FakeUniverse(["000001.SZ", "000003.SZ"])) + out = driver.run() + assert isinstance(out, pd.DataFrame) + assert list(out.columns) == ["nav", "gross_return", "cost", "turnover", "net_return"] + assert out.index.name == "date" + assert not out.empty + # NAV compounds from initial_nav and is consistent with net_return. + assert (out["nav"] > 0).all() + prev = 1.0 + for _, row in out.iterrows(): + expected = prev * (1.0 + row["net_return"]) + assert abs(row["nav"] - expected) < 1e-9 + prev = row["nav"] + + +def test_backtest_uses_next_period_returns(): + """Holding return is the NEXT period's price change, not the same-day one (BT-003).""" + panel = make_demo_panel() + # 000001.SZ rises strictly +1/day; hold one symbol so gross == its return. + driver = _build_driver(FakeUniverse(["000001.SZ"]), prices=panel, fee_rate=0.0) + out = driver.run() + cal = panel.index.get_level_values("date").unique().sort_values() + close = panel.xs("000001.SZ", level="symbol")["close"] + # For each rebalance row, the gross return must equal the close-to-close + # change over the FORWARD window [rebalance_date -> next rebalance/end], + # never a window that ends on the rebalance date itself. + reb = list(out.index) + for i, d in enumerate(reb): + start_close = close.loc[d] + if i + 1 < len(reb): + end_date = reb[i + 1] + else: + end_date = max(cal) + end_close = close.loc[end_date] + expected = end_close / start_close - 1.0 + assert out.loc[d, "gross_return"] == pytest.approx(expected) + # Sanity: a strictly-rising symbol earns strictly positive gross each period. + assert (out["gross_return"] >= 0).all() + + +def test_backtest_handles_empty_weights(): + """Empty tradable universe -> earn cash_return, no turnover/cost (BT-007).""" + cash = 0.001 + driver = _build_driver(EmptyUniverse(), fee_rate=0.001, cash_return=cash) + out = driver.run() + assert not out.empty + assert (out["turnover"] == 0.0).all() + assert (out["cost"] == 0.0).all() + assert out["gross_return"].eq(cash).all() + assert out["net_return"].eq(cash).all() + # NAV grows by cash_return each period. + prev = 1.0 + for _, row in out.iterrows(): + prev *= 1.0 + cash + assert abs(row["nav"] - prev) < 1e-9 + + +def test_backtest_applies_costs(): + """A non-zero fee_rate reduces NAV versus a zero-fee run (BT-004).""" + universe = FakeUniverse(["000001.SZ", "000002.SZ", "000003.SZ"]) + panel = make_demo_panel() + free = _build_driver(universe, prices=panel, fee_rate=0.0).run() + charged = _build_driver(universe, prices=panel, fee_rate=0.01).run() + # Costs are recorded and positive on rebalances that trade. + assert (charged["cost"] > 0).any() + # Net return is gross minus cost each period. + for d in charged.index: + assert charged.loc[d, "net_return"] == pytest.approx( + charged.loc[d, "gross_return"] - charged.loc[d, "cost"] + ) + # Final NAV with fees is strictly lower than without. + assert charged["nav"].iloc[-1] < free["nav"].iloc[-1] + + +def test_execution_is_execution_port(): + """The injected SimExecution honours the shared Execution port (INV-003).""" + assert isinstance(SimExecution(fee_rate=0.0), Execution) + + +def test_no_zero_length_holding_period_emitted(): + """A final rebalance on the last trading day emits no zero-length row (LOW). + + In the demo calendar the last month-end coincides with the last trading day, + so its holding window (start, end] is empty (end <= start). Emitting it would + add a spurious nav row whose gross_return is a meaningless 0.0. The driver + must skip it, and no remaining row may have a zero-length holding window. + """ + panel = make_demo_panel() + cal = panel.index.get_level_values("date").unique().sort_values() + driver = _build_driver(FakeUniverse(["000001.SZ"]), prices=panel) + + reb_all = driver.rebalance_dates() + out = driver.run() + + # The last rebalance date equals the last trading day -> it is degenerate. + assert reb_all[-1] == cal[-1] + # That degenerate rebalance must NOT appear as a nav row. + assert cal[-1] not in out.index + # 000001.SZ rises strictly, so every EMITTED period has a positive (non-zero) + # gross return -> no zero-length (gross == 0) holding period leaked in. + assert (out["gross_return"] > 0).all() diff --git a/tests/test_bias_audit_report.py b/tests/test_bias_audit_report.py new file mode 100644 index 0000000..9128956 --- /dev/null +++ b/tests/test_bias_audit_report.py @@ -0,0 +1,62 @@ +"""Slice 13: bias-audit report generation tests. + +The bias audit must enumerate every required bias category (未来函数, PIT, +可交易过滤, ann_date, 复权, 交易成本) and record the known P0 downgrades so no +hidden bias slips through (INV-007). The writer is pure markdown; we render to a +``tmp_path`` and never touch the repo-root copy in tests (CONTRACTS §8f). +""" + +from __future__ import annotations + +from qt.reports import ( + bias_audit_required_sections, + render_bias_audit, + write_bias_audit, +) + + +def test_bias_audit_contains_required_sections(tmp_path): + """The bias audit names every required section (未来函数/PIT/.../成本).""" + path = write_bias_audit(tmp_path) + assert path.exists() + text = path.read_text(encoding="utf-8") + + for section in bias_audit_required_sections(): + assert section in text, f"bias audit missing section: {section}" + + # Spot-check the canonical category keywords are present. + for keyword in ("lookahead", "PIT", "可交易过滤", "ann_date", "复权", "交易成本"): + assert keyword in text + + +def test_bias_audit_records_known_phase0_limitations(tmp_path): + """The audit records the P0 downgrades (static universe, daily, adj_factor).""" + text = render_bias_audit() + lowered = text.lower() + + # Static universe is flagged as a PIT downgrade, not a real PIT universe. + assert "staticuniverse" in lowered + assert "降级" in text + # Tradable filter: only missing_close in P0; ST/suspend/limit deferred. + assert "missing_close" in text + # adj_factor retained but full forward-adjust deferred. + assert "adj_factor" in text + # Forward returns confined to analytics (no-lookahead boundary). + assert "analytics" in text + + +def test_bias_audit_discloses_min_listing_days_noop(): + """min_listing_days is disclosed as a configured-but-unenforced no-op (LOW).""" + text = render_bias_audit() + assert "min_listing_days" in text + # Explicitly flagged as a no-op / downgrade, not silently ignored (INV-007). + assert "no-op" in text or "no op" in text.lower() + assert "降级" in text + + +def test_bias_audit_discloses_missing_settlement_price_convention(): + """A held symbol with a NaN end close settles flat (0.0) — disclosed (LOW).""" + text = render_bias_audit() + # The convention (NaN end close -> 0.0 / flat) must be named, not hidden. + assert "结算价缺失" in text or "结算价" in text + assert "NaN" in text diff --git a/tests/test_config.py b/tests/test_config.py new file mode 100644 index 0000000..9c42e18 --- /dev/null +++ b/tests/test_config.py @@ -0,0 +1,93 @@ +"""Slice 1: config system tests (backlog section 4).""" + +from __future__ import annotations + +import textwrap +from pathlib import Path + +import pytest + +from qt.config import ConfigError, RootConfig, load_config + + +def _write(tmp_path: Path, body: str) -> str: + p = tmp_path / "cfg.yaml" + p.write_text(textwrap.dedent(body), encoding="utf-8") + return str(p) + + +_BASE = """\ +project: + name: test +data: + source: {source} + freq: D +{start_line} +{end_line} +universe: + type: static + symbols: ["000001.SZ", "000002.SZ"] +factors: + - name: momentum_20 + enabled: true + params: {{window: 20}} +alpha: + model: equal_weight + params: {{}} +portfolio: + constructor: topn_equal_weight + top_n: {top_n} + long_only: true +backtest: + rebalance: monthly +cost: + fee_rate: 0.001 +output: + root_dir: artifacts +""" + + +def _cfg_text( + source: str = "demo", + start: str | None = "2024-01-01", + end: str | None = "2024-12-31", + top_n: int = 3, +) -> str: + start_line = f' start: "{start}"' if start is not None else "" + end_line = f' end: "{end}"' if end is not None else "" + return _BASE.format( + source=source, start_line=start_line, end_line=end_line, top_n=top_n + ) + + +def test_config_requires_start_and_end(tmp_path: Path) -> None: + path = _write(tmp_path, _cfg_text(start=None, end=None)) + with pytest.raises(ConfigError) as exc: + load_config(path) + msg = str(exc.value).lower() + assert "start" in msg or "end" in msg + + +def test_config_rejects_start_after_end(tmp_path: Path) -> None: + path = _write(tmp_path, _cfg_text(start="2024-12-31", end="2024-01-01")) + with pytest.raises(ConfigError) as exc: + load_config(path) + assert "start" in str(exc.value).lower() + + +def test_config_rejects_invalid_top_n(tmp_path: Path) -> None: + path = _write(tmp_path, _cfg_text(top_n=0)) + with pytest.raises(ConfigError) as exc: + load_config(path) + assert "top_n" in str(exc.value).lower() + + +def test_config_accepts_demo_data_source(tmp_path: Path) -> None: + cfg = load_config(_write(tmp_path, _cfg_text(source="demo"))) + assert isinstance(cfg, RootConfig) + assert cfg.data.source == "demo" + + +def test_config_accepts_tushare_data_source(tmp_path: Path) -> None: + cfg = load_config(_write(tmp_path, _cfg_text(source="tushare"))) + assert cfg.data.source == "tushare" diff --git a/tests/test_config_index.py b/tests/test_config_index.py new file mode 100644 index 0000000..5588245 --- /dev/null +++ b/tests/test_config_index.py @@ -0,0 +1,25 @@ +"""Config support for the PIT index universe (universe.type == 'index').""" + +from __future__ import annotations + +import pytest +from pydantic import ValidationError + +from qt.config import UniverseCfg + + +def test_index_type_requires_index_code(): + with pytest.raises(ValidationError, match="index_code"): + UniverseCfg(type="index") + + +def test_index_type_with_code_is_valid(): + cfg = UniverseCfg(type="index", index_code="000300.SH") + assert cfg.type == "index" + assert cfg.index_code == "000300.SH" + + +def test_static_type_is_default_and_needs_no_index_code(): + cfg = UniverseCfg(symbols=["000001.SZ"]) + assert cfg.type == "static" + assert cfg.index_code is None diff --git a/tests/test_covariates_enrich.py b/tests/test_covariates_enrich.py new file mode 100644 index 0000000..598db8c --- /dev/null +++ b/tests/test_covariates_enrich.py @@ -0,0 +1,43 @@ +"""Tests for industry + market-cap covariate enrichment.""" + +from __future__ import annotations + +import pandas as pd + +from data.clean.covariates import enrich_covariates +from data.clean.schema import normalize_panel + + +def _panel(): + dates = pd.bdate_range("2024-03-01", periods=2) + rows = [] + for d in dates: + for sym, close in [("000001.SZ", 10.0), ("000002.SZ", 20.0)]: + rows.append( + {"date": d, "symbol": sym, "open": close, "high": close, "low": close, + "close": close, "volume": 1.0, "amount": 1.0, "adj_factor": 1.0} + ) + return normalize_panel(pd.DataFrame(rows)) + + +def test_enrich_industry_broadcast_per_symbol(): + out = enrich_covariates(_panel(), industry={"000001.SZ": "银行", "000002.SZ": "地产"}) + assert out.xs("000001.SZ", level="symbol")["industry"].eq("银行").all() + assert out.xs("000002.SZ", level="symbol")["industry"].eq("地产").all() + + +def test_enrich_market_cap_join_by_date_symbol(): + d = pd.Timestamp("2024-03-01") + mc = pd.DataFrame( + {"date": [d, d], "symbol": ["000001.SZ", "000002.SZ"], "market_cap": [100.0, 200.0]} + ) + out = enrich_covariates(_panel(), market_cap=mc) + assert out.loc[(d, "000001.SZ"), "market_cap"] == 100.0 + assert out.loc[(d, "000002.SZ"), "market_cap"] == 200.0 + + +def test_enrich_does_not_mutate_input(): + panel = _panel() + before = set(panel.columns) + enrich_covariates(panel, industry={}) + assert set(panel.columns) == before diff --git a/tests/test_data_feed.py b/tests/test_data_feed.py new file mode 100644 index 0000000..4b4a83c --- /dev/null +++ b/tests/test_data_feed.py @@ -0,0 +1,210 @@ +"""Slice 3 tests: DemoFeed + TushareFeed boundary (DATA-001/002/004, SEC-001/004). + +No real network. The tushare SDK is monkeypatched; a FAKE tmp config json with a +FAKE token is used everywhere — the real /home/shaofl/.../.config.json is NEVER +read. We assert the token comes from the external config and never leaks to +stdout or logging. +""" + +from __future__ import annotations + +import io +import json +import logging +from contextlib import redirect_stdout + +import pandas as pd +import pytest + +from data.clean.schema import CORE_COLUMNS, validate_panel +from data.feed.base import DataFeed +from data.feed.demo_feed import DemoFeed +from data.feed.tushare_feed import TushareFeed +from tests.fixtures.panel_factory import SYMBOLS + +FAKE_TOKEN = "FAKE_TUSHARE_TOKEN_do_not_leak_0123456789abcdef" + + +# --------------------------------------------------------------------------- # +# DemoFeed +# --------------------------------------------------------------------------- # +def test_demo_feed_returns_standard_panel(): + feed = DemoFeed() + assert isinstance(feed, DataFeed) + + symbols = SYMBOLS[:3] + start, end = "2024-01-03", "2024-01-10" + panel = feed.get_bars(symbols, start, end, freq="D") + + # Canonical shape: must pass the foundation validator unchanged. + validate_panel(panel) + for col in CORE_COLUMNS: + assert col in panel.columns + + # Honors the requested symbol list (no extras, no missing). + got_symbols = set(panel.index.get_level_values("symbol").unique()) + assert got_symbols == set(symbols) + + # Honors the [start, end] inclusive date range. + dates = panel.index.get_level_values("date") + assert dates.min() >= pd.Timestamp(start) + assert dates.max() <= pd.Timestamp(end) + + # adj_factor fixed at 1.0 (DATA-003: at least preserve adj_factor). + assert (panel["adj_factor"] == 1.0).all() + + # Deterministic: same call twice -> identical frame (no randomness, no now()). + panel2 = feed.get_bars(symbols, start, end, freq="D") + pd.testing.assert_frame_equal(panel, panel2) + + +# --------------------------------------------------------------------------- # +# TushareFeed — token sourcing +# --------------------------------------------------------------------------- # +def _write_fake_config(tmp_path, token: str = FAKE_TOKEN): + """Write a fake nested config json mimicking the real .config.json layout.""" + cfg = {"tushare": {"token": token}, "other": {"unused": 1}} + path = tmp_path / "fake_config.json" + path.write_text(json.dumps(cfg), encoding="utf-8") + return path + + +def _tushare_style_frame(symbol: str) -> pd.DataFrame: + """A raw tushare `daily` style frame (English-but-non-canonical columns).""" + return pd.DataFrame( + { + "ts_code": [symbol, symbol], + "trade_date": ["20240103", "20240104"], + "open": [10.0, 10.5], + "high": [10.8, 11.0], + "low": [9.9, 10.4], + "close": [10.6, 10.9], + "vol": [123456.0, 234567.0], # tushare uses 'vol', not 'volume' + "amount": [1_300_000.0, 2_550_000.0], + } + ) + + +def test_tushare_feed_reads_token_from_external_config(tmp_path, monkeypatch): + cfg_path = _write_fake_config(tmp_path) + + captured: dict[str, object] = {} + + class FakePro: + def daily(self, **kwargs): + return _tushare_style_frame(kwargs.get("ts_code", "000001.SZ")) + + def adj_factor(self, **kwargs): + return pd.DataFrame( + { + "ts_code": [kwargs.get("ts_code", "000001.SZ")] * 2, + "trade_date": ["20240103", "20240104"], + "adj_factor": [1.0, 1.0], + } + ) + + def fake_pro_api(token=None): + captured["token"] = token + return FakePro() + + import tushare as ts + + monkeypatch.setattr(ts, "pro_api", fake_pro_api) + + feed = TushareFeed(secret_file=str(cfg_path), token_key="tushare.token") + # Trigger lazy client construction. + feed.get_bars(["000001.SZ"], "2024-01-03", "2024-01-04") + + assert captured["token"] == FAKE_TOKEN + + +def test_tushare_feed_does_not_log_token(tmp_path, monkeypatch, caplog): + cfg_path = _write_fake_config(tmp_path) + + class FakePro: + def daily(self, **kwargs): + return _tushare_style_frame(kwargs.get("ts_code", "000001.SZ")) + + def adj_factor(self, **kwargs): + return pd.DataFrame( + { + "ts_code": [kwargs.get("ts_code", "000001.SZ")] * 2, + "trade_date": ["20240103", "20240104"], + "adj_factor": [1.0, 1.0], + } + ) + + import tushare as ts + + monkeypatch.setattr(ts, "pro_api", lambda token=None: FakePro()) + + stdout = io.StringIO() + with caplog.at_level(logging.DEBUG), redirect_stdout(stdout): + feed = TushareFeed(secret_file=str(cfg_path)) + feed.get_bars(["000001.SZ"], "2024-01-03", "2024-01-04") + # Force the feed's own repr to be safe too. + repr(feed) + + assert FAKE_TOKEN not in stdout.getvalue() + assert FAKE_TOKEN not in caplog.text + for record in caplog.records: + assert FAKE_TOKEN not in record.getMessage() + + +def test_tushare_feed_normalizes_columns(tmp_path, monkeypatch): + cfg_path = _write_fake_config(tmp_path) + + class FakePro: + def daily(self, **kwargs): + return _tushare_style_frame(kwargs.get("ts_code", "000001.SZ")) + + def adj_factor(self, **kwargs): + return pd.DataFrame( + { + "ts_code": [kwargs.get("ts_code", "000001.SZ")] * 2, + "trade_date": ["20240103", "20240104"], + "adj_factor": [1.0, 1.0], + } + ) + + import tushare as ts + + monkeypatch.setattr(ts, "pro_api", lambda token=None: FakePro()) + + feed = TushareFeed(secret_file=str(cfg_path)) + panel = feed.get_bars(["000001.SZ"], "2024-01-03", "2024-01-04") + + # Output is canonical: passes the foundation validator and has CORE_COLUMNS. + validate_panel(panel) + for col in CORE_COLUMNS: + assert col in panel.columns + + # tushare 'ts_code'/'trade_date'/'vol' were mapped to symbol/date/volume. + assert "volume" in panel.columns + assert "vol" not in panel.columns + assert "ts_code" not in panel.columns + assert "trade_date" not in panel.columns + assert list(panel.index.names) == ["date", "symbol"] + assert set(panel.index.get_level_values("symbol").unique()) == {"000001.SZ"} + + # vol values landed in volume; first row close mapped correctly. + first = panel.xs("000001.SZ", level="symbol").iloc[0] + assert first["volume"] == 123456.0 + assert first["close"] == 10.6 + + +def test_tushare_feed_missing_token_key_raises_readable_error(tmp_path, monkeypatch): + # token_key points at a nested key that does not exist -> readable error. + cfg_path = _write_fake_config(tmp_path) + feed = TushareFeed(secret_file=str(cfg_path), token_key="tushare.nope") + + import tushare as ts + + monkeypatch.setattr(ts, "pro_api", lambda token=None: object()) + + with pytest.raises(ValueError) as exc: + feed.get_bars(["000001.SZ"], "2024-01-03", "2024-01-04") + msg = str(exc.value) + assert "tushare.nope" in msg + # The error message must never leak the real token contents. + assert FAKE_TOKEN not in msg diff --git a/tests/test_data_schema.py b/tests/test_data_schema.py new file mode 100644 index 0000000..e50db0f --- /dev/null +++ b/tests/test_data_schema.py @@ -0,0 +1,73 @@ +"""Slice 2: panel schema tests (backlog section 5).""" + +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest + +from data.clean.schema import CORE_COLUMNS, INDEX_NAMES, normalize_panel, validate_panel + + +def _raw_rows() -> pd.DataFrame: + """Two symbols, two dates, deliberately out of order.""" + rows = [ + ("2024-01-02", "000002.SZ"), + ("2024-01-01", "000002.SZ"), + ("2024-01-02", "000001.SZ"), + ("2024-01-01", "000001.SZ"), + ] + data = [] + for d, s in rows: + rec = {"date": d, "symbol": s} + for c in CORE_COLUMNS: + rec[c] = 1.0 + data.append(rec) + return pd.DataFrame(data) + + +def test_normalize_panel_creates_multiindex() -> None: + panel = normalize_panel(_raw_rows()) + assert isinstance(panel.index, pd.MultiIndex) + assert list(panel.index.names) == INDEX_NAMES + date_level = panel.index.get_level_values("date") + assert pd.api.types.is_datetime64_any_dtype(date_level) + assert all(isinstance(s, str) for s in panel.index.get_level_values("symbol")) + # No exceptions from the full validator either. + validate_panel(panel) + + +def test_normalize_panel_sorts_index() -> None: + panel = normalize_panel(_raw_rows()) + assert panel.index.is_monotonic_increasing + first = panel.index[0] + assert first == (pd.Timestamp("2024-01-01"), "000001.SZ") + + +def test_normalize_panel_rejects_duplicate_index() -> None: + raw = _raw_rows() + dup = pd.concat([raw, raw.iloc[[0]]], ignore_index=True) + with pytest.raises(ValueError) as exc: + normalize_panel(dup) + assert "duplicate" in str(exc.value).lower() + + +def test_normalize_panel_requires_core_columns() -> None: + raw = _raw_rows().drop(columns=["close"]) + with pytest.raises(ValueError) as exc: + normalize_panel(raw) + assert "close" in str(exc.value).lower() + + +def test_normalize_panel_allows_nan_cells() -> None: + raw = _raw_rows() + raw.loc[0, "close"] = np.nan + panel = normalize_panel(raw) # must not raise + assert panel["close"].isna().any() + + +def test_normalize_panel_accepts_already_indexed() -> None: + panel = normalize_panel(_raw_rows()) + # Round-trip: feeding a normalized panel back in is idempotent. + again = normalize_panel(panel) + pd.testing.assert_frame_equal(panel, again) diff --git a/tests/test_factor_processing.py b/tests/test_factor_processing.py new file mode 100644 index 0000000..a9e626f --- /dev/null +++ b/tests/test_factor_processing.py @@ -0,0 +1,112 @@ +"""Slice 6: factor preprocessing pipeline (PROC-001..004). + +Cross-sectional, by-date processing: drop_missing + z-score standardization. +Zero-variance / single-name cross-sections must not crash. Output keeps the +canonical MultiIndex(date, symbol). +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from data.clean.schema import INDEX_NAMES +from factors.process.pipeline import ProcessingPipeline + + +def _factor_frame(rows: list[tuple[str, str, float]]) -> pd.DataFrame: + """Build a MultiIndex(date, symbol) factor frame from (date, symbol, value).""" + dates = [pd.Timestamp(d) for d, _, _ in rows] + symbols = [s for _, s, _ in rows] + values = [v for _, _, v in rows] + idx = pd.MultiIndex.from_arrays([dates, symbols], names=INDEX_NAMES) + return pd.DataFrame({"f": values}, index=idx) + + +def test_zscore_is_applied_by_date() -> None: + # Two dates, each its own cross-section. Day 1 values {1,2,3}; day 2 {10,20,30}. + # Z-score is computed independently per date, so both days map to the same + # standardized triple even though their raw scales differ. + frame = _factor_frame( + [ + ("2024-01-01", "000001.SZ", 1.0), + ("2024-01-01", "000002.SZ", 2.0), + ("2024-01-01", "000003.SZ", 3.0), + ("2024-01-02", "000001.SZ", 10.0), + ("2024-01-02", "000002.SZ", 20.0), + ("2024-01-02", "000003.SZ", 30.0), + ] + ) + out = ProcessingPipeline().transform(frame) + + for date in frame.index.get_level_values("date").unique(): + cross = out.xs(date, level="date")["f"] + # Population std (ddof=0) -> mean 0, std 1 per date. + assert cross.mean() == pytest_approx(0.0) + assert cross.std(ddof=0) == pytest_approx(1.0) + + day1 = out.xs(pd.Timestamp("2024-01-01"), level="date")["f"].to_numpy() + day2 = out.xs(pd.Timestamp("2024-01-02"), level="date")["f"].to_numpy() + # Identical standardized shape across the two differently-scaled days. + np.testing.assert_allclose(day1, day2) + + +def test_zscore_ignores_nan() -> None: + # One name is NaN on this date. drop_missing removes it before standardizing, + # so the surviving {2,4} cross-section standardizes on its own mean/std and + # the NaN row does not appear in the output for that date. + frame = _factor_frame( + [ + ("2024-01-01", "000001.SZ", 2.0), + ("2024-01-01", "000002.SZ", np.nan), + ("2024-01-01", "000003.SZ", 4.0), + ] + ) + out = ProcessingPipeline().transform(frame) + cross = out.xs(pd.Timestamp("2024-01-01"), level="date")["f"] + + assert "000002.SZ" not in cross.index + assert len(cross) == 2 + assert cross.mean() == pytest_approx(0.0) + assert cross.std(ddof=0) == pytest_approx(1.0) + + +def test_zscore_handles_zero_std() -> None: + # Zero-variance date (all equal) and single-name date must NOT raise; the + # documented rule is they map to 0.0 (de-meaned, no scaling). + frame = _factor_frame( + [ + ("2024-01-01", "000001.SZ", 5.0), + ("2024-01-01", "000002.SZ", 5.0), + ("2024-01-01", "000003.SZ", 5.0), + ("2024-01-02", "000001.SZ", 7.0), # single-name cross-section + ] + ) + out = ProcessingPipeline().transform(frame) + + zero_var = out.xs(pd.Timestamp("2024-01-01"), level="date")["f"] + assert (zero_var == 0.0).all() + + single = out.xs(pd.Timestamp("2024-01-02"), level="date")["f"] + assert (single == 0.0).all() + + +def test_processor_keeps_multiindex(factor_panel: pd.DataFrame) -> None: + # Output must keep the canonical MultiIndex(date, symbol) and the same + # factor columns; input must not be mutated. + before = factor_panel.copy(deep=True) + out = ProcessingPipeline().transform(factor_panel) + + assert isinstance(out.index, pd.MultiIndex) + assert list(out.index.names) == INDEX_NAMES + assert list(out.columns) == list(factor_panel.columns) + assert out.index.is_monotonic_increasing + # Immutability: the input frame is untouched. + pd.testing.assert_frame_equal(factor_panel, before) + + +def pytest_approx(value: float): + """Local tolerance helper (avoids importing pytest at module top).""" + import pytest + + return pytest.approx(value, abs=1e-9) diff --git a/tests/test_factors_financial.py b/tests/test_factors_financial.py new file mode 100644 index 0000000..fe0fa72 --- /dev/null +++ b/tests/test_factors_financial.py @@ -0,0 +1,43 @@ +"""FinancialFactor surfaces a PIT-aligned financial column as a factor.""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from data.clean.schema import normalize_panel +from factors.compute.financial import FinancialFactor + + +def _panel_with(field, values=(3.1, 5.5)): + dates = pd.bdate_range("2024-04-22", periods=2) + rows = [] + for d in dates: + for sym, val in zip(("000001.SZ", "000002.SZ"), values): + rows.append( + { + "date": d, "symbol": sym, + "open": 1.0, "high": 1.0, "low": 1.0, "close": 1.0, + "volume": 1.0, "amount": 1.0, "adj_factor": 1.0, + field: val, + } + ) + return normalize_panel(pd.DataFrame(rows)) + + +def test_financial_factor_surfaces_column(): + series = FinancialFactor("roe").compute(_panel_with("roe")) + assert series.name == "roe" + assert series.xs("000001.SZ", level="symbol").iloc[0] == 3.1 + + +def test_financial_factor_raises_when_column_absent(): + factor = FinancialFactor("roe") + panel = _panel_with("netprofit_yoy") # has no 'roe' column + with pytest.raises(ValueError, match="roe"): + factor.compute(panel) + + +def test_financial_factor_rejects_unsupported_field(): + with pytest.raises(ValueError, match="not supported"): + FinancialFactor("pe_ttm") diff --git a/tests/test_factors_momentum.py b/tests/test_factors_momentum.py new file mode 100644 index 0000000..89b43f6 --- /dev/null +++ b/tests/test_factors_momentum.py @@ -0,0 +1,144 @@ +"""Tests for the momentum_20 cross-sectional factor (Slice 5; FAC-001..004). + +The no-lookahead guarantee (INV-001 / CLAUDE.md invariant #1) is the load-bearing +property: a factor value at date t may use only bars at dates <= t. These tests +also pin the FIXED formula and event order from CONTRACTS.md s6: + + momentum_20[t] = close[t] / close[t - window] - 1 (window default = 20) + +Per-symbol grouping must prevent cross-symbol leakage, and early dates with an +insufficient window must yield NaN. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from factors.compute.momentum import MomentumFactor +from tests.fixtures.panel_factory import make_demo_panel + + +def _series_for(factor: pd.Series, symbol: str) -> pd.Series: + """Slice a (date, symbol)-indexed factor Series down to one symbol, by date.""" + return factor.xs(symbol, level="symbol") + + +def test_momentum_output_index_matches_panel(demo_panel: pd.DataFrame) -> None: + """Output is a MultiIndex(date, symbol) Series aligned 1:1 to the panel.""" + factor = MomentumFactor().compute(demo_panel) + + assert isinstance(factor, pd.Series) + assert factor.name == "momentum_20" + assert list(factor.index.names) == ["date", "symbol"] + # Same index, same order, same length as the source panel. + pd.testing.assert_index_equal(factor.index, demo_panel.index) + + +def test_momentum_window_not_enough_returns_nan(demo_panel: pd.DataFrame) -> None: + """The first ``window`` dates of every symbol are NaN; the next is finite.""" + window = 20 + factor = MomentumFactor(window=window).compute(demo_panel) + dates = demo_panel.index.get_level_values("date").unique().sort_values() + + rising = _series_for(factor, "000001.SZ").reindex(dates) + # Dates 0..window-1 have no close[t-window] -> NaN. + assert rising.iloc[:window].isna().all() + # The first date with a full window (index == window) is finite. + assert np.isfinite(rising.iloc[window]) + + +def test_momentum_computed_per_symbol(demo_panel: pd.DataFrame) -> None: + """Rising symbol is high, falling is low (negative), flat is ~0. + + Also a direct cross-symbol-leakage guard: computing the factor on the full + panel must equal computing it on each symbol's sub-panel in isolation. + """ + factor = MomentumFactor(window=20).compute(demo_panel) + dates = demo_panel.index.get_level_values("date").unique().sort_values() + + rising = _series_for(factor, "000001.SZ").reindex(dates).iloc[20] + falling = _series_for(factor, "000002.SZ").reindex(dates).iloc[20] + flat = _series_for(factor, "000003.SZ").reindex(dates).iloc[20] + + assert rising > 0.0 + assert falling < 0.0 + assert flat == 0.0 + assert rising > flat > falling + + # No cross-symbol leakage: per-symbol isolated compute matches joint compute. + for symbol in ["000001.SZ", "000002.SZ", "000003.SZ"]: + sub = demo_panel.xs(symbol, level="symbol", drop_level=False) + isolated = MomentumFactor(window=20).compute(sub) + joint = factor.loc[(slice(None), symbol)] + pd.testing.assert_series_equal( + isolated.reset_index(drop=True), + joint.reset_index(drop=True), + check_names=False, + ) + + +def test_momentum_has_no_lookahead(demo_panel: pd.DataFrame) -> None: + """Mutating a FUTURE close leaves every earlier-date factor value unchanged. + + This is the core INV-001 guarantee: the value at date t depends only on bars + at dates <= t, so perturbing a later bar cannot ripple backwards. + """ + window = 20 + base = MomentumFactor(window=window).compute(demo_panel) + dates = demo_panel.index.get_level_values("date").unique().sort_values() + + # Mutate close at the LAST date for the rising symbol (a strictly future bar + # relative to all earlier dates). Build a fresh panel; never mutate input. + future_date = dates[-1] + symbol = "000001.SZ" + mutated = demo_panel.copy(deep=True) + mutated.loc[(future_date, symbol), "close"] = 9_999.0 + + perturbed = MomentumFactor(window=window).compute(mutated) + + # Every value strictly before the mutated date must be byte-for-byte equal. + base_earlier = base[base.index.get_level_values("date") < future_date] + pert_earlier = perturbed[perturbed.index.get_level_values("date") < future_date] + pd.testing.assert_series_equal(base_earlier, pert_earlier) + + # Sanity: the mutation DID change the factor at/after the mutated date, + # otherwise the test would pass vacuously. + changed = perturbed[(future_date, symbol)] + assert changed != base[(future_date, symbol)] + + +def test_momentum_uses_previous_window() -> None: + """Exact numeric formula check on the strictly-rising 000001.SZ. + + close = 100 + t, so momentum_20[t] = (100 + t) / (100 + t - 20) - 1. + """ + window = 20 + panel = make_demo_panel() + factor = MomentumFactor(window=window).compute(panel) + dates = panel.index.get_level_values("date").unique().sort_values() + rising_close = panel["close"].xs("000001.SZ", level="symbol").reindex(dates) + rising_mom = factor.xs("000001.SZ", level="symbol").reindex(dates) + + for t in range(window, len(dates)): + expected = rising_close.iloc[t] / rising_close.iloc[t - window] - 1.0 + assert rising_mom.iloc[t] == expected + + # Pin one concrete value: close[20]/close[0]-1 = 120/100 - 1 = 0.2. + assert rising_mom.iloc[window] == 120.0 / 100.0 - 1.0 + + +def test_momentum_name_is_window_aware() -> None: + """A non-default window labels the factor ``momentum_`` (MEDIUM-2). + + The class attribute stays the canonical default; the instance name must + track the actual window so a window=10 config does not mislabel its column. + """ + assert MomentumFactor.name == "momentum_20" # class default unchanged + assert MomentumFactor(window=20).name == "momentum_20" + assert MomentumFactor(window=10).name == "momentum_10" + + # The computed series is named by the (window-aware) instance name. + panel = make_demo_panel() + series = MomentumFactor(window=10).compute(panel) + assert series.name == "momentum_10" diff --git a/tests/test_financial_pipeline.py b/tests/test_financial_pipeline.py new file mode 100644 index 0000000..07cceb6 --- /dev/null +++ b/tests/test_financial_pipeline.py @@ -0,0 +1,80 @@ +"""Pipeline dispatch for financial factors + the demo-source guard (no fabrication).""" + +from __future__ import annotations + +import logging + +import pandas as pd +import pytest +import yaml + +from factors.compute.financial import FinancialFactor +from factors.compute.momentum import MomentumFactor +from qt.config import load_config +from qt.pipeline import _build_factor, _maybe_enrich_financials + + +def _cfg(tmp_path, example_config_path, factor_name, source): + base = yaml.safe_load(open(example_config_path, encoding="utf-8")) + base["factors"] = [{"name": factor_name, "enabled": True, "params": {}}] + base["data"]["source"] = source + path = tmp_path / "cfg.yaml" + path.write_text(yaml.safe_dump(base), encoding="utf-8") + return load_config(str(path)) + + +def test_build_factor_dispatches_financial(tmp_path, example_config_path): + cfg = _cfg(tmp_path, example_config_path, "roe", "tushare") + assert isinstance(_build_factor(cfg), FinancialFactor) + + +def test_build_factor_dispatches_momentum(tmp_path, example_config_path): + cfg = _cfg(tmp_path, example_config_path, "momentum_20", "demo") + assert isinstance(_build_factor(cfg), MomentumFactor) + + +def test_financial_factor_on_demo_source_raises(tmp_path, example_config_path): + cfg = _cfg(tmp_path, example_config_path, "roe", "demo") + factor = _build_factor(cfg) + with pytest.raises(ValueError, match="cannot run on demo"): + _maybe_enrich_financials( + cfg, pd.DataFrame(), ["000001.SZ"], factor, logging.getLogger("test") + ) + + +def test_unknown_factor_name_raises(tmp_path, example_config_path): + cfg = _cfg(tmp_path, example_config_path, "totally_made_up", "demo") + with pytest.raises(ValueError, match="Unknown factor"): + _build_factor(cfg) + + +def test_financial_fetch_uses_lookback_before_start(tmp_path, example_config_path, monkeypatch): + # the financial fetch must reach BEFORE data.start so the prior disclosed + # report can be carried forward onto early trade dates. + from qt.pipeline import _FINANCIAL_LOOKBACK_DAYS, _maybe_enrich_financials + + cfg = _cfg(tmp_path, example_config_path, "roe", "tushare") + captured = {} + + class _FakeFeed: + def __init__(self, *a, **k): + pass + + def get_fina_indicator(self, symbols, start, end, fields=None): + captured["start"] = start + return pd.DataFrame( + {"symbol": [], "ann_date": [], "end_date": [], "roe": []} + ) + + monkeypatch.setattr("qt.pipeline.TushareFinancialFeed", _FakeFeed) + idx = pd.MultiIndex.from_product( + [pd.to_datetime([cfg.data.start]), ["000001.SZ"]], names=["date", "symbol"] + ) + panel = pd.DataFrame({"close": [1.0]}, index=idx) + _maybe_enrich_financials( + cfg, panel, ["000001.SZ"], FinancialFactor("roe"), logging.getLogger("test") + ) + fetched = pd.Timestamp(captured["start"]) + assert fetched <= pd.Timestamp(cfg.data.start) - pd.Timedelta( + days=_FINANCIAL_LOOKBACK_DAYS - 1 + ) diff --git a/tests/test_index_feed.py b/tests/test_index_feed.py new file mode 100644 index 0000000..d9f723e --- /dev/null +++ b/tests/test_index_feed.py @@ -0,0 +1,84 @@ +"""IndexConstituentsFeed mapping tests — no network, fake SDK.""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.index_feed import IndexConstituentsFeed + + +class _FakePro: + """Stand-in for the tushare pro client returning an index_weight frame.""" + + def index_weight(self, index_code, start_date, end_date): # noqa: ARG002 + return pd.DataFrame( + { + "index_code": ["000300.SH"] * 3, + "con_code": ["000002.SZ", "000001.SZ", "000001.SZ"], + "trade_date": ["20240131", "20240131", "20240229"], + "weight": [1.5, 2.5, 2.0], + } + ) + + +def _feed(monkeypatch): + feed = IndexConstituentsFeed("fake.json") + monkeypatch.setattr(feed, "_client", lambda: _FakePro()) + return feed + + +def test_get_constituents_maps_to_canonical_columns(monkeypatch): + feed = _feed(monkeypatch) + out = feed.get_constituents("000300.SH", "2024-01-01", "2024-02-29") + assert list(out.columns) == ["date", "symbol", "weight"] + assert out["symbol"].dtype == object + assert str(out["date"].dtype).startswith("datetime64") + + +def test_get_constituents_sorted_by_date_symbol(monkeypatch): + feed = _feed(monkeypatch) + out = feed.get_constituents("000300.SH", "2024-01-01", "2024-02-29") + pairs = list(zip(out["date"], out["symbol"])) + assert pairs == sorted(pairs) + # 20240131 cross-section: both symbols present, ascending + jan = out[out["date"] == pd.Timestamp("2024-01-31")]["symbol"].tolist() + assert jan == ["000001.SZ", "000002.SZ"] + + +def test_get_constituents_pages_long_window(monkeypatch): + # A >90-day window must be split into multiple index_weight calls and the + # snapshots concatenated, so the tushare ~6000-row cap can't drop early dates. + class _PagingPro: + def __init__(self): + self.calls = [] + + def index_weight(self, index_code, start_date, end_date): # noqa: ARG002 + self.calls.append((start_date, end_date)) + return pd.DataFrame( + { + "index_code": ["000300.SH"], + "con_code": ["000001.SZ"], + "trade_date": [start_date], # one snapshot per window + "weight": [1.0], + } + ) + + feed = IndexConstituentsFeed("fake.json") + pro = _PagingPro() + monkeypatch.setattr(feed, "_client", lambda: pro) + out = feed.get_constituents("000300.SH", "2024-01-01", "2024-06-30") + assert len(pro.calls) >= 2 # window was paged + assert out["date"].nunique() >= 2 # snapshots from multiple windows kept + + +def test_get_constituents_empty_result_is_schema_shaped(monkeypatch): + feed = IndexConstituentsFeed("fake.json") + + class _Empty: + def index_weight(self, **_kw): + return pd.DataFrame() + + monkeypatch.setattr(feed, "_client", lambda: _Empty()) + out = feed.get_constituents("000300.SH", "2024-01-01", "2024-02-29") + assert list(out.columns) == ["date", "symbol", "weight"] + assert len(out) == 0 diff --git a/tests/test_index_pipeline.py b/tests/test_index_pipeline.py new file mode 100644 index 0000000..b8e8101 --- /dev/null +++ b/tests/test_index_pipeline.py @@ -0,0 +1,79 @@ +"""Pipeline wiring for PIT index universe. + +These tests focus on the pipeline boundary, not tushare itself. The feed is +monkeypatched so no network or token is touched. +""" + +from __future__ import annotations + +import pandas as pd + +import qt.pipeline as pipeline +from qt.config import RootConfig + + +class _FakeLogger: + def info(self, *_args, **_kwargs): + return None + + +def _index_cfg() -> RootConfig: + return RootConfig( + data={ + "source": "tushare", + "freq": "D", + "start": "2024-02-15", + "end": "2024-03-31", + "external_secret_file": "/tmp/fake.json", + "tushare_token_key": "tushare.token", + "output_name": "daily", + }, + universe={ + "type": "index", + "index_code": "000300.SH", + "symbols": [], + "filters": {"missing_close": True}, + }, + factors=[{"name": "momentum_20", "enabled": True, "params": {"window": 20}}], + alpha={"model": "equal_weight", "params": {}}, + portfolio={"constructor": "topn_equal_weight", "top_n": 3}, + backtest={"rebalance": "monthly"}, + cost={"fee_rate": 0.001}, + output={ + "root_dir": "artifacts", + "data_dir": "artifacts/data", + "factor_dir": "artifacts/factors", + "report_dir": "artifacts/reports", + "log_dir": "artifacts/logs", + }, + ) + + +def test_index_universe_fetches_pre_start_snapshot_for_asof(monkeypatch): + """Pipeline must request pre-start snapshots so first as-of date is valid.""" + calls: list[tuple[str, str, str]] = [] + + class _FakeIndexFeed: + def __init__(self, secret_file, token_key="tushare.token"): + self.secret_file = secret_file + self.token_key = token_key + + def get_constituents(self, index_code: str, start: str, end: str) -> pd.DataFrame: + calls.append((index_code, start, end)) + return pd.DataFrame( + { + "date": pd.to_datetime(["2024-01-31", "2024-03-29"]), + "symbol": ["000001.SZ", "000002.SZ"], + "weight": [1.0, 1.0], + } + ) + + monkeypatch.setattr(pipeline, "IndexConstituentsFeed", _FakeIndexFeed) + + universe, symbols = pipeline._build_universe(_index_cfg(), _FakeLogger()) + + assert symbols == ["000001.SZ", "000002.SZ"] + assert calls == [("000300.SH", "2023-02-10", "2024-03-31")] + # The run starts after the Jan snapshot but before the Mar snapshot. Without + # the pre-start fetch this as-of membership would be empty. + assert universe.members(pd.Timestamp("2024-02-15")) == ["000001.SZ"] diff --git a/tests/test_index_universe.py b/tests/test_index_universe.py new file mode 100644 index 0000000..c07a9d8 --- /dev/null +++ b/tests/test_index_universe.py @@ -0,0 +1,83 @@ +"""PIT index universe — the core fix for survivorship / membership look-ahead.""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from universe.index_universe import PITIndexUniverse + + +def _constituents(): + # Two snapshots. C leaves and D joins at the Feb snapshot. + rows = [ + ("2024-01-31", "000001.SZ"), + ("2024-01-31", "000002.SZ"), + ("2024-01-31", "000003.SZ"), # C: in the index in Jan, dropped in Feb + ("2024-02-29", "000001.SZ"), + ("2024-02-29", "000002.SZ"), + ("2024-02-29", "000004.SZ"), # D: only joins at the Feb snapshot + ] + return pd.DataFrame( + { + "date": pd.to_datetime([r[0] for r in rows]), + "symbol": [r[1] for r in rows], + "weight": 1.0, + } + ) + + +def test_members_use_latest_snapshot_on_or_before_date(): + uni = PITIndexUniverse(_constituents()) + # mid-Feb -> the Jan snapshot is the latest on-or-before + assert uni.members(pd.Timestamp("2024-02-15")) == ["000001.SZ", "000002.SZ", "000003.SZ"] + # after the Feb snapshot + assert uni.members(pd.Timestamp("2024-03-10")) == ["000001.SZ", "000002.SZ", "000004.SZ"] + + +def test_members_no_lookahead_into_future_snapshot(): + uni = PITIndexUniverse(_constituents()) + # 000004.SZ only joins on 2024-02-29; it must NOT appear before that snapshot + assert "000004.SZ" not in uni.members(pd.Timestamp("2024-02-15")) + + +def test_members_keep_dropped_name_for_its_era(): + # survivorship: 000003.SZ left in Feb but must be a member for Jan/early-Feb + uni = PITIndexUniverse(_constituents()) + assert "000003.SZ" in uni.members(pd.Timestamp("2024-02-01")) + assert "000003.SZ" not in uni.members(pd.Timestamp("2024-03-01")) + + +def test_members_empty_before_first_snapshot(): + uni = PITIndexUniverse(_constituents()) + assert uni.members(pd.Timestamp("2024-01-01")) == [] + + +def test_tradable_filters_missing_close(): + uni = PITIndexUniverse(_constituents()) + date = pd.Timestamp("2024-02-15") + panel = pd.DataFrame( + {"close": [10.0, float("nan"), 12.0]}, + index=pd.MultiIndex.from_tuples( + [(date, "000001.SZ"), (date, "000002.SZ"), (date, "000003.SZ")], + names=["date", "symbol"], + ), + ) + # 000002.SZ has a NaN close -> excluded; result stays a subset of members + assert uni.tradable(date, panel) == ["000001.SZ", "000003.SZ"] + + +def test_tradable_empty_when_no_market_data_for_date(): + uni = PITIndexUniverse(_constituents()) + empty = pd.DataFrame( + {"close": []}, + index=pd.MultiIndex.from_arrays( + [pd.DatetimeIndex([]), pd.Index([], dtype=object)], names=["date", "symbol"] + ), + ) + assert uni.tradable(pd.Timestamp("2024-02-15"), empty) == [] + + +def test_requires_date_and_symbol_columns(): + with pytest.raises(ValueError, match="date.*symbol|constituents"): + PITIndexUniverse(pd.DataFrame({"foo": [1]})) diff --git a/tests/test_neutralize.py b/tests/test_neutralize.py new file mode 100644 index 0000000..b733232 --- /dev/null +++ b/tests/test_neutralize.py @@ -0,0 +1,83 @@ +"""Tests for industry + size cross-sectional neutralization.""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from factors.process.neutralize import neutralize_by_date + + +def _cross_section(date, symbols, factor, industry, mcap): + idx = pd.MultiIndex.from_product([[pd.Timestamp(date)], symbols], names=["date", "symbol"]) + return ( + pd.Series(factor, index=idx, dtype=float), + pd.Series(industry, index=idx), + pd.Series(mcap, index=idx, dtype=float), + ) + + +def test_residual_is_orthogonal_to_size_and_industry(): + syms = [f"{i:06d}.SZ" for i in range(12)] + ind = ["A"] * 6 + ["B"] * 6 + log_mcap = np.linspace(20.0, 23.0, 12) + mcap = np.exp(log_mcap) + # factor = 3*log_mcap + industry offset (+5 / -5) + a pattern orthogonal to both + extra = np.array([1, -1, 1, -1, 1, -1, 1, -1, 1, -1, 1, -1], dtype=float) + offset = np.where(np.array(ind) == "A", 5.0, -5.0) + factor = 3.0 * log_mcap + offset + extra + f, i, m = _cross_section("2024-03-04", syms, factor, ind, mcap) + + resid = neutralize_by_date(f, i, m) + # residual must be (numerically) uncorrelated with log market cap ... + assert abs(np.corrcoef(resid.to_numpy(), log_mcap)[0, 1]) < 1e-6 + # ... and have ~zero mean within each industry (size+industry removed) + by_ind = resid.groupby(pd.Series(ind, index=resid.index)).mean() + assert by_ind.abs().max() < 1e-9 + + +def test_missing_inputs_become_nan_not_fabricated(): + syms = [f"{i:06d}.SZ" for i in range(8)] # enough names to keep DOF > 0 + f, i, m = _cross_section( + "2024-03-04", + syms, + list(range(1, 9)), + ["A", "A", "A", "A", "B", "B", "B", "B"], + list(np.exp(np.arange(8) + 20.0)), + ) + i.iloc[0] = np.nan # missing industry for one name + resid = neutralize_by_date(f, i, m) + assert pd.isna(resid.iloc[0]) # missing-industry name -> NaN + assert resid.iloc[1:].notna().all() # the other 7 (DOF > 0) get residuals + + +def test_degenerate_cross_section_returns_nan(): + f, i, m = _cross_section("2024-03-04", ["A.SZ", "B.SZ"], [1.0, 2.0], ["X", "Y"], [10, 20]) + resid = neutralize_by_date(f, i, m) # only 2 names (< 3) -> NaN + assert resid.isna().all() + + +def test_saturated_cross_section_returns_nan_not_zeros(): + # 3 names, 2 industries -> n_params = 1 + 2 = 3, residual DOF = 0. + # A saturated fit would yield ~0 residuals; we must return NaN instead. + f, i, m = _cross_section( + "2024-03-04", ["A.SZ", "B.SZ", "C.SZ"], [1.0, 2.0, 3.0], ["X", "X", "Y"], [10, 20, 30] + ) + resid = neutralize_by_date(f, i, m) + assert resid.isna().all() + + +def test_independent_per_date(): + syms = [f"{i:06d}.SZ" for i in range(6)] + rows_f, rows_i, rows_m = [], [], [] + for d, base in [("2024-03-04", 0.0), ("2024-03-05", 100.0)]: + f, i, m = _cross_section( + d, syms, np.arange(6) + base, ["A", "A", "A", "B", "B", "B"], np.exp(np.arange(6) + 20) + ) + rows_f.append(f) + rows_i.append(i) + rows_m.append(m) + factor = pd.concat(rows_f) + resid = neutralize_by_date(factor, pd.concat(rows_i), pd.concat(rows_m)) + # both dates produce finite residuals independently + assert resid.groupby(level="date").apply(lambda s: s.notna().any()).all() diff --git a/tests/test_panel_store.py b/tests/test_panel_store.py new file mode 100644 index 0000000..64585e4 --- /dev/null +++ b/tests/test_panel_store.py @@ -0,0 +1,110 @@ +"""Tests for PanelStore (Slice 2 store part; DATA-008/009/010). + +A PanelStore persists a canonical (date, symbol) panel to parquet and reads it +back, optionally filtering by a closed date interval and/or a symbol subset. The +round-trip must preserve the panel exactly: MultiIndex(date, symbol) names, the +column set/order, dtypes, sort order, and values (NaN cells included). + +Tests use ``tmp_path`` only — never a real ``artifacts/`` directory. +""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from data.clean.schema import INDEX_NAMES, validate_panel +from data.store.panel_store import PanelStore + + +def test_panel_store_roundtrip_preserves_panel(tmp_path, demo_panel): + """write() then read() returns an equivalent panel (index/columns/values).""" + store = PanelStore(root=str(tmp_path)) + store.write("daily", demo_panel) + + loaded = store.read("daily") + + # Still a valid canonical panel. + validate_panel(loaded) + assert list(loaded.index.names) == INDEX_NAMES + + # Same columns, same order. + assert list(loaded.columns) == list(demo_panel.columns) + + # Index and values are equivalent (NaN cells preserved). + pd.testing.assert_frame_equal(loaded, demo_panel, check_like=False) + + +def test_panel_store_filters_by_date(tmp_path, demo_panel): + """read(start, end) returns only rows within the closed [start, end] interval.""" + store = PanelStore(root=str(tmp_path)) + store.write("daily", demo_panel) + + all_dates = demo_panel.index.get_level_values("date").unique().sort_values() + start = all_dates[5] + end = all_dates[10] + + loaded = store.read("daily", start=start, end=end) + + got_dates = loaded.index.get_level_values("date").unique() + # Closed interval: both endpoints included, nothing outside. + assert got_dates.min() == start + assert got_dates.max() == end + assert (got_dates >= start).all() + assert (got_dates <= end).all() + # Exactly the 6 days in [start, end] (indices 5..10 inclusive). + assert len(got_dates) == 6 + + # Filtering by date must not drop any symbols present in that window. + expected = demo_panel.loc[(demo_panel.index.get_level_values("date") >= start) + & (demo_panel.index.get_level_values("date") <= end)] + pd.testing.assert_frame_equal(loaded, expected, check_like=False) + + +def test_panel_store_filters_by_symbols(tmp_path, demo_panel): + """read(symbols=[...]) returns only the requested symbol subset.""" + store = PanelStore(root=str(tmp_path)) + store.write("daily", demo_panel) + + wanted = ["000001.SZ", "000003.SZ"] + loaded = store.read("daily", symbols=wanted) + + got_symbols = sorted(loaded.index.get_level_values("symbol").unique().tolist()) + assert got_symbols == sorted(wanted) + + expected = demo_panel.loc[ + demo_panel.index.get_level_values("symbol").isin(wanted) + ] + pd.testing.assert_frame_equal(loaded, expected, check_like=False) + + +def test_panel_store_string_date_filter(tmp_path, demo_panel): + """Date filter accepts 'YYYY-MM-DD' strings, not only Timestamps.""" + store = PanelStore(root=str(tmp_path)) + store.write("daily", demo_panel) + + loaded = store.read("daily", start="2024-01-03", end="2024-01-05") + got_dates = loaded.index.get_level_values("date").unique() + assert got_dates.min() >= pd.Timestamp("2024-01-03") + assert got_dates.max() <= pd.Timestamp("2024-01-05") + + +def test_panel_store_overwrite_guard(tmp_path, demo_panel): + """overwrite=False on an existing file raises a readable error; True replaces.""" + store = PanelStore(root=str(tmp_path)) + store.write("daily", demo_panel) + + with pytest.raises(ValueError, match="already exists"): + store.write("daily", demo_panel, overwrite=False) + + # Default overwrite=True succeeds and stays consistent on re-read. + store.write("daily", demo_panel) + loaded = store.read("daily") + pd.testing.assert_frame_equal(loaded, demo_panel, check_like=False) + + +def test_panel_store_read_missing_raises(tmp_path): + """Reading a name that was never written raises a readable error.""" + store = PanelStore(root=str(tmp_path)) + with pytest.raises(FileNotFoundError, match="No stored panel"): + store.read("does_not_exist") diff --git a/tests/test_phase0_pipeline.py b/tests/test_phase0_pipeline.py new file mode 100644 index 0000000..005a387 --- /dev/null +++ b/tests/test_phase0_pipeline.py @@ -0,0 +1,214 @@ +"""Slice 12: Phase 0 end-to-end pipeline tests (TEST-007, CLI-002, INV-006/007). + +These run the REAL spine (qt.pipeline.run_phase0) against the offline DemoFeed — +no network, no tushare token. Every write is redirected under ``tmp_path`` via a +config whose ``output`` dirs point inside the temp dir, so the suite never +pollutes the repo's ``artifacts/`` (CONTRACTS §8f, SEC-003). +""" + +from __future__ import annotations + +import math +from pathlib import Path + +import numpy as np +import pytest +import yaml + +import qt.pipeline as pipeline +from qt.config import load_config +from qt.pipeline import _build_feed, run_phase0 + + +def _write_tmp_config( + tmp_path: Path, + example_config_path: str, + *, + start: str = "2024-01-01", + end: str = "2024-06-30", + source: str = "demo", + external_secret_file: str | None = None, + name: str = "config.yaml", +) -> Path: + """Copy the example config but redirect every output dir under ``tmp_path``.""" + raw = yaml.safe_load(Path(example_config_path).read_text(encoding="utf-8")) + out = tmp_path / "artifacts" + raw["data"]["source"] = source + # Keep the run small + fast: a short window is plenty for momentum_20. + raw["data"]["start"] = start + raw["data"]["end"] = end + if external_secret_file is not None: + raw["data"]["external_secret_file"] = external_secret_file + raw["output"] = { + "root_dir": str(out), + "data_dir": str(out / "data"), + "factor_dir": str(out / "factors"), + "report_dir": str(out / "reports"), + "log_dir": str(out / "logs"), + "overwrite": True, + } + cfg_path = tmp_path / name + cfg_path.write_text(yaml.safe_dump(raw), encoding="utf-8") + return cfg_path + + +def test_phase0_pipeline_runs_with_demo_data(tmp_path, example_config_path): + """The full demo pipeline runs and returns a populated result (TEST-007).""" + cfg_path = _write_tmp_config(tmp_path, example_config_path) + result = run_phase0(str(cfg_path)) + + assert result.panel_rows > 0 + assert result.panel_symbols == 5 + assert result.factor_name == "momentum_20" + # NAV table has the contract columns and at least one rebalance row. + assert list(result.nav_table.columns) == [ + "nav", + "gross_return", + "cost", + "turnover", + "net_return", + ] + assert not result.nav_table.empty + # Performance dict has the P0 metrics. + for key in ("annual_return", "max_drawdown", "volatility", "sharpe"): + assert key in result.performance + + +def test_phase0_pipeline_writes_expected_artifacts(tmp_path, example_config_path): + """All four expected artifacts land under the configured output dir (§15).""" + cfg_path = _write_tmp_config(tmp_path, example_config_path) + result = run_phase0(str(cfg_path)) + + expected = [ + result.data_path, + result.factor_path, + result.report_path, + result.log_path, + ] + for path in expected: + assert path.exists(), f"missing artifact: {path}" + # SEC-003: nothing is written outside the temp output dir. + assert str(tmp_path) in str(path) + + +def test_phase0_summary_mentions_static_universe_downgrade(tmp_path, example_config_path): + """The summary discloses the static-universe PIT downgrade (INV-007).""" + cfg_path = _write_tmp_config(tmp_path, example_config_path) + result = run_phase0(str(cfg_path)) + + text = result.report_path.read_text(encoding="utf-8") + assert "DOWNGRADES" in text + lowered = text.lower() + assert "static universe" in lowered or "staticuniverse" in lowered + assert "pit" in lowered + # The daily-data and simple-fallback downgrades are also disclosed. + assert "daily" in lowered + assert "alphalens" in lowered or "quantstats" in lowered + + +def test_phase0_pipeline_is_reentrant(tmp_path, example_config_path): + """Re-running over already-written files must not fail (INV-006).""" + cfg_path = _write_tmp_config(tmp_path, example_config_path) + first = run_phase0(str(cfg_path)) + assert first.report_path.exists() + + # Second run over the same (now-populated) output dir must succeed. + second = run_phase0(str(cfg_path)) + assert second.report_path.exists() + assert second.panel_rows == first.panel_rows + assert second.factor_name == first.factor_name + + +def test_phase0_headline_metrics_are_finite_and_sane(tmp_path, example_config_path): + """Monthly nav must annualize at 12/yr -> finite, sane headline (HIGH-1). + + Before the fix, ``performance_summary`` used the daily default (252/yr) on a + monthly nav, exploding annual_return to ~2e12 % and sharpe to ~17. With the + correct cadence the metrics are finite and within a sane band. + """ + cfg_path = _write_tmp_config( + tmp_path, example_config_path, start="2024-01-01", end="2024-12-31" + ) + result = run_phase0(str(cfg_path)) + perf = result.performance + + annual_return = perf["annual_return"] + sharpe = perf["sharpe"] + volatility = perf["volatility"] + assert math.isfinite(annual_return) + assert math.isfinite(sharpe) + assert math.isfinite(volatility) + # Sane band: |annual_return| < 50 (i.e. < 5000%); the old bug was ~2e10. + assert abs(annual_return) < 50.0 + assert abs(sharpe) < 50.0 + + +def test_phase0_report_has_no_hidden_na_quantile(tmp_path, example_config_path): + """No quantile bucket is a hidden inf shown as 'n/a' (MEDIUM-1). + + Over the full calendar the falling symbol used to cross zero, producing inf + forward returns whose bucket mean rendered as 'n/a'. With the fix every + bucket cell is finite. + """ + cfg_path = _write_tmp_config( + tmp_path, example_config_path, start="2024-01-01", end="2024-12-31" + ) + result = run_phase0(str(cfg_path)) + + q = result.quantile_returns + assert not q.empty + assert not np.isinf(q.to_numpy()).any() + # The report's quantile table must not show an inf-driven 'n/a'. + means = q.mean(axis=0) + assert np.isfinite(means.to_numpy()).all() + + +def test_phase0_tushare_source_requires_secret_file(tmp_path, example_config_path): + """source='tushare' with no secret file -> a readable error, not demo data (HIGH-3).""" + cfg_path = _write_tmp_config( + tmp_path, example_config_path, source="tushare", external_secret_file="" + ) + cfg = load_config(str(cfg_path)) + with pytest.raises(ValueError) as exc: + _build_feed(cfg) + msg = str(exc.value) + assert "external_secret_file" in msg + assert "tushare" in msg.lower() + + +def test_phase0_tushare_source_routes_to_tushare_feed( + tmp_path, example_config_path, monkeypatch +): + """source='tushare' with a secret path dispatches to TushareFeed (HIGH-3). + + No network or token read happens: TushareFeed is monkeypatched, so this + asserts the DISPATCH, not the data. A 'tushare' config must never be served + DemoFeed data silently. + """ + cfg_path = _write_tmp_config( + tmp_path, + example_config_path, + source="tushare", + external_secret_file=str(tmp_path / "fake.config.json"), + ) + cfg = load_config(str(cfg_path)) + + captured: dict[str, object] = {} + + class _FakeTushareFeed: + def __init__(self, secret_file: str, token_key: str = "tushare.token"): + captured["secret_file"] = secret_file + captured["token_key"] = token_key + + monkeypatch.setattr(pipeline, "TushareFeed", _FakeTushareFeed) + + feed = _build_feed(cfg) + assert isinstance(feed, _FakeTushareFeed) + assert captured["secret_file"] == str(tmp_path / "fake.config.json") + # The demo source must still route to the offline DemoFeed. + demo_cfg = load_config( + str(_write_tmp_config(tmp_path, example_config_path, name="demo.yaml")) + ) + from data.feed.demo_feed import DemoFeed + + assert isinstance(_build_feed(demo_cfg), DemoFeed) diff --git a/tests/test_pit_financials.py b/tests/test_pit_financials.py new file mode 100644 index 0000000..e8fe2ea --- /dev/null +++ b/tests/test_pit_financials.py @@ -0,0 +1,97 @@ +"""Tests for ann_date point-in-time financial alignment (the disclosure red-line).""" + +from __future__ import annotations + +import pandas as pd + +from data.clean.pit_financials import asof_financials + + +def _index(dates, symbols=("000001.SZ",)): + return pd.MultiIndex.from_product( + [pd.to_datetime(dates), list(symbols)], names=["date", "symbol"] + ) + + +def _fina(): + # prior annual (ann 2023-12-31) + Q1 (end 2024-03-31 but ANNOUNCED 2024-04-20) + return pd.DataFrame( + { + "symbol": ["000001.SZ", "000001.SZ"], + "ann_date": ["20231231", "20240420"], + "end_date": ["20230930", "20240331"], + "roe": [8.0, 3.1], + } + ) + + +def test_asof_uses_ann_date_not_end_date(): + idx = _index(["2024-04-10", "2024-04-19", "2024-04-20", "2024-04-21"]) + out = asof_financials(idx, _fina(), ["roe"]) + + def roe_on(d): + return out.xs(pd.Timestamp(d), level="date")["roe"].iloc[0] + + # BEFORE the Q1 disclosure (ann 04-20): must still be the prior report (8.0), + # NOT the Q1 figure (3.1) — an end_date join would wrongly leak 3.1 here. + assert roe_on("2024-04-10") == 8.0 + assert roe_on("2024-04-19") == 8.0 + # ON/AFTER disclosure: the Q1 figure becomes visible. + assert roe_on("2024-04-20") == 3.1 + assert roe_on("2024-04-21") == 3.1 + + +def test_asof_carries_forward_report_disclosed_before_window(): + # a single report disclosed 2023-12-31; the whole trade window is mid-2024. + fina = pd.DataFrame( + {"symbol": ["000001.SZ"], "ann_date": ["20231231"], + "end_date": ["20230930"], "roe": [8.0]} + ) + idx = _index(["2024-06-03", "2024-06-10", "2024-06-17"]) + out = asof_financials(idx, fina, ["roe"]) + # the prior disclosed report carries forward to every trade date (no NaN gap) + assert (out["roe"] == 8.0).all() + + +def test_asof_nan_before_first_disclosure(): + idx = _index(["2023-06-01"]) # before the earliest ann_date (2023-12-31) + out = asof_financials(idx, _fina(), ["roe"]) + assert pd.isna(out["roe"].iloc[0]) + + +def test_asof_no_future_leak_when_future_report_changes(): + idx = _index(["2024-04-19"]) + base = asof_financials(idx, _fina(), ["roe"])["roe"].iloc[0] + # mutate the FUTURE (Q1, ann 04-20) report's value — must not change 04-19 + fina2 = _fina() + fina2.loc[fina2["ann_date"] == "20240420", "roe"] = 999.0 + after = asof_financials(idx, fina2, ["roe"])["roe"].iloc[0] + assert base == after == 8.0 + + +def test_asof_dedupes_identical_disclosures(): + fina = pd.concat([_fina(), _fina()], ignore_index=True) # duplicated rows + idx = _index(["2024-04-21"]) + out = asof_financials(idx, fina, ["roe"]) + assert out["roe"].iloc[0] == 3.1 + + +def test_asof_multiple_symbols_independent(): + fina = pd.DataFrame( + { + "symbol": ["000001.SZ", "000002.SZ"], + "ann_date": ["20240420", "20240101"], + "roe": [3.1, 5.5], + } + ) + idx = _index(["2024-04-21"], symbols=["000001.SZ", "000002.SZ"]) + out = asof_financials(idx, fina, ["roe"]) + assert out.xs("000001.SZ", level="symbol")["roe"].iloc[0] == 3.1 + assert out.xs("000002.SZ", level="symbol")["roe"].iloc[0] == 5.5 + + +def test_asof_raises_on_missing_field(): + import pytest + + with pytest.raises(ValueError, match="missing field"): + asof_financials(_index(["2024-04-21"]), _fina(), ["nonexistent"]) diff --git a/tests/test_portfolio_topn.py b/tests/test_portfolio_topn.py new file mode 100644 index 0000000..1e59b5d --- /dev/null +++ b/tests/test_portfolio_topn.py @@ -0,0 +1,133 @@ +"""Tests for the TopN equal-weight portfolio constructor (Slice 8). + +Covers requirements PF-001/002/003/004/005/009: + - select the N highest scores, + - weights sum to 1 (float tol < 1e-9), + - NaN scores are ignored, + - fewer candidates than N -> equal-weight the actual count (still sum 1), + - no candidates -> empty Series (no crash), + - long-only (no negative weights). + +Uses the shared ``scores_factory`` fixture (conftest) so we don't invent demo +data. ``make_scores`` ranks 000001..000005 ascending (000005 highest) with +000004 set to NaN. +""" + +from __future__ import annotations + +import numpy as np +import pandas as pd + +from portfolio.base import PortfolioConstructor +from portfolio.construct import TopNEqualWeight + +WEIGHT_TOL = 1e-9 + + +def test_topn_selects_highest_scores(scores_factory): + """PF-001: the top_n highest scores are the only symbols selected.""" + scores = scores_factory("2024-01-01") # 000005>000003>000002>000001, 000004 NaN + constructor = TopNEqualWeight(top_n=2) + + weights = constructor.build(scores) + + # Two highest non-NaN scores are 000005 (0.5) and 000003 (0.3). + assert set(weights.index) == {"000005.SZ", "000003.SZ"} + assert "000001.SZ" not in weights.index + assert "000002.SZ" not in weights.index + + +def test_topn_weights_sum_to_one(scores_factory): + """PF-002: selected weights sum to 1.0 within tolerance, equal weight 1/k.""" + scores = scores_factory("2024-01-01") + constructor = TopNEqualWeight(top_n=3) + + weights = constructor.build(scores) + + assert len(weights) == 3 + assert abs(weights.sum() - 1.0) < WEIGHT_TOL + # Equal weight: every selected position is 1/3. + assert np.allclose(weights.to_numpy(), 1.0 / 3.0, atol=WEIGHT_TOL) + + +def test_topn_ignores_nan_scores(scores_factory): + """PF-003: NaN-score symbols never enter the portfolio.""" + scores = scores_factory("2024-01-01") # 000004.SZ is NaN + # Ask for more than the number of valid candidates. + constructor = TopNEqualWeight(top_n=5) + + weights = constructor.build(scores) + + assert "000004.SZ" not in weights.index + # 4 valid (non-NaN) candidates remain; all selected, equal-weighted. + assert len(weights) == 4 + assert abs(weights.sum() - 1.0) < WEIGHT_TOL + + +def test_topn_handles_less_than_n_candidates(scores_factory): + """PF-005: fewer candidates than N -> equal-weight the actual count, sum 1.""" + # Only two non-NaN scores available. + scores = pd.Series( + {"A.SZ": 0.9, "B.SZ": 0.1, "C.SZ": np.nan}, name="score" + ) + constructor = TopNEqualWeight(top_n=10) + + weights = constructor.build(scores) + + assert set(weights.index) == {"A.SZ", "B.SZ"} + assert len(weights) == 2 + assert abs(weights.sum() - 1.0) < WEIGHT_TOL + assert np.allclose(weights.to_numpy(), 0.5, atol=WEIGHT_TOL) + + +def test_topn_returns_empty_when_no_candidates(): + """PF-004: no candidates -> empty Series, no crash, no NaN/inf weights.""" + scores = pd.Series( + {"A.SZ": np.nan, "B.SZ": np.nan}, name="score" + ) + constructor = TopNEqualWeight(top_n=3) + + weights = constructor.build(scores) + + assert isinstance(weights, pd.Series) + assert len(weights) == 0 + assert weights.sum() == 0.0 # empty sum is 0, no crash + + +def test_topn_returns_empty_when_input_is_empty(): + """PF-004 edge: an empty score Series yields an empty weight Series.""" + scores = pd.Series(dtype=float, name="score") + constructor = TopNEqualWeight(top_n=3) + + weights = constructor.build(scores) + + assert isinstance(weights, pd.Series) + assert len(weights) == 0 + + +def test_topn_is_long_only(scores_factory): + """PF-009: P0 portfolio has no negative weights even with negative scores.""" + scores = pd.Series( + {"A.SZ": -0.5, "B.SZ": -0.2, "C.SZ": -0.9}, name="score" + ) + constructor = TopNEqualWeight(top_n=2, long_only=True) + + weights = constructor.build(scores) + + # Highest (least negative) scores: A (-0.2... actually -0.5) -> A and B. + assert (weights >= 0).all() + assert abs(weights.sum() - 1.0) < WEIGHT_TOL + + +def test_topn_subclasses_constructor_and_is_pure(scores_factory): + """Contract: TopNEqualWeight IS-A PortfolioConstructor and never mutates input.""" + scores = scores_factory("2024-01-01") + before = scores.copy() + constructor = TopNEqualWeight(top_n=2) + + assert isinstance(constructor, PortfolioConstructor) + + _ = constructor.build(scores) + + # Input Series is unchanged (immutable style). + pd.testing.assert_series_equal(scores, before) diff --git a/tests/test_processing_neutralize.py b/tests/test_processing_neutralize.py new file mode 100644 index 0000000..9ca6944 --- /dev/null +++ b/tests/test_processing_neutralize.py @@ -0,0 +1,36 @@ +"""ProcessingPipeline neutralization wiring (covariates required, no silent skip).""" + +from __future__ import annotations + +import numpy as np +import pandas as pd +import pytest + +from factors.process.pipeline import ProcessingPipeline + + +def _factor_panel(): + syms = [f"{i:06d}.SZ" for i in range(6)] + idx = pd.MultiIndex.from_product( + [[pd.Timestamp("2024-03-04")], syms], names=["date", "symbol"] + ) + return pd.DataFrame({"f": np.arange(6, dtype=float)}, index=idx) + + +def test_neutralize_enabled_without_covariates_raises(): + pipe = ProcessingPipeline(drop_missing=False, standardize=False, neutralize=True) + with pytest.raises(ValueError, match="industry/market_cap"): + pipe.transform(_factor_panel()) + + +def test_neutralize_runs_with_covariates(): + fp = _factor_panel() + industry = pd.Series(["A", "A", "A", "B", "B", "B"], index=fp.index) + mcap = pd.Series(np.exp(np.arange(6) + 20.0), index=fp.index) + pipe = ProcessingPipeline( + drop_missing=True, standardize=True, neutralize=True, + industry=industry, market_cap=mcap, + ) + out = pipe.transform(fp) + assert isinstance(out.index, pd.MultiIndex) + assert out["f"].notna().any() diff --git a/tests/test_project_bootstrap.py b/tests/test_project_bootstrap.py new file mode 100644 index 0000000..76aefb5 --- /dev/null +++ b/tests/test_project_bootstrap.py @@ -0,0 +1,44 @@ +"""Slice 0: project bootstrap tests.""" + +from __future__ import annotations + +import importlib +from pathlib import Path + +import pytest + +CORE_PACKAGES = [ + "data", + "universe", + "factors", + "alpha", + "portfolio", + "runtime", + "analytics", + "qt", +] + + +@pytest.mark.parametrize("pkg", CORE_PACKAGES) +def test_import_core_packages(pkg: str) -> None: + """Every layer package imports cleanly after the editable install.""" + assert importlib.import_module(pkg) is not None + + +def test_cli_module_importable() -> None: + """The CLI module imports and exposes a callable main().""" + cli = importlib.import_module("qt.cli") + assert callable(cli.main) + assert callable(cli.build_parser) + + +def test_validate_example_config() -> None: + """The shipped example config parses into a RootConfig.""" + from qt.config import RootConfig, load_config + + path = Path(__file__).resolve().parents[1] / "config" / "example.yaml" + cfg = load_config(str(path)) + assert isinstance(cfg, RootConfig) + assert cfg.data.source == "demo" + assert cfg.portfolio.top_n == 3 + assert cfg.factors[0].name == "momentum_20" diff --git a/tests/test_real_path_config.py b/tests/test_real_path_config.py new file mode 100644 index 0000000..726c78d --- /dev/null +++ b/tests/test_real_path_config.py @@ -0,0 +1,34 @@ +"""The tushare real-data path config loads and is disclosed as REAL (not demo).""" + +from __future__ import annotations + +from pathlib import Path + +from qt.config import load_config +from qt.pipeline import _collect_downgrades + + +def _real_path(example_config_path): + return str(Path(example_config_path).parent / "example_tushare.yaml") + + +def test_real_path_config_loads(example_config_path): + cfg = load_config(_real_path(example_config_path)) + assert cfg.data.source == "tushare" + assert cfg.universe.type == "index" + assert cfg.universe.index_code == "000300.SH" + assert cfg.processing.neutralize.enabled is True + + +def test_downgrades_mark_real_path(example_config_path): + items = _collect_downgrades(load_config(_real_path(example_config_path))) + assert items[0].startswith("DATA PATH = REAL tushare") + assert any("PIT index membership" in x for x in items) + assert any("neutralized" in x for x in items) + + +def test_downgrades_mark_demo_path(example_config_path): + items = _collect_downgrades(load_config(example_config_path)) + assert items[0].startswith("DATA PATH = DEMO") + assert any("Static universe" in x for x in items) + assert any("No neutralization" in x for x in items) diff --git a/tests/test_sim_execution.py b/tests/test_sim_execution.py new file mode 100644 index 0000000..50e6973 --- /dev/null +++ b/tests/test_sim_execution.py @@ -0,0 +1,107 @@ +"""Tests for SimExecution (backlog Slice 9; reqs BT-004, cost/turnover). + +SimExecution is the backtest adapter for the Execution port. These tests pin the +turnover/cost/return math in isolation: turnover is the full L1 distance between +target and current weights (CONTRACTS / BT-004 turnover_formula = l1), cost is +``turnover * fee_rate``, and ``last_return`` is the gross holding return net of +that cost. No other slice is imported. +""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from runtime.execution import BacktestExecution, Execution +from runtime.backtest.sim_execution import SimExecution + + +def test_turnover_from_empty_position_is_one_for_full_investment(): + """Empty book -> fully invested target: turnover is the full L1 sum (= 1.0).""" + exe = SimExecution(fee_rate=0.0) + target = pd.Series({"000001.SZ": 0.5, "000002.SZ": 0.5}) + exe.rebalance_to(target, pd.Timestamp("2024-02-01")) + assert exe.last_turnover == pytest.approx(1.0) + + +def test_turnover_between_positions_is_l1_half_or_documented(): + """Turnover between two books is the documented full L1 (sum of |Δw|).""" + exe = SimExecution(fee_rate=0.0) + first = pd.Series({"000001.SZ": 0.5, "000002.SZ": 0.5}) + exe.rebalance_to(first, pd.Timestamp("2024-02-01")) + # Rotate fully out of 000002 into 000003: aligned over the union of symbols + # {1,2,3}: |0.5-0.5| + |0.5-0| + |0-0.5| = 1.0 (full L1, per CONTRACTS). + second = pd.Series({"000001.SZ": 0.5, "000003.SZ": 0.5}) + exe.rebalance_to(second, pd.Timestamp("2024-03-01")) + assert exe.last_turnover == pytest.approx(1.0) + + +def test_cost_equals_turnover_times_fee_rate(): + """cost == turnover * fee_rate (BT-004).""" + fee_rate = 0.001 + exe = SimExecution(fee_rate=fee_rate) + target = pd.Series({"000001.SZ": 0.6, "000002.SZ": 0.4}) + exe.rebalance_to(target, pd.Timestamp("2024-02-01")) + assert exe.last_cost == pytest.approx(exe.last_turnover * fee_rate) + assert exe.last_cost == pytest.approx(1.0 * fee_rate) + + +def test_last_return_is_net_of_cost(): + """last_return == gross holding return - cost of forming the position.""" + fee_rate = 0.001 + exe = SimExecution(fee_rate=fee_rate) + target = pd.Series({"000001.SZ": 0.5, "000002.SZ": 0.5}) + exe.rebalance_to(target, pd.Timestamp("2024-02-01")) + # Inject the per-symbol gross holding returns for the period that follows. + holding_returns = pd.Series({"000001.SZ": 0.10, "000002.SZ": 0.00}) + net = exe.settle(holding_returns) + gross = 0.5 * 0.10 + 0.5 * 0.00 # = 0.05 + expected = gross - 1.0 * fee_rate + assert net == pytest.approx(expected) + assert exe.last_return() == pytest.approx(expected) + + +def test_positions_update_after_rebalance(): + """positions() reflects the latest target after rebalance_to.""" + exe = SimExecution(fee_rate=0.0) + assert exe.positions().empty # starts empty + target = pd.Series({"000001.SZ": 0.3, "000005.SZ": 0.7}) + exe.rebalance_to(target, pd.Timestamp("2024-02-01")) + pos = exe.positions() + assert set(pos.index) == {"000001.SZ", "000005.SZ"} + assert pos["000001.SZ"] == pytest.approx(0.3) + assert pos["000005.SZ"] == pytest.approx(0.7) + + +def test_sim_execution_is_execution_subclass(): + """SimExecution honours the Execution port (INV-003).""" + assert issubclass(SimExecution, Execution) + + +def test_sim_execution_is_backtest_execution_subclass(): + """SimExecution implements the backtest sub-port AND the base port (HIGH-2). + + The driver depends on ``settle`` / ``last_cost`` / ``last_turnover``; those + must be guaranteed by a port, not just happen to exist on SimExecution. + """ + sim = SimExecution(fee_rate=0.0) + assert isinstance(sim, Execution) + assert isinstance(sim, BacktestExecution) + assert issubclass(BacktestExecution, Execution) + # The backtest-only hooks are abstract on the sub-port (a live adapter that + # implements only Execution is never forced to provide them). + for member in ("settle", "last_cost", "last_turnover"): + assert member in BacktestExecution.__abstractmethods__ + + +def test_live_execution_port_stays_minimal(): + """The base Execution port exposes ONLY the three live methods (HIGH-2). + + Backtest-only members (settle / last_cost / last_turnover) must NOT be + abstract on the live port — keeping 'backtest == live' honest (INV-003). + """ + assert Execution.__abstractmethods__ == frozenset( + {"rebalance_to", "positions", "last_return"} + ) + for backtest_only in ("settle", "last_cost", "last_turnover"): + assert backtest_only not in Execution.__abstractmethods__ diff --git a/tests/test_tradability_enrich.py b/tests/test_tradability_enrich.py new file mode 100644 index 0000000..e8ac0cd --- /dev/null +++ b/tests/test_tradability_enrich.py @@ -0,0 +1,102 @@ +"""Tests for enrich_tradability (joining flags onto the panel).""" + +from __future__ import annotations + +import pandas as pd + +from data.clean.schema import normalize_panel +from data.clean.tradability import enrich_tradability + + +def _panel(): + dates = pd.bdate_range("2024-03-01", periods=3) + rows = [] + for d in dates: + for sym, close in [("000001.SZ", 10.0), ("000002.SZ", 20.0)]: + rows.append( + { + "date": d, "symbol": sym, + "open": close, "high": close, "low": close, "close": close, + "volume": 1.0, "amount": 1.0, "adj_factor": 1.0, + } + ) + return normalize_panel(pd.DataFrame(rows)) + + +def test_enrich_suspended_flag(): + d = pd.Timestamp("2024-03-01") + out = enrich_tradability(_panel(), suspended={(d, "000001.SZ")}) + assert bool(out.loc[(d, "000001.SZ"), "suspended"]) is True + assert bool(out.loc[(d, "000002.SZ"), "suspended"]) is False + + +def test_enrich_is_st_from_intervals(): + intervals = {"000001.SZ": [(pd.Timestamp("2024-01-01"), None, True)]} + out = enrich_tradability(_panel(), st_intervals=intervals) + assert out.xs("000001.SZ", level="symbol")["is_st"].all() + assert not out.xs("000002.SZ", level="symbol")["is_st"].any() + + +def test_is_st_uses_latest_starting_name(): + # ST era ends 2024-02-15; renamed non-ST after -> March is NOT ST + intervals = { + "000001.SZ": [ + (pd.Timestamp("2023-01-01"), pd.Timestamp("2024-02-15"), True), + (pd.Timestamp("2024-02-16"), None, False), + ] + } + out = enrich_tradability(_panel(), st_intervals=intervals) + assert not out.xs("000001.SZ", level="symbol")["is_st"].any() + + +def test_enrich_limit_flags(): + d = pd.Timestamp("2024-03-01") + limits = pd.DataFrame( + { + "date": [d, d], + "symbol": ["000001.SZ", "000002.SZ"], + "up_limit": [10.0, 99.0], # 000001 close==up_limit -> at_up_limit + "down_limit": [1.0, 20.0], # 000002 close==down_limit -> at_down_limit + } + ) + out = enrich_tradability(_panel(), limits=limits) + assert bool(out.loc[(d, "000001.SZ"), "at_up_limit"]) is True + assert bool(out.loc[(d, "000001.SZ"), "at_down_limit"]) is False + assert bool(out.loc[(d, "000002.SZ"), "at_down_limit"]) is True + + +def test_enrich_does_not_mutate_input(): + panel = _panel() + before = set(panel.columns) + enrich_tradability(panel, suspended=set()) + assert set(panel.columns) == before + + +def test_limit_flag_uses_raw_close_and_survives_front_adjust(): + # adj_factor varies (1.0 then 2.0) so qfq close != raw close on the first date. + from data.clean.adjust import front_adjust + + d0, d1 = pd.Timestamp("2024-03-01"), pd.Timestamp("2024-03-04") + + def row(d, close, af): + return {"date": d, "symbol": "000001.SZ", "open": close, "high": close, + "low": close, "close": close, "volume": 1.0, "amount": 1.0, "adj_factor": af} + + raw = normalize_panel(pd.DataFrame([row(d0, 10.0, 1.0), row(d1, 12.0, 2.0)])) + limits = pd.DataFrame( + {"date": [d0], "symbol": ["000001.SZ"], "up_limit": [10.0], "down_limit": [1.0]} + ) + # CORRECT order: enrich on RAW close (10 == up_limit 10) -> at_up_limit True + enriched = enrich_tradability(raw, limits=limits) + assert bool(enriched.loc[(d0, "000001.SZ"), "at_up_limit"]) is True + + adjusted = front_adjust(enriched) + # front-adjust scales the d0 close (anchor af=2.0 -> ratio 0.5 -> qfq 5.0) ... + assert adjusted.loc[(d0, "000001.SZ"), "close"] != 10.0 + # ... yet the limit flag (computed on raw) survives unchanged. + assert bool(adjusted.loc[(d0, "000001.SZ"), "at_up_limit"]) is True + + # WRONG order (regression guard): enriching AFTER front-adjust compares the qfq + # close (5.0) to the raw limit (10.0) -> would miss the limit. We must NOT do this. + wrong = enrich_tradability(front_adjust(raw), limits=limits) + assert bool(wrong.loc[(d0, "000001.SZ"), "at_up_limit"]) is False diff --git a/tests/test_tradability_filters.py b/tests/test_tradability_filters.py new file mode 100644 index 0000000..1e2b9d0 --- /dev/null +++ b/tests/test_tradability_filters.py @@ -0,0 +1,66 @@ +"""Shared tradability filter helper (suspended / ST / price-limit + missing close).""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from universe.filters import apply_tradable_filters + +D = pd.Timestamp("2024-03-04") + + +def _panel(rows: dict) -> pd.DataFrame: + idx = pd.MultiIndex.from_tuples([(D, s) for s in rows], names=["date", "symbol"]) + return pd.DataFrame([rows[s] for s in rows], index=idx) + + +def test_always_drops_missing_close(): + panel = _panel({"A": {"close": 10.0}, "B": {"close": float("nan")}}) + assert apply_tradable_filters(["A", "B"], D, panel, {}) == ["A"] + + +def test_drops_suspended_when_enabled(): + panel = _panel( + {"A": {"close": 10.0, "suspended": False}, "B": {"close": 11.0, "suspended": True}} + ) + assert apply_tradable_filters(["A", "B"], D, panel, {"suspended": True}) == ["A"] + # toggle off -> both kept + assert apply_tradable_filters(["A", "B"], D, panel, {"suspended": False}) == ["A", "B"] + + +def test_drops_st_when_enabled(): + panel = _panel( + {"A": {"close": 10.0, "is_st": False}, "B": {"close": 11.0, "is_st": True}} + ) + assert apply_tradable_filters(["A", "B"], D, panel, {"st": True}) == ["A"] + + +def test_drops_at_limit_when_enabled(): + panel = _panel( + { + "A": {"close": 10.0, "at_up_limit": False, "at_down_limit": False}, + "B": {"close": 11.0, "at_up_limit": True, "at_down_limit": False}, + "C": {"close": 9.0, "at_up_limit": False, "at_down_limit": True}, + } + ) + assert apply_tradable_filters(["A", "B", "C"], D, panel, {"limit_up_down": True}) == ["A"] + + +def test_noop_when_flag_column_absent(): + # filters requested but the panel carries no flag columns (e.g. demo) -> no-op + panel = _panel({"A": {"close": 10.0}, "B": {"close": 11.0}}) + flt = {"suspended": True, "st": True, "limit_up_down": True} + assert apply_tradable_filters(["A", "B"], D, panel, flt) == ["A", "B"] + + +def test_missing_date_returns_empty(): + panel = _panel({"A": {"close": 10.0}}) + assert apply_tradable_filters(["A"], pd.Timestamp("2024-03-05"), panel, {}) == [] + + +def test_requires_close_column(): + idx = pd.MultiIndex.from_tuples([(D, "A")], names=["date", "symbol"]) + panel = pd.DataFrame({"open": [10.0]}, index=idx) + with pytest.raises(ValueError, match="close"): + apply_tradable_filters(["A"], D, panel, {}) diff --git a/tests/test_tushare_covariates.py b/tests/test_tushare_covariates.py new file mode 100644 index 0000000..807dde7 --- /dev/null +++ b/tests/test_tushare_covariates.py @@ -0,0 +1,36 @@ +"""TushareCovariatesFeed mapping tests — no network, fake SDK.""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.tushare_covariates import TushareCovariatesFeed + + +class _Pro: + def stock_basic(self, fields): # noqa: ARG002 + return pd.DataFrame( + {"ts_code": ["000001.SZ", "999999.SZ"], "industry": ["银行", "其他"]} + ) + + def daily_basic(self, ts_code, start_date, end_date, fields): # noqa: ARG002 + return pd.DataFrame( + {"ts_code": [ts_code], "trade_date": ["20240301"], "total_mv": [123.0]} + ) + + +def _feed(monkeypatch): + feed = TushareCovariatesFeed("x.json") + monkeypatch.setattr(feed, "_client", lambda: _Pro()) + return feed + + +def test_industry_filters_to_requested(monkeypatch): + assert _feed(monkeypatch).industry(["000001.SZ"]) == {"000001.SZ": "银行"} + + +def test_market_cap_maps_columns(monkeypatch): + out = _feed(monkeypatch).market_cap(["000001.SZ"], "2024-03-01", "2024-03-31") + assert list(out.columns) == ["date", "symbol", "market_cap"] + assert out.iloc[0]["market_cap"] == 123.0 + assert str(out["date"].dtype).startswith("datetime64") diff --git a/tests/test_tushare_fina.py b/tests/test_tushare_fina.py new file mode 100644 index 0000000..e96db13 --- /dev/null +++ b/tests/test_tushare_fina.py @@ -0,0 +1,41 @@ +"""TushareFinancialFeed mapping tests — no network, fake SDK.""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.tushare_fina import TushareFinancialFeed + + +class _Pro: + def fina_indicator(self, ts_code, start_date, end_date, fields): # noqa: ARG002 + return pd.DataFrame( + { + "ts_code": [ts_code], + "ann_date": ["20240420"], + "end_date": ["20240331"], + "roe": [3.1], + } + ) + + +def test_get_fina_indicator_maps_columns(monkeypatch): + feed = TushareFinancialFeed("x.json") + monkeypatch.setattr(feed, "_client", lambda: _Pro()) + out = feed.get_fina_indicator(["000001.SZ"], "2024-01-01", "2024-12-31", fields=["roe"]) + assert list(out.columns) == ["symbol", "ann_date", "end_date", "roe"] + assert out.iloc[0]["symbol"] == "000001.SZ" + assert out.iloc[0]["roe"] == 3.1 + + +def test_get_fina_indicator_empty_is_schema_shaped(monkeypatch): + feed = TushareFinancialFeed("x.json") + + class _Empty: + def fina_indicator(self, **_kw): + return pd.DataFrame() + + monkeypatch.setattr(feed, "_client", lambda: _Empty()) + out = feed.get_fina_indicator(["000001.SZ"], "2024-01-01", "2024-12-31", fields=["roe"]) + assert list(out.columns) == ["symbol", "ann_date", "end_date", "roe"] + assert len(out) == 0 diff --git a/tests/test_tushare_flags.py b/tests/test_tushare_flags.py new file mode 100644 index 0000000..2cfdfc9 --- /dev/null +++ b/tests/test_tushare_flags.py @@ -0,0 +1,65 @@ +"""TushareFlagsFeed mapping tests — no network, fake SDK.""" + +from __future__ import annotations + +import pandas as pd + +from data.feed.tushare_flags import TushareFlagsFeed + + +class _Pro: + def suspend_d(self, ts_code, start_date, end_date, suspend_type): # noqa: ARG002 + if ts_code == "000001.SZ": + return pd.DataFrame( + {"ts_code": ["000001.SZ"], "trade_date": ["20240304"], "suspend_type": ["S"]} + ) + return pd.DataFrame() + + def namechange(self, ts_code): # noqa: ARG002 + # duplicated rows + an ST and a non-ST interval + return pd.DataFrame( + { + "ts_code": [ts_code, ts_code, ts_code], + "name": ["*ST X", "*ST X", "X"], + "start_date": ["20240301", "20240301", "20230101"], + "end_date": [None, None, "20240229"], + "ann_date": ["20240229", "20240229", "20221231"], + "change_reason": ["ST", "ST", "撤销"], + } + ) + + def stk_limit(self, ts_code, start_date, end_date): # noqa: ARG002 + return pd.DataFrame( + {"trade_date": ["20240304"], "ts_code": [ts_code], "up_limit": [11.0], "down_limit": [9.0]} + ) + + +def _feed(monkeypatch): + feed = TushareFlagsFeed("x.json") + monkeypatch.setattr(feed, "_client", lambda: _Pro()) + return feed + + +def test_suspended_returns_date_symbol_pairs(monkeypatch): + feed = _feed(monkeypatch) + out = feed.suspended(["000001.SZ", "000002.SZ"], "2024-03-01", "2024-03-31") + assert (pd.Timestamp("2024-03-04"), "000001.SZ") in out + assert all(sym != "000002.SZ" for _, sym in out) # 000002 not suspended + + +def test_st_intervals_dedupe_and_flag(monkeypatch): + feed = _feed(monkeypatch) + intervals = feed.st_intervals(["000001.SZ"])["000001.SZ"] + # duplicate (*ST X) rows collapsed to one; an ST interval is present + assert len(intervals) == 2 + assert any(is_st for _, _, is_st in intervals) + assert any(not is_st for _, _, is_st in intervals) + + +def test_limits_maps_columns(monkeypatch): + feed = _feed(monkeypatch) + lim = feed.limits(["000001.SZ"], "2024-03-01", "2024-03-31") + assert list(lim.columns) == ["date", "symbol", "up_limit", "down_limit"] + assert lim.iloc[0]["up_limit"] == 11.0 + assert lim.iloc[0]["down_limit"] == 9.0 + assert str(lim["date"].dtype).startswith("datetime64") diff --git a/tests/test_tushare_throttle.py b/tests/test_tushare_throttle.py new file mode 100644 index 0000000..9df5d54 --- /dev/null +++ b/tests/test_tushare_throttle.py @@ -0,0 +1,58 @@ +"""Tests for the shared rate-limit + retry helper (SEC-004) and TushareFeed wiring. + +No network, no token read. ``request_with_retry`` is exercised with a fake +callable and a monkeypatched ``time.sleep`` (so tests are instant). +""" + +from __future__ import annotations + +import pytest + +from data.feed import throttle +from data.feed.throttle import request_with_retry +from data.feed.tushare_feed import TushareFeed + + +@pytest.fixture +def no_sleep(monkeypatch): + """Record sleeps instead of actually sleeping (keeps tests instant).""" + recorded: list[float] = [] + monkeypatch.setattr(throttle.time, "sleep", recorded.append) + return recorded + + +def test_retries_then_succeeds(no_sleep): + calls = {"n": 0} + + def flaky(**_kw): + calls["n"] += 1 + if calls["n"] < 3: + raise RuntimeError("transient") + return "ok" + + assert request_with_retry(flaky, max_retries=5) == "ok" + assert calls["n"] == 3 # failed twice, third attempt succeeded + + +def test_raises_after_exhausting_retries(no_sleep): + def always_fail(**_kw): + raise RuntimeError("upstream boom") + + with pytest.raises(RuntimeError, match="2 attempt"): + request_with_retry(always_fail, max_retries=2) + + +def test_throttles_per_rate_limit(no_sleep): + request_with_retry(lambda **_kw: "ok", rate_limit=120) # 120/min -> 0.5s + assert no_sleep == [0.5] + + +def test_noop_when_rate_limit_unset(no_sleep): + request_with_retry(lambda **_kw: "ok") + assert no_sleep == [] + + +def test_tushare_feed_call_delegates(no_sleep): + feed = TushareFeed("x.json", rate_limit=60, max_retries=3) # -> 1.0s spacing + assert feed._call(lambda **_kw: "ok") == "ok" + assert 1.0 in no_sleep diff --git a/tests/test_universe.py b/tests/test_universe.py new file mode 100644 index 0000000..ba4e250 --- /dev/null +++ b/tests/test_universe.py @@ -0,0 +1,63 @@ +"""Tests for the universe layer (Slice 4): StaticUniverse. + +Covers UNI-001/002/004 plus the empty-universe edge case. The PIT downgrade +(UNI-003) is a documentation requirement, asserted here only via the docstring +contract that ``members`` ignores ``date``. +""" + +from __future__ import annotations + +import pandas as pd + +from tests.fixtures.panel_factory import NAN_DAY, SYMBOLS, make_demo_panel +from universe.static import StaticUniverse + + +def _date_at(panel: pd.DataFrame, day_index: int) -> pd.Timestamp: + """Return the ``day_index``-th distinct trade date in ``panel``.""" + dates = panel.index.get_level_values("date").unique().sort_values() + return dates[day_index] + + +def test_static_universe_members_returns_config_symbols(): + universe = StaticUniverse(symbols=SYMBOLS) + # PIT downgrade: members ignore the date and return the configured list. + members_early = universe.members(pd.Timestamp("2024-01-01")) + members_late = universe.members(pd.Timestamp("2024-06-30")) + assert members_early == list(SYMBOLS) + assert members_late == list(SYMBOLS) + + +def test_tradable_excludes_missing_close(): + panel = make_demo_panel() + universe = StaticUniverse(symbols=SYMBOLS) + nan_date = _date_at(panel, NAN_DAY) # 000004.SZ has a NaN close here + + tradable = universe.tradable(nan_date, panel) + + assert "000004.SZ" not in tradable + # Every other symbol has a valid close on that date and stays tradable. + for symbol in SYMBOLS: + if symbol != "000004.SZ": + assert symbol in tradable + + +def test_tradable_intersects_members(): + panel = make_demo_panel() + universe = StaticUniverse(symbols=SYMBOLS) + some_date = _date_at(panel, NAN_DAY) + + members = set(universe.members(some_date)) + tradable = set(universe.tradable(some_date, panel)) + + assert tradable.issubset(members) + + +def test_empty_universe_is_allowed(): + panel = make_demo_panel() + universe = StaticUniverse(symbols=[]) + some_date = _date_at(panel, 0) + + assert universe.members(some_date) == [] + # Empty universe must not crash; tradable is also empty. + assert universe.tradable(some_date, panel) == [] diff --git a/universe/base.py b/universe/base.py new file mode 100644 index 0000000..31d1bd1 --- /dev/null +++ b/universe/base.py @@ -0,0 +1,41 @@ +"""Universe port: which symbols exist and which are tradable on a given date. + +The universe defines the cross-section. ``members`` is the (ideally PIT) index +membership; ``tradable`` narrows it to what can actually be traded that day +(missing price, suspended, ST, limit-locked). P0 ``StaticUniverse`` is a PIT +*downgrade* and must be documented as such (UNI-003). +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +import pandas as pd + + +class Universe(ABC): + """Abstract point-in-time stock universe.""" + + @abstractmethod + def members(self, date: pd.Timestamp) -> list[str]: + """Return the symbols that are members of the universe on ``date``. + + For a true PIT universe this reflects historical index membership. + StaticUniverse returns the configured list regardless of date (downgrade). + """ + raise NotImplementedError + + @abstractmethod + def tradable(self, date: pd.Timestamp, panel: pd.DataFrame) -> list[str]: + """Return the subset of members that can be traded on ``date``. + + Args: + date: the cross-section date. + panel: canonical market panel (MultiIndex(date, symbol)) used to + apply filters (missing close, suspended, ST, limit up/down). + + Returns: + A list of tradable symbols; always a subset of ``members(date)``. + May be empty (must not crash downstream). + """ + raise NotImplementedError diff --git a/universe/filters.py b/universe/filters.py new file mode 100644 index 0000000..62eec76 --- /dev/null +++ b/universe/filters.py @@ -0,0 +1,55 @@ +"""Shared tradability filtering for any Universe (used by static + index). + +``apply_tradable_filters`` takes a candidate member list and a date cross-section +of the (enriched) panel and drops names that are not tradable that day: + + * missing close (always — UNI-004), + * suspended (if ``filters['suspended']`` and a ``suspended`` flag is present), + * ST (if ``filters['st']`` and an ``is_st`` flag is present), + * at price limit (if ``filters['limit_up_down']`` and the limit flags are present). + +Each optional filter is a no-op when its config toggle is off OR when the +corresponding flag column is absent (e.g. the offline DemoFeed carries no flags), +so demo/P0 runs are unaffected and a filter only bites once its data is wired in. +""" + +from __future__ import annotations + +import pandas as pd + + +def apply_tradable_filters( + members: list[str], + date: pd.Timestamp, + panel: pd.DataFrame, + filters: dict | None = None, +) -> list[str]: + """Return the subset of ``members`` tradable on ``date`` (order preserved).""" + if not members: + return [] + if "close" not in panel.columns: + raise ValueError("tradable() needs a panel with a 'close' column.") + flt = filters or {} + target = pd.Timestamp(date).normalize() + try: + cross = panel.xs(target, level="date") + except KeyError: + return [] # no market data for that date -> nothing tradable + + out: list[str] = [] + for symbol in members: + if symbol not in cross.index: + continue + row = cross.loc[symbol] + if pd.isna(row["close"]): + continue # missing close (UNI-004, always) + if flt.get("suspended") and bool(row.get("suspended", False)): + continue + if flt.get("st") and bool(row.get("is_st", False)): + continue + if flt.get("limit_up_down") and ( + bool(row.get("at_up_limit", False)) or bool(row.get("at_down_limit", False)) + ): + continue + out.append(symbol) + return out diff --git a/universe/index_universe.py b/universe/index_universe.py new file mode 100644 index 0000000..7a71fe9 --- /dev/null +++ b/universe/index_universe.py @@ -0,0 +1,63 @@ +"""PITIndexUniverse: point-in-time index membership (fixes the UNI-003 downgrade). + +Given periodic constituent snapshots (from :class:`IndexConstituentsFeed`), this +universe answers ``members(date)`` with the constituents of the LATEST snapshot on +or before ``date`` — an as-of join. That is what makes it point-in-time: + + * No look-ahead: a name that only joins the index at a FUTURE snapshot is never + returned for an earlier date. + * No survivorship bias: a name dropped from the index later is still returned + for the dates when it was a member, so historical backtests see the real + investable set of the time. + +It consumes membership already pulled by the data layer (constituents DataFrame); +it does not touch tushare itself, so it is unit-testable with synthetic snapshots. +""" + +from __future__ import annotations + +import bisect + +import pandas as pd + +from universe.base import Universe +from universe.filters import apply_tradable_filters + + +class PITIndexUniverse(Universe): + """A point-in-time universe backed by dated index-constituent snapshots.""" + + def __init__(self, constituents: pd.DataFrame, filters: dict | None = None) -> None: + if not {"date", "symbol"}.issubset(constituents.columns): + raise ValueError( + "constituents must have 'date' and 'symbol' columns " + "(use IndexConstituentsFeed.get_constituents)." + ) + c = constituents.copy() + c["date"] = pd.to_datetime(c["date"]).dt.normalize() + c["symbol"] = c["symbol"].astype(str) + # snapshot date -> sorted unique symbols on that date + self._by_date: dict[pd.Timestamp, list[str]] = { + date: sorted(group["symbol"].unique().tolist()) + for date, group in c.groupby("date") + } + self._snapshots: list[pd.Timestamp] = sorted(self._by_date) + self._filters = dict(filters or {}) + + def members(self, date: pd.Timestamp) -> list[str]: + """Constituents of the latest snapshot on or before ``date`` (as-of).""" + target = pd.Timestamp(date).normalize() + # rightmost snapshot <= target + idx = bisect.bisect_right(self._snapshots, target) - 1 + if idx < 0: + return [] # before the first known snapshot + return list(self._by_date[self._snapshots[idx]]) + + def tradable(self, date: pd.Timestamp, panel: pd.DataFrame) -> list[str]: + """PIT members tradable on ``date`` via the shared tradability filters. + + Drops missing-close names (UNI-004) and, when the matching ``filters`` + toggle is on and the flag column is present, suspended / ST / at-limit + names (P1 — :mod:`universe.filters`). + """ + return apply_tradable_filters(self.members(date), date, panel, self._filters) diff --git a/universe/static.py b/universe/static.py new file mode 100644 index 0000000..8710eb3 --- /dev/null +++ b/universe/static.py @@ -0,0 +1,60 @@ +"""StaticUniverse: a fixed, configured stock universe (Slice 4, P0). + +This is the P0 implementation of the :class:`~universe.base.Universe` port. It +holds a configured list of symbols and applies the only P0 tradability filter: +drop symbols whose ``close`` is missing/NaN on the cross-section date (UNI-004). + +PIT DOWNGRADE (UNI-003) — IMPORTANT +----------------------------------- +``members(date)`` returns the configured symbol list *regardless of date*. It +does **not** reflect true historical index membership (point-in-time +constituents). Using a today-known list for past dates introduces survivorship / +look-ahead membership bias. This downgrade is intentional for P0 and MUST be +recorded in the bias audit / phase0 report; do not present this class as a real +PIT universe. +""" + +from __future__ import annotations + +import pandas as pd + +from universe.base import Universe +from universe.filters import apply_tradable_filters + + +class StaticUniverse(Universe): + """A universe with a fixed, date-independent membership list. + + Args: + symbols: the configured universe symbols (tushare style, e.g. + ``"000001.SZ"``). Stored as an immutable tuple; ``members`` returns a + fresh list each call so callers cannot mutate internal state. + filters: optional filter toggles (e.g. ``{"missing_close": True}``). Only + the ``missing_close`` filter is active in P0; it is always applied so + an empty/None mapping is fine. Other keys are accepted and reserved + for P1 filters (suspended/st/limit) without changing behaviour now. + """ + + def __init__(self, symbols: list[str], filters: dict | None = None) -> None: + self._symbols: tuple[str, ...] = tuple(str(s) for s in symbols) + self._filters: dict = dict(filters) if filters else {} + + def members(self, date: pd.Timestamp) -> list[str]: + """Return the configured symbols, ignoring ``date`` (PIT downgrade). + + See the module docstring: this does NOT reflect true historical index + membership. The ``date`` argument is part of the :class:`Universe` + contract but is deliberately unused here. + """ + return list(self._symbols) + + def tradable(self, date: pd.Timestamp, panel: pd.DataFrame) -> list[str]: + """Return members tradable on ``date`` via the shared tradability filters. + + Always drops missing-close names (UNI-004); also drops suspended / ST / + at-limit names when the matching ``filters`` toggle is on AND the panel + carries the corresponding flag column (P1 — :mod:`universe.filters`). The + result is a subset of ``members(date)`` preserving configured order and + never raises on an empty/absent cross-section. + """ + return apply_tradable_filters(self.members(date), date, panel, self._filters)