diff --git a/BIAS_AUDIT.md b/BIAS_AUDIT.md index e92655e..bfc0acb 100644 --- a/BIAS_AUDIT.md +++ b/BIAS_AUDIT.md @@ -24,10 +24,11 @@ - `missing_close`(总是开):截面日 `close` 为 NaN 的标的不可交易(UNI-004)。 - 统一在 `universe.filters.apply_tradable_filters` 按 `UniverseFilters` 开关执行;flag 由 `data.clean.tradability.enrich_tradability` 从 tushare `suspend_d` / `namechange` / `stk_limit` 富化到 panel(StaticUniverse 与 PITIndexUniverse 共用)。demo 无 flag 数据时各过滤自动 no-op。 - **ST(UNI-006)**:`namechange` 名称区间含 'ST'/'*ST' 即标记,按 date 取生效名称(实证:`000005.SZ` 2024 全程 ST,正确剔除)。 -- **涨跌停(UNI-007)**:用**未复权 raw close** 与当日 raw `up_limit`/`down_limit` 比较,标记 `at_up_limit`/`at_down_limit`(qfq 复权价仅用于因子/回测收益;flag 富化在 front_adjust **之前**完成,故比较的是同口径 raw 价)。实证:`000005.SZ` 2024-02-01 触跌停。当前选股层对两个方向都剔除;**方向感知**(买入只看涨停、持有跌停不强卖)属执行层,后续细化。 +- **涨跌停(UNI-007)**:用**未复权 raw close** 与当日 raw `up_limit`/`down_limit` 比较,标记 `at_up_limit`/`at_down_limit`(qfq 复权价仅用于因子/回测收益;flag 富化在 front_adjust **之前**完成,故比较的是同口径 raw 价)。实证:`000005.SZ` 2024-02-01 触跌停。 +- **方向感知执行(UNI-007 / P2-2,已实现)**:选股层(`apply_tradable_filters`)与执行层(`runtime.fills.simulate_fills`)**拆分**。执行可行性按 panel flag 实时判定、与选股 toggle 无关:`at_up_limit` 挡**买入/加仓**,`at_down_limit` 挡**卖出/减仓**,`suspended`/缺收盘价双向挡。被挡的交易 **carry forward**(绝不强行成交不可能的单),现金一致 sell-then-buy(卖在前释放现金、买在后,现金不足按比例部分成交,**无杠杆**),换手/成本只算**实际成交**,闲置现金按 `cash_return` 计息。demo 无 flag → 全可成交 → P0/P1 数字不变。每个调仓期的 blocked buys/sells/carried/executed turnover 记入回测 feasibility log,phase2 报告有专门小节。 - **停牌(UNI-005)**:`suspend_d` 标记停牌日。**实测发现**:tushare 全天停牌当日**无 bar** → 已被 `missing_close` 剔除,故显式 suspended flag 与之重叠;其价值在盘中停牌(`suspend_timing`)或会给停牌日 bar 的数据源,属防御性。 - 退市 / 无数据标的(如 `000003.SZ`)同样表现为不在 panel 而被剔除。PIT 历史成分见上节。 -- `universe.min_listing_days` 已在配置中(默认 60),但仍 **未执行**(no-op,降级):新上市标的不会被剔除。显式披露(INV-007),后续接上市日期后强制。 +- **`universe.min_listing_days`(UNI-008,P2-2)**:作为**买入/选股资格**过滤。**真实路径已执行**——从 tushare `stock_basic.list_date` 富化每只票上市日,某调仓日`age < min_listing_days` 的新上市标的剔除(边界 `age == min` 放行);**缺 list_date 视为数据缺口,保留并披露**,绝不静默剔除。**demo 路径无上市日 → 仍 no-op(显式披露的降级)**,不伪造上市日。 ## ann_date 财务对齐 diff --git a/CLAUDE.md b/CLAUDE.md index b22400a..112cb48 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -57,7 +57,7 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri ## 开发约定 - **交流中文**；代码/注释/commit message 用**英文**。 -- **Git**：feature 分支 + PR。**PR #1（P0+P1）已 merge 到 `main`**；当前在 `p2-real-baseline` 分支推进 P2-1。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 +- **Git**：feature 分支 + PR。**PR #1（P0+P1）、PR #2（P2-1）已 merge 到 `main`**；当前在 `p2-execution-realism` 分支推进 P2-2。commit 用 conventional 格式，**无 attribution**（不加 Co-Authored-By）。 - **不过度设计**：按路线图 MVP 先打通一条端到端链路，再加层（architecture.html §11，Phase 0→3）。 - **secrets** 一律走外部 `.config.json`；repo `.gitignore` 已排除数据产物(`*.parquet`等)、缓存、`tmp/`（仅留架构文档）。 - 文件小而专（<800 行），immutable 优先。 @@ -74,6 +74,12 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri - 路径感知降级披露（demo/static vs tushare/index/ann_date，绝不把 demo 当真实验证） - ✅ 真数据实证（tushare，非 CI）：复权除权日 raw−5.74%→qfq+0.99% / CSI300 全年 24 快照 328 名换手 / ann_date Q1 延后至 04-20 / 中性化 corr −0.617→0。详见 `BIAS_AUDIT.md`、`artifacts/reports/phase1_summary.md`。 - ✅ **Phase 2-1 真实数据可复现基准**（`p2-real-baseline` 分支，**PR #2 OPEN**）：新 run mode `run-phase2-baseline` + `config/phase2_real_baseline.yaml`（上证50 `000016.SH`，2023-07~2024-06）。**复用 P0/P1 全套机器，不扩因子、不调参**。一次真实跑 ~11min（68 成分 / 25 loaded 快照 / in-window distinct 60 / 11 settled 调仓，候选 12 末日跳过），输出 `artifacts/reports/phase2_real_baseline.md`（gitignored）：数据窗口 / PIT 成分摘要(loaded vs in-window) / ann_date 覆盖率(100%) / 可交易过滤命中(首命中互斥) / 每期持仓 / 换手成本 / IC(≈0.008) / 绩效(年化−17.6%，**亏损动量基准，非业绩声明**) / 全部 P2 降级。诊断只读、demo 源拒绝、token 不入报告。 -- ✅ 质量门：`pytest` **182 passed**（P0=93 / P1=75 / P2-1=14）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`）+ `run-phase0`（demo）均 OK。 -- ⚠️ 剩余 P2（已显式披露）：行业标签用**当前值**非 PIT、涨跌停未方向感知、`min_listing_days` no-op、日线 only、简版 IC/绩效未走 alphalens/quantstats、demo 路径非真数据。 -- 路线图下一步：财务因子组合 / 历史 PIT 行业 / 更细交易约束（architecture.html §11）。 +- ✅ **Phase 2-2 执行真实性**（`p2-execution-realism` 分支，未 PR）：**拆分 selection（选谁）与 execution feasibility（能否成交）**。 + - 方向感知执行 `runtime/fills.py::simulate_fills`：涨停挡买 / 跌停挡卖 / 停牌·缺收盘双向挡;按 panel flag 实时判定,与选股 toggle 无关。 + - **现金一致 sell-then-buy**:卖在前释放现金、买在后,现金不足按比例部分成交 → **无杠杆**;被挡交易 carry forward;换手/成本只算实际成交;闲置现金按 driver 的 `cash_return` 计息（BT-007）。 + - `universe.min_listing_days` **真实路径已执行**（`stock_basic.list_date` 富化,买入资格过滤,边界 age==min 放行,缺 list_date 保留并披露）;demo 无上市日 → 披露 no-op。 + - 回测 `feasibility_log()`：每调仓期 blocked buys/sells/carried/executed turnover/invested;phase2 报告新增 **Execution feasibility** 小节。 + - **保不变量**:demo 无 flag → 全可成交 → P0/P1 数字不变;无未来函数、PIT/ann_date/real-demo 分离不变。 +- ✅ 质量门：`pytest` **204 passed**（P0=94 / P1=75 / P2-1=14 / P2-2=21）；`ruff` clean；`validate-config`（demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`）+ `run-phase0`（demo）均 OK。 +- ⚠️ 剩余 P2（已显式披露）：行业标签用**当前值**非 PIT、日线 only、简版 IC/绩效未走 alphalens/quantstats、demo 路径非真数据。 +- 路线图下一步：财务因子组合 / 历史 PIT 行业 / 分钟级 / alphalens·quantstats 接入（architecture.html §11）。 diff --git a/RUNBOOK.md b/RUNBOOK.md index c860260..bb5cb53 100644 --- a/RUNBOOK.md +++ b/RUNBOOK.md @@ -146,12 +146,43 @@ Guards (correctness / honesty): - **Demo source is refused.** `run-phase2-baseline` raises a readable error on `data.source != 'tushare'` — a "baseline" on offline demo data carries no PIT / ann_date / tradability meaning and must not masquerade as a real validation. -- **Holdings are reconstructed read-only** (universe → scores → `constructor.build`, - the same chain the driver runs); the reporting never sees forward returns at the - factor stage and never re-derives returns. +- **Holdings are the driver's ACHIEVED book** (`BacktestDriver.holdings_log()`, + post execution-feasibility) — the actual positions held each period, NOT the + constructor's desired target (a blocked sell shows the carried name, a blocked + buy is absent). The reporting never sees forward returns at the factor stage. - **No secret leak.** The report echoes only non-sensitive config (window, universe, factor); the token / secret file path is never written into it. +## Phase 2-2 — execution realism (direction-aware fills + min_listing_days) + +P2-2 closes execution/tradability gaps in the backtest. No new factor, no parameter +tuning — it changes how the driver *fills* a target and how selection eligibility is +computed, and it applies to every real-path run (`run-phase0` and +`run-phase2-baseline`). + +**Selection vs execution feasibility (split):** + +- *Selection* (`universe.tradable` / `apply_tradable_filters`): missing_close / + suspended / ST / limit toggles, plus **`min_listing_days`** (UNI-008) — a + buy/selection filter that drops names younger than `min_listing_days` as of each + date. Real path enriches `list_date` from `stock_basic`; a missing list_date is a + disclosed data gap (kept, never silently dropped); demo has no listing dates → a + disclosed no-op. +- *Execution feasibility* (`runtime.fills.simulate_fills`): read off the panel flags, + independent of the selection toggles — `at_up_limit` blocks **buys**, `at_down_limit` + blocks **sells**, `suspended`/missing-close blocks **both**. + +**Cash-coherent fill model:** sells execute first (freeing cash), buys are funded from +available cash and scaled down proportionally if blocked sells starved them (the book +never sums to > 1 — no leverage). Blocked trades carry the current position forward; +turnover/cost count only executed trades; idle cash earns the driver's `cash_return` +(BT-007). The demo panel carries no flags, so every trade is feasible and P0/P1 +numbers are unchanged. + +The phase2 baseline report (`artifacts/reports/phase2_real_baseline.md`) gains an +**Execution feasibility** section: per-rebalance blocked buys / blocked sells / carried +positions / executed turnover / invested fraction, from `BacktestDriver.feasibility_log()`. + ## Quality gate ```bash @@ -173,11 +204,19 @@ Implemented in P1 (real path): - **ann_date financial alignment** — figures used only after disclosure date. - **Industry + size neutralization** — per-date OLS residual. +Resolved in P2-2 (was deferred): + +- **Direction-aware limits/suspension** — now in the execution layer (up-limit + blocks buys, down-limit blocks sells, suspended/missing blocks both; blocked + trades carry forward, executed-only turnover). No longer a crude both-direction + selection drop. +- **min_listing_days** — enforced on the real path (`stock_basic.list_date`) as a + buy/selection filter; demo stays a disclosed no-op. + Still downgraded / deferred (disclosed): - **Demo path** uses offline `DemoFeed` — NOT real data (no PIT/financial meaning). - **Static universe** option remains a PIT downgrade (use `type: index` for real). - **Industry tag is current** (`stock_basic`), not point-in-time — mild downgrade. -- **min_listing_days** configured but not enforced (no-op). -- **Daily bars only**; **simple IC/perf** (not alphalens/quantstats); - limit filter is not yet trade-direction-aware (P2). +- **Daily bars only**; **simple IC / performance** (numpy/pandas, not + alphalens-reloaded / quantstats). diff --git a/TEST_REPORT.md b/TEST_REPORT.md index dc8963f..87cd106 100644 --- a/TEST_REPORT.md +++ b/TEST_REPORT.md @@ -1,4 +1,4 @@ -# TEST_REPORT — Phase 1 (bias-boundary) +# TEST_REPORT — Phase 0 + Phase 1 + Phase 2 (bias-boundary → execution realism) ## Commands @@ -8,6 +8,8 @@ Run from the repo root with the project python (env `quant_mf`): /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m pytest -q /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m ruff check . /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/example.yaml +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/example_tushare.yaml +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli validate-config --config config/phase2_real_baseline.yaml /home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli run-phase0 --config config/example.yaml ``` @@ -15,14 +17,14 @@ Run from the repo root with the project python (env `quant_mf`): | Gate | Command | Result | |---|---|---| -| Unit + integration | `pytest -q` | **168 passed, 0 failed** | +| Unit + integration | `pytest -q` | **204 passed, 0 failed** | | Lint | `ruff check .` | **All checks passed** | -| Config validation | `validate-config` (demo + `example_tushare.yaml`) | exit `0`, prints `OK` | +| Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml`) | exit `0`, prints `OK` | | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` | -Counts below are the actual `pytest --collect-only` numbers (sum = 168). +Counts below are the actual per-file `pytest` numbers (sum = 204). -## Per-file breakdown — Phase 0 core (93) +## Per-file breakdown — Phase 0 core (94) | Test file | Tests | Area | |---|---|---| @@ -41,7 +43,7 @@ Counts below are the actual `pytest --collect-only` numbers (sum = 168). | `test_analytics_factor.py` | 5 | IC / quantile | | `test_analytics_performance.py` | 3 | performance metrics | | `test_phase0_pipeline.py` | 8 | end-to-end pipeline | -| `test_bias_audit_report.py` | 4 | bias audit doc | +| `test_bias_audit_report.py` | 5 | bias audit doc (+1: P2-2 disclosures) | ## Per-file breakdown — Phase 1 bias-boundary (75) @@ -65,38 +67,41 @@ Counts below are the actual `pytest --collect-only` numbers (sum = 168). | `test_covariates_enrich.py` | 3 | industry + market_cap enrichment | | `test_tushare_covariates.py` | 2 | stock_basic + daily_basic feed | | `test_real_path_config.py` | 3 | demo vs real-path downgrade disclosure | -| **Total (P0 + P1)** | **168** | | -## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) +## Per-file breakdown — Phase 2-1 real-data baseline (14) + +| Test file | Tests | Feature | +|---|---|---| +| `test_phase2_baseline.py` | 14 | collectors, demo/real guard, report-field contract, no-secret-leak, settled-vs-candidate dates, loaded-vs-in-window membership | + +## Per-file breakdown — Phase 2-2 execution realism (21) -Verified directly against tushare (results recorded in `BIAS_AUDIT.md` and -`artifacts/reports/phase1_summary.md`): +| Test file | Tests | Red-line / feature | +|---|---|---| +| `test_fills.py` | 10 | direction-aware fill sim (`simulate_fills`) + panel→feasibility adapter; cash-coherent sell-then-buy, no leverage, executed-only turnover | +| `test_driver_feasibility.py` | 5 | end-to-end: down-limit carries, up-limit blocks buy, suspended no-trade, feasibility log == nav index | +| `test_min_listing_days.py` | 6 | `min_listing_days` buy-eligibility boundaries (age <, ==, >; missing list_date kept; no-op cases) | +| **Total (P0 + P1 + P2-1 + P2-2)** | **204** | | + +## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free) -- **front-adjust**: 平安银行 2024-06-14 ex-dividend — raw −5.74% vs qfq +0.99%; - momentum_20 shifts up to 6.77pp. -- **PIT index**: CSI300 2024 — 24 snapshots, 328 distinct names (28 in / 28 out); - `members(2024-06-15)` uses the 2024-06-03 snapshot; a dropped name stays in its era. -- **ann_date**: 平安银行 Q1 (end 2024-03-31, ann 2024-04-20) — as-of roe stays the - prior annual (10.24) until 04-19, switches to Q1 (3.12) on 04-22; no leak. -- **neutralization**: 12 names / 4 industries — corr(momentum, log_mcap) - −0.617 → −0.000; per-industry residual means ≈ 0. +- **P1** (front-adjust / PIT / ann_date / neutralization): see `BIAS_AUDIT.md` and + `artifacts/reports/phase1_summary.md`. +- **P2-1** baseline (SSE50, ~11 min): `artifacts/reports/phase2_real_baseline.md` — + settled-date diagnostics, ann_date coverage, tradability funnel. ## Notes - No test hits the network or reads the tushare token (TEST-002, INV-004): the whole suite runs on `DemoFeed` / fixtures / monkeypatched SDKs. -- Financial factors, the PIT index universe, tradability filters and neutralization - all require the real tushare path; on demo they raise a readable error rather than - fabricate (verified by `test_financial_pipeline.py`, `_build_universe`, the - neutralize guard). -- The demo portfolio's annualized return is intentionally extreme (demo price paths - include a 3x jump); P0/P1 do not optimize for realistic returns — pipeline - correctness (event order, costs, no-lookahead) is what is under test. -- P1 acceptance hardening (locked by tests): price-limit flags compare the RAW - (unadjusted) close to the raw `stk_limit`, enriched BEFORE front-adjust - (`test_tradability_enrich::test_limit_flag_uses_raw_close_and_survives_front_adjust`); - financials are fetched ~16 months before `start` so the prior disclosed report - carries forward (`test_pit_financials::test_asof_carries_forward_report_disclosed_before_window`, - `test_financial_pipeline::test_financial_fetch_uses_lookback_before_start`); - neutralization returns NaN on a saturated cross-section instead of fabricated ~0 - residuals (`test_neutralize::test_saturated_cross_section_returns_nan_not_zeros`). +- **P2-2 execution realism (locked by tests):** selection eligibility and execution + feasibility are split. `runtime.fills.simulate_fills` is the cash-coherent + sell-then-buy model — at-up-limit blocks buys, at-down-limit blocks sells, + suspended/missing blocks both; blocked trades carry forward, turnover/cost count + only executed trades, and idle cash earns the driver's `cash_return` (BT-007). The + demo path has no flags, so every trade is feasible and P0/P1 numbers are unchanged. +- `universe.min_listing_days` is enforced on the real path (list_date from + `stock_basic`) as a buy/selection filter; a missing list_date is kept and disclosed; + the demo path stays a disclosed no-op. +- A duplicate test-function name across two files was found and renamed during P2-2 + (it had been silently shadowing one test in the full-suite run). diff --git a/data/clean/covariates.py b/data/clean/covariates.py index effc079..63c2387 100644 --- a/data/clean/covariates.py +++ b/data/clean/covariates.py @@ -37,3 +37,19 @@ def enrich_covariates( out["market_cap"] = mc["market_cap"].reindex(out.index) return out + + +def enrich_listing( + panel: pd.DataFrame, listing_dates: dict[str, pd.Timestamp] +) -> pd.DataFrame: + """Return a NEW panel with a per-symbol ``list_date`` column (for UNI-008). + + Broadcasts each symbol's listing date to every date. A symbol absent from + ``listing_dates`` gets ``NaT`` (a disclosed data gap, never treated as young). + Pure: never mutates the input panel. + """ + validate_panel(panel) + out = panel.copy() + symbols = out.index.get_level_values("symbol") + out["list_date"] = [listing_dates.get(str(s), pd.NaT) for s in symbols] + return out diff --git a/data/feed/tushare_covariates.py b/data/feed/tushare_covariates.py index 39472d7..2258468 100644 --- a/data/feed/tushare_covariates.py +++ b/data/feed/tushare_covariates.py @@ -59,6 +59,26 @@ def industry(self, symbols: list[str]) -> dict[str, str]: df = df[df["ts_code"].astype(str).isin(wanted)] return {str(r.ts_code): r.industry for r in df.itertuples()} + def listing_dates(self, symbols: list[str]) -> dict[str, pd.Timestamp]: + """Return {symbol: list_date} from ``stock_basic.list_date`` (for UNI-008). + + Used by the ``min_listing_days`` selection filter. A symbol absent from + ``stock_basic`` simply does not appear in the map (the caller treats an + unknown listing date as a disclosed data gap, never as a young name). + """ + pro = self._client() + df = self._call(pro.stock_basic, fields="ts_code,list_date") + if df is None or len(df) == 0: + return {} + wanted = set(map(str, symbols)) + df = df[df["ts_code"].astype(str).isin(wanted)] + out: dict[str, pd.Timestamp] = {} + for r in df.itertuples(): + out[str(r.ts_code)] = pd.to_datetime( + str(r.list_date), format="%Y%m%d", errors="coerce" + ) + return out + def market_cap(self, symbols: list[str], start: str, end: str) -> pd.DataFrame: """Return DataFrame[date, symbol, market_cap] from daily_basic.total_mv.""" pro = self._client() diff --git a/qt/phase2_baseline.py b/qt/phase2_baseline.py index ad5adf3..e5a7596 100644 --- a/qt/phase2_baseline.py +++ b/qt/phase2_baseline.py @@ -11,9 +11,11 @@ turnover / cost · IC / quantile returns · performance · P2 downgrades. It reuses :mod:`qt.pipeline`'s step helpers verbatim (same universe / panel / -factor / neutralization / backtest), so "baseline == the real pipeline". The -extra reporting (holdings, filter hits, coverage) is **read-only**: it never -changes the strategy and never sees forward returns at the factor stage. +factor / neutralization / backtest), so "baseline == the real pipeline". +Per-period holdings are the ACHIEVED book recorded by the driver during the run +(``BacktestDriver.holdings_log()``, post execution-feasibility); the other extra +reporting (filter hits, ann_date coverage) is read-only diagnostics. Nothing here +changes the strategy or sees forward returns at the factor stage. Guard: the baseline REQUIRES ``data.source='tushare'``. Running it on the demo source is a category error (no PIT / ann_date / tradability meaning) and raises @@ -46,6 +48,7 @@ _make_logger, _maybe_enrich_covariates, _maybe_enrich_financials, + _maybe_enrich_listing, _periods_per_year, _process_factors, ) @@ -74,12 +77,17 @@ class Phase2Result: panel_symbols: int # universe / PIT membership universe_summary: dict + # min_listing_days list_date coverage (known vs disclosed data gap, this run) + list_date_known: int + list_date_total: int # ann_date financial coverage (diagnostic) financial_field: str financial_coverage_overall: float financial_coverage_by_rebalance: pd.DataFrame # tradability filter hits tradability_hits: pd.DataFrame + # execution feasibility (direction-aware fills) + feasibility_log: pd.DataFrame # rebalance + holdings (rebalance_dates are the SETTLED dates == nav_table.index) rebalance_dates: tuple[pd.Timestamp, ...] candidate_rebalance_dates: tuple[pd.Timestamp, ...] @@ -224,36 +232,6 @@ def tradability_hit_stats( return frame -def reconstruct_holdings( - scores: _FrameScores, - universe, - panel: pd.DataFrame, - constructor: TopNEqualWeight, - rebalance_dates: list[pd.Timestamp], -) -> pd.DataFrame: - """Rebuild per-period target holdings READ-ONLY (mirrors the driver's step). - - For each rebalance date: tradable members -> scores -> constructor.build, the - exact same chain :meth:`BacktestDriver._step` runs. We only read the resulting - target weights (the holdings); we never settle or see forward returns, so this - is a faithful, side-effect-free view of what the backtest held each period. - Returns a long-form frame with columns ``[date, symbol, weight, rank]``. - """ - rows: list[dict] = [] - for date in rebalance_dates: - target_date = pd.Timestamp(date).normalize() - tradable = list(universe.tradable(target_date, panel)) - if not tradable: - continue - cross_scores = scores.get(target_date, tradable) - weights = constructor.build(cross_scores) - for rank, (sym, w) in enumerate(weights.items(), start=1): - rows.append( - {"date": target_date, "symbol": str(sym), "weight": float(w), "rank": rank} - ) - return pd.DataFrame(rows, columns=["date", "symbol", "weight", "rank"]) - - def financial_coverage_at_dates( aligned_col: pd.Series, universe, @@ -306,9 +284,9 @@ def _phase2_downgrades(cfg: RootConfig, financial_field: str) -> tuple[str, ...] f"'{financial_field}' (how well disclosed reports populate the universe " "as-of); it is NOT the alpha factor in this baseline (the factor is the " "configured price factor). No financial signal is traded here.", - "Per-period holdings are RECONSTRUCTED read-only (universe -> scores -> " - "constructor.build), the same chain the backtest driver runs; they are a " - "faithful view, not an independent recomputation of returns.", + "Per-period holdings are the ACHIEVED book recorded by the backtest driver " + "(post execution-feasibility), NOT the constructor's desired target: a " + "blocked sell shows the carried name and a blocked buy is absent.", "This is a SMALL-SCALE baseline (a small index over a short window) for " "reproducibility/plumbing validation, NOT a performance claim; numbers are " "not optimized and must not be read as a strategy result.", @@ -347,13 +325,14 @@ def run_phase2_baseline(config_path: str) -> Phase2Result: factor = _build_factor(cfg) panel = _maybe_enrich_financials(cfg, panel, symbols, factor, logger) panel = _maybe_enrich_covariates(cfg, panel, symbols, logger) + panel = _maybe_enrich_listing(cfg, panel, symbols, logger) factor_panel = _compute_factor_panel(cfg, panel, factor, logger) processed = _process_factors(cfg, factor_panel, panel) score_panel = _build_scores(processed, EqualWeightAlpha()) scores = _FrameScores(score_panel) constructor = TopNEqualWeight(cfg.portfolio.top_n, long_only=cfg.portfolio.long_only) - execution = SimExecution(fee_rate=cfg.cost.fee_rate, cash_return=cfg.backtest.cash_return) + execution = SimExecution(fee_rate=cfg.cost.fee_rate) driver = BacktestDriver( universe=universe, scores=scores, @@ -367,6 +346,7 @@ def run_phase2_baseline(config_path: str) -> Phase2Result: ) candidate_dates = driver.rebalance_dates() nav_table = driver.run() + feasibility_log = driver.feasibility_log() # The diagnostics (coverage / filter hits / holdings) MUST key off the dates # that were actually held + settled — i.e. nav_table.index — NOT the candidate # rebalance dates. The driver skips a terminal rebalance with no forward @@ -411,7 +391,19 @@ def run_phase2_baseline(config_path: str) -> Phase2Result: hits = tradability_hit_stats( universe, panel, settled_dates, cfg.universe.filters.model_dump() ) - holdings = reconstruct_holdings(scores, universe, panel, constructor, settled_dates) + # ACHIEVED holdings (post-feasibility) from the driver — the actual book held, + # NOT the constructor's desired target (which differs once a trade is blocked). + holdings = driver.holdings_log() + + # list_date coverage for the min_listing_days disclosure (how many names had a + # known listing date this run vs a disclosed data gap). + if "list_date" in panel.columns: + _ld = panel["list_date"].groupby(level="symbol").first() + list_date_known = int(_ld.notna().sum()) + list_date_total = int(len(_ld)) + else: + list_date_known = 0 + list_date_total = 0 dates = panel.index.get_level_values("date") result = Phase2Result( @@ -425,10 +417,13 @@ def run_phase2_baseline(config_path: str) -> Phase2Result: universe_summary=summarize_universe( universe, cfg.universe.type, cfg.data.start, cfg.data.end ), + list_date_known=list_date_known, + list_date_total=list_date_total, financial_field=financial_field, financial_coverage_overall=coverage_overall, financial_coverage_by_rebalance=coverage_by_rebalance, tradability_hits=hits, + feasibility_log=feasibility_log, rebalance_dates=tuple(settled_dates), candidate_rebalance_dates=tuple(candidate_dates), skipped_terminal_dates=tuple(skipped_dates), diff --git a/qt/pipeline.py b/qt/pipeline.py index 63334e8..8bce489 100644 --- a/qt/pipeline.py +++ b/qt/pipeline.py @@ -37,7 +37,7 @@ from analytics.factor import compute_ic, forward_returns, ic_summary, quantile_returns from analytics.performance import performance_summary from data.clean.adjust import front_adjust -from data.clean.covariates import enrich_covariates +from data.clean.covariates import enrich_covariates, enrich_listing from data.clean.pit_financials import asof_financials from data.clean.tradability import enrich_tradability from data.feed.base import DataFeed @@ -233,18 +233,38 @@ def _collect_downgrades(cfg: RootConfig) -> tuple[str, ...]: else: neutral = "No neutralization in this run (raw cross-sectional factor)." + if real and cfg.universe.min_listing_days > 0: + listing = ( + f"universe.min_listing_days={cfg.universe.min_listing_days} is ENFORCED " + "(real path): names younger than that as of each date are excluded from " + "selection (UNI-008); a missing list_date is a disclosed data gap (kept, " + "never silently dropped)." + ) + else: + listing = ( + f"universe.min_listing_days={cfg.universe.min_listing_days} is NOT enforced " + "on this path (no listing dates on the demo source); disclosed no-op (UNI-008)." + ) + execution = ( + "Execution feasibility is direction-aware (UNI-007 / P2-2): at-up-limit blocks " + "buys, at-down-limit blocks sells, suspended/missing-close blocks both; blocked " + "trades carry forward and turnover/cost count only executed trades (no forced " + "impossible trades; idle cash from blocked buys earns cash_return). The demo " + "panel carries no flags, so every trade is feasible there (P0/P1 unchanged)." + ) + items = [ path, membership, financials, neutral, + execution, f"Daily ({cfg.data.freq}) bars only; minute-level link is deferred.", "IC / quantile returns use a simple numpy/pandas implementation, NOT " "alphalens-reloaded (simple-vs-alphalens fallback, INV-007).", "Performance metrics use a simple numpy/pandas implementation, NOT " "quantstats (simple-vs-quantstats fallback, INV-007).", - f"universe.min_listing_days is configured ({cfg.universe.min_listing_days}) " - "but NOT enforced (no-op); newly listed names are not excluded (INV-007).", + listing, ] return tuple(items) @@ -267,6 +287,7 @@ def run_phase0(config_path: str) -> Phase0Result: factor = _build_factor(cfg) panel = _maybe_enrich_financials(cfg, panel, symbols, factor, logger) panel = _maybe_enrich_covariates(cfg, panel, symbols, logger) + panel = _maybe_enrich_listing(cfg, panel, symbols, logger) factor_panel = _compute_factor_panel(cfg, panel, factor, logger) processed = _process_factors(cfg, factor_panel, panel) @@ -361,6 +382,10 @@ def _build_universe( backtest can settle names that later left the index (no survivorship bias). """ filters = cfg.universe.filters.model_dump() + # min_listing_days is a buy/selection-eligibility filter (UNI-008); thread it + # into the shared filter dict so both static and index universes enforce it + # when a list_date column is present (real path) and no-op otherwise (demo). + filters["min_listing_days"] = int(cfg.universe.min_listing_days or 0) if cfg.universe.type == "static": symbols = list(cfg.universe.symbols) if not symbols: @@ -527,6 +552,33 @@ def _maybe_enrich_financials( return enriched +def _maybe_enrich_listing( + cfg: RootConfig, panel: pd.DataFrame, symbols: list[str], logger: logging.Logger +) -> pd.DataFrame: + """Attach a per-symbol ``list_date`` column when min_listing_days is enforced. + + Real (tushare) path only: a demo run has no listing dates, so the + ``min_listing_days`` selection filter stays a DISCLOSED no-op there rather + than fabricating ages. A missing list_date is carried as NaT (data gap; the + filter keeps such names and the report discloses the count). + """ + if int(cfg.universe.min_listing_days or 0) <= 0: + return panel + if cfg.data.source != "tushare": + return panel # demo: no listing dates -> filter is a disclosed no-op + feed = TushareCovariatesFeed( + cfg.data.external_secret_file, token_key=cfg.data.tushare_token_key + ) + listing = feed.listing_dates(symbols) + panel = enrich_listing(panel, listing) + n_known = sum(1 for v in listing.values() if pd.notna(v)) + logger.info( + "listing: list_date enriched (%d/%d known) for min_listing_days=%d", + n_known, len(symbols), cfg.universe.min_listing_days, + ) + return panel + + def _compute_factor_panel( cfg: RootConfig, panel: pd.DataFrame, @@ -607,7 +659,7 @@ def _run_backtest( ) -> pd.DataFrame: """Wire the universe + scores + constructor + execution into the driver.""" constructor = TopNEqualWeight(cfg.portfolio.top_n, long_only=cfg.portfolio.long_only) - execution = SimExecution(fee_rate=cfg.cost.fee_rate, cash_return=cfg.backtest.cash_return) + execution = SimExecution(fee_rate=cfg.cost.fee_rate) driver = BacktestDriver( universe=universe, scores=_FrameScores(scores), diff --git a/qt/reports.py b/qt/reports.py index 41ef41a..6372947 100644 --- a/qt/reports.py +++ b/qt/reports.py @@ -127,6 +127,7 @@ def write_phase0_summary(result: "Phase0Result") -> Path: "## Universe / PIT membership", "## Financial ann_date coverage", "## Tradability filter hits", + "## Execution feasibility", "## Rebalance dates", "## Holdings per period", "## Turnover & cost", @@ -202,6 +203,35 @@ def _tradability_block(hits: pd.DataFrame) -> str: return head + table +def _feasibility_block(log: pd.DataFrame) -> str: + """Render the direction-aware execution-feasibility funnel (P2-2).""" + if log is None or log.empty: + return "_(no settled rebalances)_\n" + tot_bb = int(log["blocked_buys"].sum()) + tot_bs = int(log["blocked_sells"].sum()) + tot_cc = int(log["cash_constrained_buys"].sum()) + avg_inv = float(log["invested"].mean()) + head = ( + f"_Direction-aware fills: at-up-limit blocks buys, at-down-limit blocks " + f"sells, suspended/missing blocks both; blocked trades carry forward and " + f"turnover/cost count only executed trades._\n\n" + f"- total blocked buys: **{tot_bb}** · blocked sells: **{tot_bs}** · " + f"cash-constrained buys: **{tot_cc}**\n" + f"- avg invested fraction (1 − idle cash): **{_fmt(avg_inv)}**\n\n" + ) + table = ( + "| Rebalance date | blocked_buys | blocked_sells | carried | exec_turnover | invested |\n" + "|---|---|---|---|---|---|\n" + ) + for date, r in log.iterrows(): + table += ( + f"| {_date_str(date)} | {int(r['blocked_buys'])} | {int(r['blocked_sells'])} | " + f"{int(r['carried'])} | {_fmt(float(r['executed_turnover']))} | " + f"{_fmt(float(r['invested']))} |\n" + ) + return head + table + + def _holdings_block(holdings: pd.DataFrame) -> str: """Render per-period holdings (complete; one line per rebalance date).""" if holdings is None or holdings.empty: @@ -269,6 +299,13 @@ def render_phase2_baseline(result: "Phase2Result") -> str: lines.append("\n## Universe / PIT membership\n") lines.append(_universe_block(result.universe_summary)) + if result.list_date_total: + missing = result.list_date_total - result.list_date_known + lines.append( + f"- min_listing_days `list_date` coverage (this run): " + f"**{result.list_date_known}/{result.list_date_total}** known, " + f"**{missing}** missing (kept as a disclosed data gap, never excluded)\n" + ) lines.append("\n## Financial ann_date coverage\n") lines.append( @@ -282,6 +319,9 @@ def render_phase2_baseline(result: "Phase2Result") -> str: lines.append("\n## Tradability filter hits\n") lines.append(_tradability_block(result.tradability_hits)) + lines.append("\n## Execution feasibility\n") + lines.append(_feasibility_block(result.feasibility_log)) + lines.append("\n## Rebalance dates\n") reb = ", ".join(_date_str(d) for d in result.rebalance_dates) or "_(none)_" lines.append( @@ -301,8 +341,10 @@ def render_phase2_baseline(result: "Phase2Result") -> str: lines.append("\n## Holdings per period\n") lines.append( - "_Holdings are listed for the settled rebalance dates only, so they match " - "the NAV/turnover table 1:1 (no phantom terminal period)._\n\n" + "_ACHIEVED holdings (the actual book held after execution feasibility), NOT " + "the constructor's desired target: a name whose sell was blocked appears " + "carried here, a name whose buy was blocked is absent. Settled dates only, " + "so they match the NAV/turnover table 1:1._\n\n" ) lines.append(_holdings_block(result.holdings)) @@ -401,15 +443,25 @@ def render_bias_audit() -> str: "- **涨跌停(UNI-007)**:用**未复权 raw close** 与当日 raw `up_limit`/" "`down_limit` 比较,标记 `at_up_limit`/`at_down_limit`(qfq 复权价仅用于因子/" "回测收益;flag 富化在 front_adjust **之前**完成,故比较的是同口径 raw 价)。" - "实证:`000005.SZ` 2024-02-01 触跌停。当前选股层对两个方向都剔除;**方向感知**" - "(买入只看涨停、持有跌停不强卖)属执行层,后续细化。\n" + "实证:`000005.SZ` 2024-02-01 触跌停。\n" + "- **方向感知执行(UNI-007 / P2-2,已实现)**:选股层(`apply_tradable_filters`)" + "与执行层(`runtime.fills.simulate_fills`)**拆分**。执行可行性按 panel flag 实时" + "判定、与选股 toggle 无关:`at_up_limit` 挡**买入/加仓**,`at_down_limit` 挡" + "**卖出/减仓**,`suspended`/缺收盘价双向挡。被挡的交易 **carry forward**(绝不" + "强行成交不可能的单),现金一致 sell-then-buy(卖在前释放现金、买在后,现金不足" + "按比例部分成交,**无杠杆**),换手/成本只算**实际成交**,闲置现金按 `cash_return` " + "计息。demo 无 flag → 全可成交 → P0/P1 数字不变。每个调仓期的 blocked buys/sells/" + "carried/executed turnover 记入回测 feasibility log,phase2 报告有专门小节。\n" "- **停牌(UNI-005)**:`suspend_d` 标记停牌日。**实测发现**:tushare 全天停牌" "当日**无 bar** → 已被 `missing_close` 剔除,故显式 suspended flag 与之重叠;" "其价值在盘中停牌(`suspend_timing`)或会给停牌日 bar 的数据源,属防御性。\n" "- 退市 / 无数据标的(如 `000003.SZ`)同样表现为不在 panel 而被剔除。PIT 历史" "成分见上节。\n" - "- `universe.min_listing_days` 已在配置中(默认 60),但仍 **未执行**(no-op," - "降级):新上市标的不会被剔除。显式披露(INV-007),后续接上市日期后强制。\n\n" + "- **`universe.min_listing_days`(UNI-008,P2-2)**:作为**买入/选股资格**过滤。" + "**真实路径已执行**——从 tushare `stock_basic.list_date` 富化每只票上市日,某调仓日" + "`age < min_listing_days` 的新上市标的剔除(边界 `age == min` 放行);**缺 list_date " + "视为数据缺口,保留并披露**,绝不静默剔除。**demo 路径无上市日 → 仍 no-op(显式披露" + "的降级)**,不伪造上市日。\n\n" "## ann_date 财务对齐\n\n" "- 状态: **已实现(P1)**。\n" "- 财务因子(`roe` / `netprofit_yoy`)经 `data.clean.pit_financials.asof_financials` " diff --git a/runtime/backtest/driver.py b/runtime/backtest/driver.py index 357b4a4..c8d8967 100644 --- a/runtime/backtest/driver.py +++ b/runtime/backtest/driver.py @@ -29,6 +29,7 @@ import pandas as pd from runtime.execution import BacktestExecution +from runtime.fills import feasibility_from_cross class ScoresSource(Protocol): @@ -83,6 +84,8 @@ def __init__( self._fee_rate = float(fee_rate) self._initial_nav = float(initial_nav) self._cash_return = float(cash_return) + self._feasibility_log: list[dict] = [] + self._holdings_log: list[dict] = [] # -- calendar --------------------------------------------------------- # def _calendar(self) -> pd.DatetimeIndex: @@ -142,6 +145,8 @@ def run(self) -> pd.DataFrame: cal = self._calendar() rows: list[dict] = [] nav = self._initial_nav + self._feasibility_log = [] # re-entrant: fresh per run + self._holdings_log = [] for i, date in enumerate(reb): end = reb[i + 1] if i + 1 < len(reb) else cal[-1] # A rebalance on the final trading day has no forward holding window @@ -167,27 +172,113 @@ def run(self) -> pd.DataFrame: def _step( self, date: pd.Timestamp, end: pd.Timestamp ) -> tuple[float, float, float, float]: - """One rebalance: universe -> scores -> build -> rebalance -> settle. + """One rebalance: universe -> scores -> build -> feasible fill -> settle. + + Selection (``universe.tradable``) decides what the strategy WANTS; the + execution-feasibility flags (limits / suspension / missing, read from the + date's cross-section) decide what it can actually trade. Blocked trades + carry forward and turnover/cost count only executed trades (no forced + impossible trades). An empty tradable universe still routes through the + fill so held names that turned untradeable are carried, not force-sold. Returns ``(turnover, cost, gross_return, net_return)`` for the period held from ``date`` (exclusive) to ``end`` (inclusive). """ - tradable = list(self._universe.tradable(date, self._prices)) - if not tradable: - # Cash book: no trade, no turnover/cost, earns cash_return (BT-007). - # The driver owns cash semantics so cash_return is honoured even when - # the execution adapter's own cash_return differs. - self._execution.rebalance_to(pd.Series(dtype=float), date) - cost = self._execution.last_cost - gross = self._cash_return - return (self._execution.last_turnover, cost, gross, gross - cost) - scores = self._scores.get(date, tradable) current = self._execution.positions() - target = self._constructor.build(scores, current) - self._execution.rebalance_to(target, date) - holding = self._holding_returns(date, end, list(target.index)) - net = self._execution.settle(holding) + tradable = list(self._universe.tradable(date, self._prices)) + if tradable: + scores = self._scores.get(date, tradable) + target = self._constructor.build(scores, current) + else: + target = pd.Series(dtype=float) # nothing to hold -> exit what we can + + symbols = sorted(set(current.index) | set(target.index)) + can_buy, can_sell = self._feasibility(date, symbols) + self._execution.rebalance_to(target, date, can_buy=can_buy, can_sell=can_sell) + + achieved = self._execution.positions() + holding = self._holding_returns(date, end, list(achieved.index)) + invested_net = self._execution.settle(holding) + # The driver owns cash semantics (BT-007): the uninvested fraction — + # nonzero when blocked/cash-starved buys left cash idle — earns the + # driver's cash_return, even if the execution adapter's differs. + idle = 1.0 - float(achieved.sum()) + net = invested_net + idle * self._cash_return turnover = self._execution.last_turnover cost = self._execution.last_cost gross = net + cost + self._record_feasibility(date, achieved) + self._record_holdings(date, achieved) return (turnover, cost, gross, net) + + # -- execution feasibility ------------------------------------------- # + def _feasibility( + self, date: pd.Timestamp, symbols: list[str] + ) -> tuple[dict, dict]: + """Per-symbol (can_buy, can_sell) from the date's cross-section.""" + if not symbols: + return {}, {} + try: + cross = self._prices.xs(pd.Timestamp(date).normalize(), level="date") + except KeyError: + cross = self._prices.iloc[0:0] + return feasibility_from_cross(cross, symbols) + + def _record_feasibility(self, date: pd.Timestamp, achieved: pd.Series) -> None: + """Append the most recent fill's feasibility diagnostics to the log.""" + fill = getattr(self._execution, "last_fill", None) + if fill is None: + return + self._feasibility_log.append( + { + "date": date, + "blocked_buys": len(fill.blocked_buys), + "blocked_sells": len(fill.blocked_sells), + "cash_constrained_buys": len(fill.cash_constrained_buys), + "carried": len(fill.carried), + "executed_turnover": float(fill.executed_turnover), + "invested": float(achieved.sum()), + } + ) + + def feasibility_log(self) -> pd.DataFrame: + """Per-settled-rebalance execution-feasibility diagnostics (date-indexed). + + Columns: ``blocked_buys``, ``blocked_sells``, ``cash_constrained_buys``, + ``carried`` (counts), ``executed_turnover``, ``invested`` (sum of achieved + weights; ``1 - invested`` is idle cash). Aligns 1:1 with the NAV table's + settled rebalance dates. + """ + cols = [ + "blocked_buys", "blocked_sells", "cash_constrained_buys", + "carried", "executed_turnover", "invested", + ] + if not self._feasibility_log: + return pd.DataFrame(columns=cols, index=pd.Index([], name="date")) + return pd.DataFrame(self._feasibility_log).set_index("date") + + def _record_holdings(self, date: pd.Timestamp, achieved: pd.Series) -> None: + """Record the ACHIEVED book (post-feasibility) held for this period.""" + for sym, weight in achieved.items(): + self._holdings_log.append( + {"date": date, "symbol": str(sym), "weight": float(weight)} + ) + + def holdings_log(self) -> pd.DataFrame: + """Per-settled-rebalance ACHIEVED holdings (long-form date,symbol,weight,rank). + + These are the ACTUAL positions held after execution feasibility — a name + whose sell was blocked appears here carried at its old weight, and a name + whose buy was blocked is absent. This is the auditable book, NOT the + constructor's desired target (which can differ once a trade is blocked). + Ranked by weight desc then symbol for determinism; aligns with the NAV / + feasibility log's settled dates. + """ + cols = ["date", "symbol", "weight", "rank"] + if not self._holdings_log: + return pd.DataFrame(columns=cols) + df = pd.DataFrame(self._holdings_log).sort_values( + ["date", "weight", "symbol"], ascending=[True, False, True] + ) + df["rank"] = df.groupby("date").cumcount() + 1 + return df.reset_index(drop=True)[cols] diff --git a/runtime/backtest/sim_execution.py b/runtime/backtest/sim_execution.py index 835d73f..0bf382d 100644 --- a/runtime/backtest/sim_execution.py +++ b/runtime/backtest/sim_execution.py @@ -20,6 +20,7 @@ import pandas as pd from runtime.execution import BacktestExecution +from runtime.fills import FillResult, simulate_fills class SimExecution(BacktestExecution): @@ -33,25 +34,24 @@ class SimExecution(BacktestExecution): replaces the book. """ - def __init__(self, fee_rate: float = 0.0, cash_return: float = 0.0) -> None: + def __init__(self, fee_rate: float = 0.0) -> None: + # NOTE (P2-2): idle-cash return is owned by the backtest driver (BT-007), + # not this adapter, so SimExecution no longer takes a cash_return — settle() + # returns the invested book's return only and the driver adds idle*cash. if fee_rate < 0: raise ValueError(f"fee_rate must be >= 0, got {fee_rate!r}") self._fee_rate = float(fee_rate) - self._cash_return = float(cash_return) self._positions: pd.Series = pd.Series(dtype=float) self._last_turnover: float = 0.0 self._last_cost: float = 0.0 self._last_return: float = 0.0 + self._last_fill: FillResult | None = None # -- properties ------------------------------------------------------- # @property def fee_rate(self) -> float: return self._fee_rate - @property - def cash_return(self) -> float: - return self._cash_return - @property def last_turnover(self) -> float: """L1 turnover recorded by the most recent ``rebalance_to``.""" @@ -62,28 +62,38 @@ def last_cost(self) -> float: """Trading cost (turnover * fee_rate) of the most recent rebalance.""" return self._last_cost + @property + def last_fill(self) -> FillResult | None: + """Feasibility diagnostics of the most recent rebalance (or None).""" + return self._last_fill + # -- Execution port --------------------------------------------------- # - def rebalance_to(self, target_weights: pd.Series, date: pd.Timestamp) -> None: + def rebalance_to( + self, + target_weights: pd.Series, + date: pd.Timestamp, + *, + can_buy=None, + can_sell=None, + ) -> None: """Move the book toward ``target_weights`` as of the close of ``date``. - Computes full-L1 turnover against the current book over the union of - symbols, stores the resulting cost, and replaces the current positions - with a clean copy of the target. Does not mutate the input. The new book - is held from the next trading day (fixed event order). + Simulates only the FEASIBLE fills via :func:`runtime.fills.simulate_fills` + (cash-coherent sell-then-buy): blocked trades carry forward and turnover/ + cost count only what actually executed. With ``can_buy``/``can_sell`` both + ``None`` every trade is feasible, so the achieved book equals the target + and the L1 turnover is identical to a naive rebalance (offline/demo path + unchanged). Does not mutate the input; the new book is held from the next + trading day (fixed event order). """ target = self._clean_weights(target_weights) - current = self._positions - union = current.index.union(target.index) - aligned_target = target.reindex(union, fill_value=0.0) - aligned_current = current.reindex(union, fill_value=0.0) - turnover = float((aligned_target - aligned_current).abs().sum()) - self._last_turnover = turnover - self._last_cost = turnover * self._fee_rate - # Settle to the (gross) cost-only return for this rebalance until the - # holding period is settled; keeps last_return defined right after a - # rebalance with no holding info yet. + fill = simulate_fills(self._positions, target, can_buy, can_sell) + self._last_fill = fill + self._last_turnover = fill.executed_turnover + self._last_cost = fill.executed_turnover * self._fee_rate + # last_return is the cost-only return until the holding period settles. self._last_return = -self._last_cost - self._positions = target.copy() + self._positions = fill.achieved.copy() def positions(self) -> pd.Series: """Return current symbol-indexed weights (after the last rebalance).""" @@ -103,13 +113,16 @@ def settle(self, holding_returns: pd.Series | None) -> float: the gross return is ``cash_return``. Returns: - Net return = gross portfolio return - cost of forming this book. - Symbols in the book but missing from ``holding_returns`` contribute - zero (treated as flat). Stored as ``last_return``. + Net return of the INVESTED book = sum(book * returns) - cost. Symbols + in the book but missing from ``holding_returns`` contribute zero (flat), + and an empty book contributes zero invested return. The idle-cash + return on the uninvested fraction ``1 - sum(book)`` is owned by the + backtest driver (which honours its own ``cash_return``, BT-007), not by + this adapter. Stored as ``last_return``. """ book = self._positions if book.empty: - gross = self._cash_return + gross = 0.0 else: if holding_returns is None: rets = pd.Series(dtype=float) diff --git a/runtime/execution.py b/runtime/execution.py index b17c308..bb23d4e 100644 --- a/runtime/execution.py +++ b/runtime/execution.py @@ -17,7 +17,14 @@ class Execution(ABC): """Abstract execution port (backtest sim or live broker).""" @abstractmethod - def rebalance_to(self, target_weights: pd.Series, date: pd.Timestamp) -> None: + def rebalance_to( + self, + target_weights: pd.Series, + date: pd.Timestamp, + *, + can_buy=None, + can_sell=None, + ) -> None: """Move the book toward ``target_weights`` as of ``date``. Args: @@ -25,6 +32,12 @@ def rebalance_to(self, target_weights: pd.Series, date: pd.Timestamp) -> None: date: the rebalance date. Per the fixed event order, this is the close of ``date``; the new position is held from the next trading day. + can_buy / can_sell: optional per-symbol execution-feasibility maps + (mapping/set/Series). When supplied, a backtest adapter simulates + only the feasible fills (blocked trades carry forward) instead of + assuming every trade executes; ``None`` -> all feasible (the + offline/demo path is unchanged). A live adapter may use them as + pre-trade checks or ignore them. A backtest adapter records turnover and trading cost here; a live adapter would emit orders. Returns None. diff --git a/runtime/fills.py b/runtime/fills.py new file mode 100644 index 0000000..3e48ede --- /dev/null +++ b/runtime/fills.py @@ -0,0 +1,189 @@ +"""Execution-feasibility fill simulation (P2-2). + +Turns a DESIRED target portfolio into the ACHIEVED book given per-symbol buy/sell +feasibility, simulating what a broker would actually fill. This is the seam where +"selection" (what the strategy wants) meets "execution feasibility" (what the +market lets you trade): limits, suspension, and missing prices block trades, and +the backtest must NOT silently pretend an impossible trade happened. + +Cash-coherent sell-then-buy model (no leverage): + + 1. ``cash = 1 - sum(current)`` (uninvested fraction). + 2. SELLS first: for each name the target reduces, if it can be sold, execute the + reduction and add the proceeds to cash; otherwise carry the position forward + (a blocked sell — a forced hold). + 3. BUYS next: the desired increases for buyable names are funded from available + cash. If blocked sells starved the cash, the buys are scaled DOWN + proportionally so the book never sums to more than 1 (no leverage). A name + that cannot be bought is skipped (a blocked buy). + 4. Turnover counts only the trades actually executed. + +Feasibility is a market reality, not a config toggle: ``can_buy`` / ``can_sell`` +come from the panel flags (at-up-limit / at-down-limit / suspended / missing +close). A symbol absent from the maps defaults to feasible, so the offline demo +path (no flags) reduces to an exact ``achieved == target`` rebalance — P0/P1 +behaviour is unchanged. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field + +import pandas as pd + +_TOL = 1e-9 + + +@dataclass(frozen=True) +class FillResult: + """Outcome of one simulated rebalance (immutable).""" + + achieved: pd.Series + executed_turnover: float + blocked_buys: list[str] = field(default_factory=list) + blocked_sells: list[str] = field(default_factory=list) + cash_constrained_buys: list[str] = field(default_factory=list) + carried: list[str] = field(default_factory=list) + + @property + def cash_constrained(self) -> bool: + """True if any buyable name was scaled down for lack of cash.""" + return bool(self.cash_constrained_buys) + + +def _as_float_map(weights) -> dict[str, float]: + """Coerce a Series/dict of weights to a clean ``{symbol: float}`` map.""" + if weights is None: + return {} + s = pd.Series(weights, dtype=float).dropna() + return {str(k): float(v) for k, v in s.items()} + + +def _feasible(flag_map, symbol: str) -> bool: + """Resolve a per-symbol bool flag; absent symbol -> feasible (True).""" + if flag_map is None: + return True + if isinstance(flag_map, (set, frozenset)): + return symbol in flag_map + try: + return bool(flag_map.get(symbol, True)) + except AttributeError: # pandas Series or mapping-like without .get semantics + return bool(flag_map[symbol]) if symbol in flag_map else True + + +def feasibility_from_cross(cross: pd.DataFrame, symbols) -> tuple[dict, dict]: + """Derive per-symbol ``(can_buy, can_sell)`` from one date's cross-section. + + Feasibility is a market reality read off the panel flags, independent of the + selection-filter config toggles: + + * a symbol with no bar that day (absent, or NaN ``close``) -> cannot trade; + * ``suspended`` -> cannot trade either direction; + * ``at_up_limit`` -> cannot BUY (price pinned at the ceiling); + * ``at_down_limit`` -> cannot SELL (price pinned at the floor). + + Missing flag columns (e.g. the offline demo panel) default to not-flagged, so + every present, non-NaN name is fully tradable and the fill reduces to an exact + rebalance. Pure: never mutates ``cross``. + """ + can_buy: dict[str, bool] = {} + can_sell: dict[str, bool] = {} + for sym in symbols: + s = str(sym) + if s not in cross.index: + can_buy[s] = False + can_sell[s] = False + continue + row = cross.loc[s] + hard = bool(pd.isna(row.get("close", float("nan")))) or bool( + row.get("suspended", False) + ) + can_buy[s] = not (hard or bool(row.get("at_up_limit", False))) + can_sell[s] = not (hard or bool(row.get("at_down_limit", False))) + return can_buy, can_sell + + +def simulate_fills( + current, + target, + can_buy=None, + can_sell=None, + *, + tol: float = _TOL, +) -> FillResult: + """Simulate feasible fills moving ``current`` toward ``target``. + + Args: + current: symbol-indexed weights currently held (may sum to < 1; the rest + is cash). Empty == an all-cash book. + target: symbol-indexed desired weights (long-only; sums to ~1). + can_buy: per-symbol buy feasibility (mapping/set/Series). Absent -> True. + can_sell: per-symbol sell feasibility. Absent -> True. + tol: numerical tolerance for "no change". + + Returns: + A :class:`FillResult` with the achieved book, executed turnover, and the + blocked-buy / blocked-sell / cash-constrained / carried diagnostics. + """ + cur = _as_float_map(current) + tgt = _as_float_map(target) + symbols = sorted(set(cur) | set(tgt)) + + achieved = {s: cur.get(s, 0.0) for s in symbols} + cash = 1.0 - sum(cur.values()) + blocked_sells: list[str] = [] + blocked_buys: list[str] = [] + cash_constrained_buys: list[str] = [] + + # 1) SELLS (target < current): feasible reductions free cash; blocked = hold. + for s in symbols: + delta = tgt.get(s, 0.0) - cur.get(s, 0.0) + if delta < -tol: # want to reduce / exit + if _feasible(can_sell, s): + achieved[s] = tgt.get(s, 0.0) + cash += -delta # proceeds of the executed sell + else: + blocked_sells.append(s) # carried at cur[s] + + # 2) BUYS (target > current): fund from cash, scale down if starved. + desired_buys = { + s: tgt[s] - cur.get(s, 0.0) + for s in symbols + if tgt.get(s, 0.0) - cur.get(s, 0.0) > tol + } + buyable = {s: d for s, d in desired_buys.items() if _feasible(can_buy, s)} + blocked_buys = [s for s in desired_buys if not _feasible(can_buy, s)] + total_buy = sum(buyable.values()) + scale = 1.0 + if total_buy > tol and cash < total_buy - tol: + scale = max(0.0, cash) / total_buy + for s, d in buyable.items(): + exec_buy = d * scale + if exec_buy > tol: + achieved[s] = cur.get(s, 0.0) + exec_buy + cash -= exec_buy + # A buyable name scaled by a short cash budget (incl. fully starved -> 0) + # is cash-constrained: this records the REASON (cash shortage); a starved + # name also appears in `carried` below, which records the OUTCOME. The two + # answer different questions, so the (intentional) overlap is fine. + if scale < 1.0 - tol: + cash_constrained_buys.append(s) + + achieved = {s: w for s, w in achieved.items() if w > tol} + carried = [ + s for s in symbols if abs(achieved.get(s, 0.0) - tgt.get(s, 0.0)) > tol + ] + executed_turnover = sum( + abs(achieved.get(s, 0.0) - cur.get(s, 0.0)) for s in symbols + ) + + book = pd.Series(achieved, dtype=float) + book.index.name = "symbol" + return FillResult( + achieved=book.sort_index(), + executed_turnover=float(executed_turnover), + blocked_buys=sorted(blocked_buys), + blocked_sells=sorted(blocked_sells), + cash_constrained_buys=sorted(cash_constrained_buys), + carried=sorted(carried), + ) diff --git a/tests/test_bias_audit_report.py b/tests/test_bias_audit_report.py index 9128956..b914357 100644 --- a/tests/test_bias_audit_report.py +++ b/tests/test_bias_audit_report.py @@ -45,13 +45,22 @@ def test_bias_audit_records_known_phase0_limitations(tmp_path): assert "analytics" in text -def test_bias_audit_discloses_min_listing_days_noop(): - """min_listing_days is disclosed as a configured-but-unenforced no-op (LOW).""" +def test_bias_audit_discloses_min_listing_days_enforced_real_noop_demo(): + """min_listing_days is enforced on the real path; the demo no-op is disclosed (P2-2).""" text = render_bias_audit() assert "min_listing_days" in text - # Explicitly flagged as a no-op / downgrade, not silently ignored (INV-007). - assert "no-op" in text or "no op" in text.lower() - assert "降级" in text + # Real path enforces it via list_date; demo stays a disclosed no-op. + assert "list_date" in text + assert "已执行" in text # real path + assert "no-op" in text and "降级" in text # demo fallback, still disclosed + + +def test_bias_audit_discloses_direction_aware_execution(): + """Direction-aware execution feasibility (limits/suspension) is disclosed (P2-2).""" + text = render_bias_audit() + assert "方向感知" in text + assert "at_up_limit" in text and "at_down_limit" in text + assert "carry forward" in text or "carry" in text def test_bias_audit_discloses_missing_settlement_price_convention(): diff --git a/tests/test_driver_feasibility.py b/tests/test_driver_feasibility.py new file mode 100644 index 0000000..9232346 --- /dev/null +++ b/tests/test_driver_feasibility.py @@ -0,0 +1,133 @@ +"""End-to-end execution-feasibility through the BacktestDriver (P2-2). + +These drive a real :class:`BacktestDriver` over a tiny flagged panel and assert the +realism contract: a name at the down-limit cannot be sold (carried forward), a name +at the up-limit cannot be bought, suspended names cannot trade either way, and +turnover/cost count only the trades that actually executed — no forced impossible +trades. The per-rebalance feasibility log records the blocks. +""" + +from __future__ import annotations + +import pandas as pd + +from portfolio.construct import TopNEqualWeight +from runtime.backtest.driver import BacktestDriver +from runtime.backtest.sim_execution import SimExecution + +# Calendar spans Jan..early-Mar 2024 so the settled rebalances are Jan-end and +# Feb-end (the final March date is the terminal skip, BT-003). +_DATES = pd.bdate_range("2024-01-01", "2024-03-05") +_JAN, _FEB = pd.Timestamp("2024-01-31"), pd.Timestamp("2024-02-29") + + +class _Universe: + def members(self, date): + return ["A", "B"] + + def tradable(self, date, panel): + return ["A", "B"] + + +class _Scores: + """A is top in January, B is top in February (forces an A->B switch).""" + + def get(self, date, symbols): + d = pd.Timestamp(date).normalize() + pref = "A" if d <= _JAN else "B" + return pd.Series({s: (2.0 if s == pref else 1.0) for s in symbols}) + + +def _panel(flags: dict | None = None) -> pd.DataFrame: + rows = [] + for d in _DATES: + for s in ("A", "B"): + row = { + "date": d, "symbol": s, "close": 10.0, + "suspended": False, "at_up_limit": False, "at_down_limit": False, + } + if flags and (d, s) in flags: + row.update(flags[(d, s)]) + rows.append(row) + return pd.DataFrame(rows).set_index(["date", "symbol"]).sort_index() + + +def _driver(panel, fee_rate=0.001): + return BacktestDriver( + universe=_Universe(), + scores=_Scores(), + constructor=TopNEqualWeight(1), + execution=SimExecution(fee_rate=fee_rate), + prices=panel, + rebalance="monthly", + fee_rate=fee_rate, + initial_nav=1.0, + ) + + +def test_driver_blocked_sell_at_down_limit_carries_position(): + # On Feb-end A is at the down-limit -> the A->B switch can't sell A. + panel = _panel({(_FEB, "A"): {"at_down_limit": True}}) + driver = _driver(panel) + driver.run() + log = driver.feasibility_log() + feb = log.loc[_FEB] + assert feb["blocked_sells"] == 1 # A could not be sold + assert feb["executed_turnover"] == 0.0 # nothing executed (no cash for B) + assert feb["invested"] == 1.0 # A carried at full weight + + +def test_blocked_buy_at_up_limit_leaves_cash(): + # On Jan-end A is at the up-limit -> the initial buy of A is blocked. + panel = _panel({(_JAN, "A"): {"at_up_limit": True}}) + driver = _driver(panel) + driver.run() + log = driver.feasibility_log() + jan = log.loc[_JAN] + assert jan["blocked_buys"] == 1 + assert jan["invested"] == 0.0 # nothing bought -> all cash + assert jan["executed_turnover"] == 0.0 + + +def test_suspended_blocks_trading_both_ways(): + # A held from January; on Feb-end A is suspended -> cannot exit A, cannot + # enter B. Book is carried unchanged. + panel = _panel({(_FEB, "A"): {"suspended": True}}) + driver = _driver(panel) + driver.run() + feb = driver.feasibility_log().loc[_FEB] + assert feb["blocked_sells"] == 1 + assert feb["invested"] == 1.0 + assert feb["executed_turnover"] == 0.0 + + +def test_no_flags_executes_full_switch(): + # Sanity: with no blocks the A->B switch fully executes (turnover 2.0). + driver = _driver(_panel()) + driver.run() + feb = driver.feasibility_log().loc[_FEB] + assert feb["blocked_sells"] == 0 and feb["blocked_buys"] == 0 + assert feb["executed_turnover"] == 2.0 # sold A (1) + bought B (1) + assert feb["invested"] == 1.0 + + +def test_feasibility_log_aligns_with_nav_index(): + driver = _driver(_panel()) + nav = driver.run() + log = driver.feasibility_log() + assert list(log.index) == list(nav.index) # settled dates only, 1:1 + + +def test_holdings_log_reports_achieved_not_desired_target(): + # The auditability red-line (review HIGH): when A's sell is blocked at the + # down-limit, the DESIRED target is B but the ACHIEVED book is the carried A. + # holdings_log must report A (what was held), never B (what was wanted). + panel = _panel({(_FEB, "A"): {"at_down_limit": True}}) + driver = _driver(panel) + driver.run() + h = driver.holdings_log() + feb = h[h["date"] == _FEB] + assert list(feb["symbol"]) == ["A"] # carried (achieved), NOT desired B + assert "B" not in set(feb["symbol"]) + # and it equals the execution's actual end-of-run book (Feb is the last step) + assert set(feb["symbol"]) == set(driver._execution.positions().index) diff --git a/tests/test_fills.py b/tests/test_fills.py new file mode 100644 index 0000000..9393b72 --- /dev/null +++ b/tests/test_fills.py @@ -0,0 +1,140 @@ +"""Execution-feasibility fill simulation (P2-2 core, pure + network-free). + +``simulate_fills`` turns a DESIRED target into an ACHIEVED book given per-symbol +buy/sell feasibility, using a cash-coherent sell-then-buy model: + + * sells execute first (feasible ones), freeing cash; + * buys are funded from available cash, scaled down proportionally if blocked + sells starved them (no leverage — the book never sums to > 1); + * blocked trades carry the current position forward; + * turnover/cost count only the trades actually executed. + +These tests pin every branch the backtest driver relies on. +""" + +from __future__ import annotations + +import pandas as pd +import pytest + +from runtime.fills import feasibility_from_cross, simulate_fills + + +def _s(d: dict) -> pd.Series: + return pd.Series(d, dtype=float) + + +def _cross(rows: dict) -> pd.DataFrame: + """Build a symbol-indexed cross-section from {symbol: {col: val}}.""" + frame = pd.DataFrame.from_dict(rows, orient="index") + frame.index.name = "symbol" + return frame + + +def test_feasibility_from_cross_directional_flags(): + cross = _cross( + { + "OK": {"close": 10.0, "suspended": False, "at_up_limit": False, "at_down_limit": False}, + "UP": {"close": 10.0, "suspended": False, "at_up_limit": True, "at_down_limit": False}, + "DN": {"close": 10.0, "suspended": False, "at_up_limit": False, "at_down_limit": True}, + "SUS": {"close": 10.0, "suspended": True, "at_up_limit": False, "at_down_limit": False}, + "NAN": {"close": float("nan"), "suspended": False, "at_up_limit": False, "at_down_limit": False}, + } + ) + can_buy, can_sell = feasibility_from_cross(cross, ["OK", "UP", "DN", "SUS", "NAN", "ABSENT"]) + assert can_buy["OK"] and can_sell["OK"] + assert not can_buy["UP"] and can_sell["UP"] # up-limit: no buy, can sell + assert can_buy["DN"] and not can_sell["DN"] # down-limit: can buy, no sell + assert not can_buy["SUS"] and not can_sell["SUS"] # suspended: neither + assert not can_buy["NAN"] and not can_sell["NAN"] # no price: neither + assert not can_buy["ABSENT"] and not can_sell["ABSENT"] # no bar: neither + + +def test_feasibility_from_cross_missing_flag_columns_default_feasible(): + cross = _cross({"A": {"close": 10.0}, "B": {"close": 10.0}}) + can_buy, can_sell = feasibility_from_cross(cross, ["A", "B"]) + assert all(can_buy.values()) and all(can_sell.values()) + + +def test_all_feasible_achieves_target_exactly(): + # demo-equivalence: every trade feasible -> achieved == target, no leverage. + current = _s({"A": 0.5, "B": 0.5}) + target = _s({"C": 0.5, "D": 0.5}) + res = simulate_fills(current, target) + assert res.achieved.reindex(["C", "D"]).tolist() == [0.5, 0.5] + assert "A" not in res.achieved.index and "B" not in res.achieved.index + assert res.executed_turnover == pytest.approx(2.0) # sold 1.0, bought 1.0 + assert not res.blocked_buys and not res.blocked_sells + + +def test_first_period_from_cash_buys_full_target(): + res = simulate_fills(_s({}), _s({"A": 0.5, "B": 0.5})) + assert res.achieved.sum() == pytest.approx(1.0) + assert res.executed_turnover == pytest.approx(1.0) + + +def test_blocked_buy_at_up_limit_is_not_added(): + # want to buy C but C is at up-limit (can't buy) -> C not added; cash held. + current = _s({}) + target = _s({"C": 0.5, "D": 0.5}) + res = simulate_fills(current, target, can_buy={"C": False}) + assert "C" not in res.achieved.index + assert res.achieved.get("D", 0.0) == pytest.approx(0.5) + assert res.blocked_buys == ["C"] + assert res.achieved.sum() == pytest.approx(0.5) # rest is cash, no leverage + + +def test_blocked_sell_at_down_limit_carries_position(): + # want to exit A but A is at down-limit (can't sell) -> A carried forward. + current = _s({"A": 0.5, "B": 0.5}) + target = _s({"B": 1.0}) # exit A, double B + res = simulate_fills(current, target, can_sell={"A": False}) + assert res.achieved.get("A", 0.0) == pytest.approx(0.5) # forced hold + assert res.blocked_sells == ["A"] + assert "A" in res.carried + # no leverage: A's blocked sell freed no cash, so B can't grow past 0.5. + assert res.achieved.sum() == pytest.approx(1.0) + assert res.achieved.get("B", 0.0) == pytest.approx(0.5) + assert res.cash_constrained is True + + +def test_suspended_blocks_both_directions(): + current = _s({"A": 0.5, "B": 0.5}) + target = _s({"A": 1.0}) # want to increase A, exit B + res = simulate_fills( + current, target, can_buy={"A": False}, can_sell={"B": False} + ) + # A can't be bought (stays 0.5), B can't be sold (stays 0.5). + assert res.achieved.get("A", 0.0) == pytest.approx(0.5) + assert res.achieved.get("B", 0.0) == pytest.approx(0.5) + assert "A" in res.blocked_buys and "B" in res.blocked_sells + + +def test_executed_turnover_counts_only_executed_trades(): + # exit A (blocked), buy C (feasible but cash-starved by A's blocked sell). + current = _s({"A": 1.0}) + target = _s({"C": 1.0}) + res = simulate_fills(current, target, can_sell={"A": False}) + # A can't be sold -> no cash -> C can't be bought at all. + assert res.achieved.get("A", 0.0) == pytest.approx(1.0) + assert res.achieved.get("C", 0.0) == pytest.approx(0.0) + assert res.executed_turnover == pytest.approx(0.0) # nothing executed + + +def test_partial_fill_when_one_of_two_sells_blocked(): + # current A,B (0.5 each); target C,D (0.5 each). A can't sell, B can. + current = _s({"A": 0.5, "B": 0.5}) + target = _s({"C": 0.5, "D": 0.5}) + res = simulate_fills(current, target, can_sell={"A": False}) + # B sold -> 0.5 cash. Buys C,D want 1.0 total but only 0.5 cash -> scaled 0.5x. + assert res.achieved.get("A", 0.0) == pytest.approx(0.5) # carried + assert res.achieved.get("C", 0.0) == pytest.approx(0.25) + assert res.achieved.get("D", 0.0) == pytest.approx(0.25) + assert res.achieved.sum() == pytest.approx(1.0) # 0.5 + 0.25 + 0.25, no leverage + assert res.cash_constrained is True + + +def test_unknown_symbol_defaults_to_feasible(): + # a symbol absent from can_buy/can_sell maps to feasible (True). + res = simulate_fills(_s({}), _s({"Z": 1.0}), can_buy={"Y": False}) + assert res.achieved.get("Z", 0.0) == pytest.approx(1.0) diff --git a/tests/test_min_listing_days.py b/tests/test_min_listing_days.py new file mode 100644 index 0000000..d8043e9 --- /dev/null +++ b/tests/test_min_listing_days.py @@ -0,0 +1,68 @@ +"""universe.min_listing_days as a buy/selection-eligibility filter (P2-2). + +Newly-listed names (age < min_listing_days as of the cross-section date) are +excluded from selection. Boundaries: age < min excluded, age == min allowed, and +a missing ``list_date`` is handled honestly — NOT silently dropped (a data gap +must not shrink the universe) — and is disclosed by the caller. +""" + +from __future__ import annotations + +import pandas as pd + +from universe.filters import apply_tradable_filters + + +def _cross_panel(list_dates: dict, date: pd.Timestamp) -> pd.DataFrame: + """One-date panel with close + per-symbol list_date (NaT where unknown).""" + rows = [] + for sym, ld in list_dates.items(): + rows.append( + {"date": date, "symbol": sym, "close": 10.0, + "list_date": pd.Timestamp(ld) if ld is not None else pd.NaT} + ) + return pd.DataFrame(rows).set_index(["date", "symbol"]).sort_index() + + +def test_age_below_min_is_excluded(): + date = pd.Timestamp("2024-03-01") + panel = _cross_panel({"YOUNG": "2024-02-20"}, date) # ~10 days old + out = apply_tradable_filters(["YOUNG"], date, panel, {"min_listing_days": 60}) + assert out == [] + + +def test_age_at_min_is_allowed(): + date = pd.Timestamp("2024-03-01") + # exactly 60 days old -> allowed (boundary is inclusive). + panel = _cross_panel({"BORDER": date - pd.Timedelta(days=60)}, date) + out = apply_tradable_filters(["BORDER"], date, panel, {"min_listing_days": 60}) + assert out == ["BORDER"] + + +def test_age_above_min_is_allowed(): + date = pd.Timestamp("2024-03-01") + panel = _cross_panel({"OLD": "2010-01-01"}, date) + out = apply_tradable_filters(["OLD"], date, panel, {"min_listing_days": 60}) + assert out == ["OLD"] + + +def test_missing_list_date_is_kept_not_silently_dropped(): + date = pd.Timestamp("2024-03-01") + panel = _cross_panel({"UNKNOWN": None}, date) # NaT list_date + out = apply_tradable_filters(["UNKNOWN"], date, panel, {"min_listing_days": 60}) + assert out == ["UNKNOWN"] # data gap must not exclude the name + + +def test_zero_or_absent_min_is_noop(): + date = pd.Timestamp("2024-03-01") + panel = _cross_panel({"YOUNG": "2024-02-29"}, date) + assert apply_tradable_filters(["YOUNG"], date, panel, {"min_listing_days": 0}) == ["YOUNG"] + assert apply_tradable_filters(["YOUNG"], date, panel, {}) == ["YOUNG"] + + +def test_no_list_date_column_is_noop(): + # panel without a list_date column at all (e.g. demo) -> filter is a no-op. + date = pd.Timestamp("2024-03-01") + idx = pd.MultiIndex.from_tuples([(date, "X")], names=["date", "symbol"]) + panel = pd.DataFrame({"close": [10.0]}, index=idx) + assert apply_tradable_filters(["X"], date, panel, {"min_listing_days": 60}) == ["X"] diff --git a/tests/test_phase2_baseline.py b/tests/test_phase2_baseline.py index 74682bb..ee705ac 100644 --- a/tests/test_phase2_baseline.py +++ b/tests/test_phase2_baseline.py @@ -17,7 +17,6 @@ from qt.phase2_baseline import ( Phase2Result, financial_coverage_at_dates, - reconstruct_holdings, summarize_universe, tradability_hit_stats, ) @@ -184,33 +183,8 @@ def test_tradability_hit_stats_missing_close_counted(): # --------------------------------------------------------------------------- # -# reconstruct_holdings +# driver achieved holdings (the source the report uses) # --------------------------------------------------------------------------- # -def test_reconstruct_holdings_mirrors_topn_build(): - date = pd.Timestamp("2024-01-31") - syms = ["000001.SZ", "000002.SZ", "000003.SZ"] - panel = pd.DataFrame( - {"close": [1.0, 1.0, 1.0]}, - index=pd.MultiIndex.from_product([[date], syms], names=["date", "symbol"]), - ) - score_panel = pd.Series( - [0.3, 0.1, 0.2], - index=pd.MultiIndex.from_product([[date], syms], names=["date", "symbol"]), - name="score", - ) - uni = StaticUniverse(syms, {}) - holdings = reconstruct_holdings( - _FrameScores(score_panel), uni, panel, TopNEqualWeight(2), [date] - ) - assert len(holdings) == 2 # top_n=2 - held = set(holdings["symbol"]) - assert held == {"000001.SZ", "000003.SZ"} # two highest scores - assert holdings["weight"].tolist() == [0.5, 0.5] - # rank 1 is the highest score (000001.SZ @ 0.3) - top = holdings.sort_values("rank").iloc[0] - assert top["symbol"] == "000001.SZ" - - def test_holdings_dates_equal_settled_nav_index_not_candidates(): # Regression for the HIGH finding: the driver SKIPS a terminal rebalance with # no forward holding period, so the diagnostics must key off nav_table.index @@ -243,7 +217,7 @@ def test_holdings_dates_equal_settled_nav_index_not_candidates(): nav = driver.run() assert len(nav) == len(candidate_dates) - 1 # terminal date skipped (BT-003) - holdings = reconstruct_holdings(scores, uni, panel, constructor, list(nav.index)) + holdings = driver.holdings_log() # the ACHIEVED book the report uses assert set(holdings["date"]) == set(nav.index) assert candidate_dates[-1] not in set(holdings["date"]) # no phantom terminal @@ -291,6 +265,11 @@ def _synthetic_result() -> Phase2Result: cov = pd.DataFrame( {"date": [date], "n_members": [2], "n_covered": [1], "coverage": [0.5]} ) + feas = pd.DataFrame( + {"blocked_buys": [1], "blocked_sells": [0], "cash_constrained_buys": [0], + "carried": [1], "executed_turnover": [1.0], "invested": [0.5]}, + index=pd.Index([date], name="date"), + ) qret = pd.DataFrame({1: [0.01], 2: [0.02]}, index=[date]) return Phase2Result( config=cfg, @@ -301,10 +280,13 @@ def _synthetic_result() -> Phase2Result: panel_rows=2, panel_symbols=2, universe_summary=summarize_universe(_pit_universe(), "index", "2024-01-01", "2024-03-01"), + list_date_known=67, + list_date_total=68, financial_field="roe", financial_coverage_overall=0.5, financial_coverage_by_rebalance=cov, tradability_hits=hits, + feasibility_log=feas, rebalance_dates=(date,), candidate_rebalance_dates=(date, pd.Timestamp("2024-02-29")), skipped_terminal_dates=(pd.Timestamp("2024-02-29"),), @@ -343,3 +325,9 @@ def test_render_reports_holdings_and_rebalance_dates(): md = render_phase2_baseline(_synthetic_result()) assert "000001.SZ" in md and "000002.SZ" in md # holdings listed assert "2024-01-31" in md # rebalance date listed + + +def test_render_discloses_list_date_coverage_this_run(): + md = render_phase2_baseline(_synthetic_result()) + assert "67/68" in md # known/total for THIS run + assert "missing" in md.lower() # the data gap is disclosed, not just generic diff --git a/universe/filters.py b/universe/filters.py index 62eec76..9f6f0fe 100644 --- a/universe/filters.py +++ b/universe/filters.py @@ -36,6 +36,9 @@ def apply_tradable_filters( except KeyError: return [] # no market data for that date -> nothing tradable + min_days = int(flt.get("min_listing_days") or 0) + has_list_date = "list_date" in cross.columns + out: list[str] = [] for symbol in members: if symbol not in cross.index: @@ -51,5 +54,12 @@ def apply_tradable_filters( bool(row.get("at_up_limit", False)) or bool(row.get("at_down_limit", False)) ): continue + # min_listing_days (UNI-008): exclude names younger than min_days as of + # this date. A missing/NaT list_date is a DATA GAP, not a young name, so + # the name is kept (never silently dropped); callers disclose the gap. + if min_days > 0 and has_list_date: + ld = row.get("list_date") + if pd.notna(ld) and (target - pd.Timestamp(ld)).days < min_days: + continue out.append(symbol) return out