feat(phase2): small-scale real-data reproducibility baseline (SSE50, auditable report) - #2
Merged
Merged
Conversation
Add a `run-phase2-baseline` run mode that runs the existing P0/P1 spine end-to-end on the real tushare path over a small universe (SSE50) + short window (~11 min) and emits a rich diagnostic report. No new factor, no parameter search: it validates the real-data plumbing and is reproducible. - qt/phase2_baseline.py: orchestration reusing the pipeline step helpers verbatim + read-only collectors (universe/PIT summary, tradability hit funnel, holdings reconstruction, ann_date coverage). Refuses the demo source before any network call (real-data only). - qt/reports.py: render_phase2_baseline + required-section contract. - qt/cli.py: run-phase2-baseline subcommand. - universe/index_universe.py: read-only membership_snapshots() accessor. - qt/pipeline.py: _make_logger gains an optional name (phase0/phase2 split). - config/phase2_real_baseline.yaml: SSE50 (000016.SH), 2023-07..2024-06. - tests/test_phase2_baseline.py: 12 network-free tests (collectors, demo guard, report-field contract, no-secret-leak, tradability first-match). Tradability hits use first-match-wins exclusive buckets mirroring apply_tradable_filters (sum(hits) == candidates - tradable). Holdings are reconstructed read-only (no forward returns at the factor stage). The tushare token is never read into the report; the report artifact stays git-ignored under artifacts/.
…ndow membership Two report/audit-cadence fixes found in review (report correctness, not strategy): 1. The diagnostics (ann_date coverage, tradability hits, per-period holdings) keyed off driver.rebalance_dates() (candidate dates), but BacktestDriver.run() SKIPS a terminal rebalance with no forward holding period (BT-003). The report therefore listed holdings for a date that had no NAV/turnover row (phantom period). Now all three diagnostics key off nav_table.index (the settled dates), so holdings match the NAV table 1:1; candidate dates and the skipped terminal date are disclosed separately. Regression test drives the real BacktestDriver over a synthetic panel whose last rebalance is the last trading day and asserts holdings.date == nav_table.index (terminal date absent). 2. summarize_universe folded the ~370-day pre-start lookback snapshots into the "over window" summary (first snapshot 2022-06-30 for a 2023-07 window). It now reports LOADED snapshots (incl. lookback, for transparency) separately from the IN-WINDOW membership (snapshots dated within [start,end] plus the as-of anchor at start); distinct-names / size / churn are over the in-window view. Test proves a name living only in a superseded pre-window snapshot is excluded. Real re-run: 11 settled holdings == 11 NAV rows; in-window distinct names 60 (vs 25 loaded snapshots); tradability funnel 550 member-days -> 549 tradable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2-1: a small-scale, real-data (tushare) reproducibility baseline. It runs the existing P0/P1 spine end-to-end on the real path over a small universe (上证50 / SSE50) and a ~1-year window, finishing in ~11 min, and emits a rich, auditable diagnostic report. No new factor, no parameter search, no live trading — the deliverable is a reproducible, self-auditing report, not a strategy result.
New surface:
qt/cli.py—run-phase2-baseline --config <path>subcommand (refuses the demo source; real-data only).qt/phase2_baseline.py— orchestration that reuses the pipeline step helpers verbatim, plus read-only collectors (PIT membership summary, ann_date coverage, tradability hit funnel, holdings reconstruction).qt/reports.py—render_phase2_baseline+ a required-section contract.universe/index_universe.py— read-onlymembership_snapshots()accessor.qt/pipeline.py—_make_loggergains an optionalname(phase0/phase2 log split).config/phase2_real_baseline.yaml— SSE50 (000016.SH), 2023-07..2024-06, momentum_20, all filters + neutralization on.Report contents (
artifacts/reports/phase2_real_baseline.md, git-ignored)Data window · PIT membership (loaded vs in-window) · ann_date as-of financial coverage · tradability filter-hit funnel · rebalance dates · per-period holdings · per-period turnover/cost · IC / quantile returns · performance summary · all P2 downgrades.
Real run (tushare; not in CI)
roecoverage 100% (a data-quality diagnostic, not the alpha factor).sum(hits) == candidates - tradable.Correctness guards
constructor.build), the same chain the driver runs; the factor stage never sees forward returns.run-phase2-baselineraises before any network call on a non-tushare source.nav_table.index(settled dates), so holdings match the NAV table 1:1; the terminal rebalance with no forward period (BT-003) is disclosed as skipped, not held.Review loop
A code-review pass flagged 1 HIGH (non-exclusive filter-hit counts) + 2 MEDIUM; a human audit pass flagged the settled-vs-candidate cadence (HIGH) and the lookback/in-window membership (MEDIUM). All were fixed and locked by tests before this PR (see commit
8de99a4).Test plan
pytest— 182 passed (P0=93 / P1=75 / P2-1=14), all network-free.ruff check .— clean.validate-config—example.yaml,example_tushare.yaml,phase2_real_baseline.yamlall OK.run-phase0 --config config/example.yaml— OK.run-phase2-baseline(manual, tushare) — 11 settled holdings == 11 NAV rows; report regenerated from final code.artifacts//*.parquettracked.