Skip to content

feat: Phase 0 MVP + Phase 1 bias-boundary (PIT · ann_date · tradability · neutralization) - #1

Merged
StackOverFlow11 merged 15 commits into
mainfrom
data
Jun 8, 2026
Merged

feat: Phase 0 MVP + Phase 1 bias-boundary (PIT · ann_date · tradability · neutralization)#1
StackOverFlow11 merged 15 commits into
mainfrom
data

Conversation

@StackOverFlow11

@StackOverFlow11 StackOverFlow11 commented Jun 7, 2026

Copy link
Copy Markdown
Owner

Summary

A reproducible A-share cross-sectional multi-factor framework, taken from a runnable
Phase 0 MVP to a Phase 1 "does-not-fool-itself" research framework: every major
cross-sectional bias boundary is implemented on the real (tushare) data path, validated
against real data, and disclosed. The offline demo path stays deterministic and network-free.

7 layers: data -> universe -> factors -> alpha -> portfolio -> runtime(backtest|live) -> analytics.
One command runs it end-to-end: python -m qt.cli run-phase0 --config config/example.yaml.

Phase 0 — runnable MVP

DemoFeed -> normalize_panel -> PanelStore(parquet) -> StaticUniverse -> momentum_20
-> z-score (by date) -> EqualWeightAlpha -> TopN equal-weight -> monthly-rebalance backtest
(cost + turnover) -> IC / performance -> markdown report. Live Execution port stays
minimal; backtest-only hooks live in a BacktestExecution sub-ABC ("backtest == live").

Phase 1 — bias boundary (the point of this PR)

Each is a correctness red-line, gated behind the real tushare path (on demo they raise a
readable error instead of fabricating), and disclosed in the run's DATA PATH / DOWNGRADES
section + BIAS_AUDIT.md:

Red-line Implementation
Front-adjust (qfq) data/clean/adjust.py — store stays raw + adj_factor, adjust in memory -> batch==incremental safe (returns anchor-invariant)
PIT index membership universe/index_universe.py + data/feed/index_feed.pyindex_weight as-of (latest snapshot <= date, survivorship-safe), 90-day paging, 370-day pre-start lookback
Tradability filters universe/filters.py + data/clean/tradability.py — suspend_d / namechange(ST) / stk_limit; limit flags compare raw close (enriched before front-adjust)
ann_date financials data/clean/pit_financials.py + FinancialFactor(roe/netprofit_yoy) — figure visible only after disclosure date (never end_date); ~16-month lookback so the prior report carries forward
Industry + size neutralization factors/process/neutralize.py — per-date OLS residual on [log(market_cap), one-hot(industry)]; saturated / no-DOF cross-sections -> NaN (no fabricated 0 residuals)
Path-aware disclosure _collect_downgrades leads with DATA PATH = DEMO vs REAL; a demo run can never be read as a real PIT/financial validation

Real-data validation (against tushare; not in CI — the suite stays network-free)

  • Front-adjust — 平安银行 2024-06-14 ex-dividend: raw -5.74% (artificial gap) vs qfq +0.99%; momentum_20 shifts up to 6.77pp.
  • PIT index — CSI300 2024: 24 snapshots, 328 distinct names (28 in / 28 out); members(2024-06-15) resolves to the 2024-06-03 snapshot; a dropped name stays a member for its era. (Fixed a real trap: index_weight caps ~6000 rows/call -> paged.)
  • ann_date — 平安银行 Q1 (end 2024-03-31, disclosed 2024-04-20): as-of roe stays the prior annual (10.24) until 04-19, switches to Q1 (3.12) on 04-22; an end_date join would have leaked it ~3 weeks early.
  • Neutralization — 12 names / 4 industries: corr(momentum, log market cap) -0.617 -> -0.000; per-industry residual means ~ 0.

Remaining P2 (disclosed, not hidden)

  • Industry tag is the current stock_basic.industry, not point-in-time -> mild industry-neutralization look-ahead.
  • Price-limit filter excludes both directions; trade-direction-aware handling is an execution-layer concern.
  • min_listing_days configured but not enforced (no-op).
  • Daily bars only; simple numpy/pandas IC + performance (not alphalens/quantstats); demo path is not real data.

Test plan

PY=/home/shaofl/Development/env_tools/envs/quant_mf/bin/python
$PY -m pytest -q                                                    # 168 passed
$PY -m ruff check .                                                 # clean
$PY -m qt.cli validate-config --config config/example.yaml          # exit 0
$PY -m qt.cli validate-config --config config/example_tushare.yaml  # exit 0 (real path)
$PY -m qt.cli run-phase0     --config config/example.yaml           # regenerates report

168 tests (P0 = 93, P1 = 75) incl. no-lookahead, ann_date-as-of, raw-close limit,
neutralize-DOF, path-disclosure regressions. Runtime env: conda quant_mf (py3.12).
No test hits the network or reads the token. Details in RUNBOOK.md, BIAS_AUDIT.md,
TEST_REPORT.md, artifacts/reports/phase1_summary.md.

- data/clean/adjust.py: front_adjust (qfq) anchored per-symbol to window-latest;
  store stays raw (+adj_factor), adjust in memory -> batch==incremental safe
  (returns are anchor-invariant for return-based factors).
- TushareFeed: wire rate_limit (calls/min) + retry with backoff (SEC-004).
- pipeline: front-adjust panel after read, before factor/backtest (demo no-op).
- BIAS_AUDIT: 复权 section now 'implemented' with the dividend evidence.
- 105 tests passed (+12).
- universe/filters.py: shared apply_tradable_filters (missing_close always +
  suspended/ST/at-limit when toggled and flags present); static + index reuse it.
- data/feed/tushare_flags.py: TushareFlagsFeed for suspend_d/namechange/stk_limit.
- data/clean/tradability.py: enrich panel with suspended/is_st/at_up_limit/
  at_down_limit bool flags.
- data/feed/secret.py: shared token reader (never echoed).
- pipeline: enrich flags only for tushare + when a flag filter is enabled (demo no-op).
- BIAS_AUDIT: tradability now implemented; honestly records that tushare full-day
  suspensions have no bar (already caught by missing_close) and the limit-direction
  nuance. 135 tests passed.
- data/clean/pit_financials.py: asof_financials joins financial fields by
  disclosure date (ann_date <= trade_date, backward merge_asof) — never end_date,
  so a report is invisible until announced. The core anti-lookahead red-line.
- data/feed/tushare_fina.py: TushareFinancialFeed (fina_indicator, ann_date+end_date).
- factors/compute/financial.py: FinancialFactor(roe|netprofit_yoy) surfacing the
  as-of column (no temporal logic; the alignment owns no-lookahead).
- pipeline: _build_factor dispatch (momentum vs financial); financial factors
  require tushare (demo source raises, no fabricated financials).
- BIAS_AUDIT: ann_date section now implemented with the 平安银行 Q1 evidence.
- 150 tests passed.
- factors/process/neutralize.py: per-date OLS residual of factor on
  [log(market_cap), one-hot(industry)]; degenerate/missing cross-sections -> NaN
  (no fabrication).
- ProcessingPipeline: neutralize step (order drop->winsorize->neutralize->zscore);
  enabled-but-missing-covariates raises a readable error.
- data/feed/tushare_covariates.py + data/clean/covariates.py: industry
  (stock_basic, current tag) + market_cap (daily_basic.total_mv) enrichment.
- pipeline: covariate enrichment + processor wiring; neutralize needs tushare.
- BIAS_AUDIT: neutralization section incl. the stock_basic current-industry PIT
  downgrade. Validated on 12 names/4 industries: corr(momentum,log_mcap)
  -0.617 -> -0.000. 161 tests passed.
- _collect_downgrades now leads with an explicit DATA PATH line (DEMO/offline vs
  REAL tushare) and reports membership / ann_date-financial / neutralization
  status per run, so a demo result can never be mistaken for a real PIT/financial
  validation and vice versa.
- config/example_tushare.yaml: documents the real path (tushare + PIT index +
  tradability filters + neutralization); validate-config accepts it.
- 164 tests passed.
- RUNBOOK: real-data (tushare) path section + example_tushare.yaml; P1
  implemented/deferred status.
- TEST_REPORT: 164 passed (exact per-file collect counts, P0=93 + P1=71) +
  real-data validation summary.
1. financials: fetch ~16 months before start (_FINANCIAL_LOOKBACK_DAYS=500) so the
   prior already-disclosed report carries forward onto early trade dates (no NaN
   gap), still ann_date-gated. Locked by carry-forward + lookback tests.
2. price-limit flags use RAW close vs raw stk_limit: tradability enrichment now runs
   BEFORE front-adjust, so qfq close is used only for factors/returns. Regression
   test with adj_factor != 1.
3. neutralize_by_date returns NaN for saturated / no-residual-DOF cross-sections
   (names <= 1 + #industries) instead of fabricated ~0 residuals.
4. BIAS_AUDIT / RUNBOOK / TEST_REPORT updated. 168 tests passed.
@StackOverFlow11 StackOverFlow11 changed the title feat: Phase 0 cross-sectional multi-factor MVP (end-to-end) feat: Phase 0 MVP + Phase 1 bias-boundary (PIT · ann_date · tradability · neutralization) Jun 8, 2026
tushare fina_indicator filters start_date/end_date by the report period
(end_date), not by ann_date. The previous docstring claimed ann_date
filtering / 'reports announced in the window', which is misleading: the
point-in-time ann_date<=trade_date alignment is done downstream in
asof_financials, not by this feed. Docstring only; no behavior change.
@StackOverFlow11

Copy link
Copy Markdown
Owner Author

验收门 / Acceptance gates — P1 bias-boundary

All gates re-run on env quant_mf (Py 3.12), repo root, current tree:

Gate Command Result
Unit + integration pytest -q 168 passed, 0 failed
Lint ruff check . All checks passed
Config validation (demo) validate-config --config config/example.yaml OK, exit 0
Config validation (real) validate-config --config config/example_tushare.yaml OK, exit 0
End-to-end (demo) run-phase0 --config config/example.yaml OK (ic_mean=0.96), report regenerated

Hygiene: no conflict markers; no tushare token in diff/repo; no artifacts//*.parquet tracked (gitignored); working tree clean apart from the docs commit below.

Last change in this pass (docs only, no behavior)

docs(feed): correct fina_indicator filter semantics — the TushareFinancialFeed.get_fina_indicator docstring claimed the tushare fina_indicator start_date/end_date filter by ann_date. It actually filters by the report period (end_date); the point-in-time ann_date <= trade_date alignment is done downstream in asof_financials. Docstring corrected; the gates above are unaffected (docs-only diff).

Note: this commit (3e7efae) is prepared locally and not yet pushed — auto-mode declined the push under the goal's "禁止 push force" boundary. Awaiting human confirmation before the branch is updated. The gates hold for both the current PR head and this commit.

Known P2 downgrades (honestly deferred, not bugs)

  • Demo path uses offline DemoFeed — not real data (no PIT/financial meaning); only the real tushare path carries the bias guarantees.
  • Industry tag is current (stock_basic.industry), not point-in-time → mild industry-neutralization look-ahead (historical PIT industry is P2).
  • min_listing_days configured but not enforced (no-op, disclosed).
  • Price-limit filter excludes both directions; trade-direction-aware handling (buy↔up-limit / hold↔down-limit) is an execution-layer concern (P2).
  • Full-day suspensions have no bar (already dropped by missing_close); the explicit suspended flag is defensive (intraday halts / other sources).
  • Daily bars only; simple numpy/pandas IC + performance (not alphalens/quantstats yet).
  • No factor combination beyond equal-weight; no portfolio optimization (by design for P1).

Status: ready for review. No merge / no force-push / no branch deletion performed — awaiting human confirmation.

@StackOverFlow11

Copy link
Copy Markdown
Owner Author

Update: 3e7efae (docstring fix) is now pushed — PR head is 3e7efae, so the gates above apply to the exact PR head. Ready for a normal merge at your discretion (no merge/force-push/branch-delete performed by me).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant