Skip to content

feat(neutralize): P2-3 point-in-time SW industry covariate (replace current-tag) - #5

Merged
StackOverFlow11 merged 4 commits into
mainfrom
p2-pit-industry
Jun 9, 2026
Merged

feat(neutralize): P2-3 point-in-time SW industry covariate (replace current-tag)#5
StackOverFlow11 merged 4 commits into
mainfrom
p2-pit-industry

Conversation

@StackOverFlow11

@StackOverFlow11 StackOverFlow11 commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

P2-3 upgrades the industry neutralization covariate from the CURRENT stock_basic.industry tag (broadcast to every date — a membership look-ahead) to a point-in-time, as-of-trade-date SW industry, with the SW level configurable (processing.neutralize.industry_level, L1/L2/L3, default L1). No new factor, no parameter tuning; the framework layering and the neutralize math are unchanged (it already drops NaN-industry rows).

What changed

  • data/feed/tushare_covariates.pypit_sw_intervals(symbols, level) reads tushare index_member_all for each symbol's SW membership intervals at the chosen level (l1/l2/l3_name, in_date/out_date; out_date None = active). The industry() current-tag accessor is retained but no longer wired into neutralization.
  • data/clean/pit_industry.py (new) — asof_industry(index, intervals) picks, per (trade_date, symbol), the industry whose [in_date, out_date) covers that date (new industry wins on a reclassification date — PIT-safe; pre-start membership carries forward; no covering interval / unknown symbol → NaN).
  • data/clean/covariates.pyenrich_pit_industry(panel, industry_series) sets a per-date industry column.
  • qt/config.pyprocessing.neutralize.industry_level (L1/L2/L3, default L1).
  • qt/pipeline.py_maybe_enrich_covariates builds the PIT industry at the configured level; logs coverage. A name with no SW history stays NaN — a disclosed coverage gap the neutralizer drops — never a silent current-tag fallback. _collect_downgrades rewritten (PIT, configurable level, UNI-010).
  • qt/phase2_baseline.py + qt/reports.py — the phase2 report discloses the actual SW level + PIT industry coverage.
  • DocsBIAS_AUDIT.md (regenerated), RUNBOOK.md, TEST_REPORT.md, CLAUDE.md, AGENTS.md.

Real-data finding (this is the important part)

Going PIT is not behaviour-neutral, and a real L1-vs-L2 comparison disproved the initial "L2 isolates granularity" hypothesis. On the SSE50 baseline (2023-07..2024-06, momentum_20):

Industry source buckets annual return Sharpe
old stock_basic.industry (current tag) ~100 (tushare's own) −17.6% −0.96
PIT SW-L1 31 −10.2% −0.57
PIT SW-L2 ~130 −9.3% −0.50

(Raw-factor IC is unchanged at 0.0083 — neutralization affects the portfolio via the processed scores, not the raw IC.)

L1 ≈ L2, so the −17.6 → −10 jump is the tushare → SW taxonomy switch, which is inherent to going point-in-time: only the SW classification carries the in_date/out_date history needed for an as-of alignment; the old stock_basic.industry tag cannot be PIT-aligned at all. It is not a granularity choice.

Default = SW-L1 (31 broad sectors): the standard granularity for industry neutralization and the DOF-safest on small cross-sections (~50 names/cross-section on SSE50; L2's ~130 sub-industries approach saturation). L2/L3 remain available via config.

Real baseline (SSE50, manual, not in CI)

Default SW-L1: run-phase2-baseline OK — 68 symbols, 11 settled rebalances, PIT SW-L1 coverage 98.53% (67/68; 1 symbol has no SW history → NaN, disclosed), annual return −10.2%. The report regenerates with the actual level + coverage line.

Quality gates

Gate Result
pytest -p no:cacheprovider 223 passed, 0 failed (P0=95 / P1=79 / P2-1=15 / P2-2=22 / P2-3=12)
ruff check . clean
validate-config × (example / example_tushare / phase2_real_baseline) all OK
run-phase0 (demo) OK (P0/P1 numbers unchanged — neutralization is off on demo)

Hygiene: no tushare token, conflict markers, or artifacts//*.parquet tracked.

Tests (network-free)

test_pit_industry.py (12): as-of switch / carry-forward / missing→NaN / overlap, enrich_pit_industry, pipeline wiring (per-date industry, configured level passed, no current-tag fallback), and industry_level config (default L1, accepts L1/L2/L3, rejects invalid, phase2 config = L1). + level-selective feed parse and report-level disclosure in the covariates / real-path / bias-audit / phase2 tests.

…t-tag broadcast)

P2-3 upgrades the industry neutralization covariate from the CURRENT
`stock_basic.industry` tag (broadcast to every date — a mild membership
look-ahead) to a POINT-IN-TIME, as-of-trade-date SW-L1 industry. The framework's
layering is unchanged; the neutralize math is untouched (it already drops
NaN-industry rows).

- data/feed/tushare_covariates.py: `pit_sw_l1_intervals(symbols)` reads tushare
  `index_member_all` for each symbol's SW-L1 membership intervals (in_date/out_date;
  out_date None = active).
- data/clean/pit_industry.py (new): `asof_industry(index, intervals)` picks, per
  (trade_date, symbol), the industry whose `[in_date, out_date)` covers that date
  (new industry wins on a reclassification date — PIT-safe; pre-start membership
  carries forward; no covering interval / unknown symbol -> NaN).
- data/clean/covariates.py: `enrich_pit_industry(panel, industry_series)` sets a
  per-date `industry` column from the as-of Series.
- qt/pipeline.py: `_maybe_enrich_covariates` now builds the PIT industry (real path)
  instead of the current tag; logs coverage. A name with no SW history stays NaN —
  a disclosed coverage gap the neutralizer drops — NEVER a silent current-tag
  fallback. `_collect_downgrades` neutralization text rewritten (PIT, UNI-010).
- qt/phase2_baseline.py + qt/reports.py: report discloses PIT industry coverage
  (fraction of (date, symbol) rows with an as-of industry).
- Docs: BIAS_AUDIT.md (regenerated), RUNBOOK.md, TEST_REPORT.md, CLAUDE.md.

Tests (217 passed, network-free): test_pit_industry.py (8) — as-of switch /
carry-forward / missing->NaN / overlap / enrich / pipeline wiring; +1 each to
test_tushare_covariates (index_member_all parse), test_real_path_config (PIT
downgrade text, not "current"), test_bias_audit_report, test_phase2_baseline
(PIT coverage rendered). The current-tag `industry()` feed method remains
(tested) but is no longer wired into neutralization.
…t L1)

Follow-up to the P2-3 PIT industry change, after a real L1/L2 comparison.

- config: add `processing.neutralize.industry_level` (L1/L2/L3, default L1);
  phase2_real_baseline.yaml sets it explicitly. `pit_sw_l1_intervals` is
  generalized to `pit_sw_intervals(symbols, level)` (reads l1/l2/l3_name from
  index_member_all); pipeline passes the configured level; report / downgrades /
  BIAS_AUDIT disclose the actual level (no hardcoded SW-L1).

Why L1 default (corrected rationale): a real comparison showed the SSE50 annual
return is old stock_basic.industry −17.6%, SW-L1 −10.2%, SW-L2 −9.3% — L1 ≈ L2.
So the −17.6→−10 change is the tushare→SW TAXONOMY switch, which is INHERENT to
going point-in-time (only SW carries in/out-date history; the old tag cannot be
PIT-aligned), NOT an industry-granularity choice. The earlier "L2 ≈ old
granularity isolates the PIT fix" rationale was disproven by the L2 run. L1 (31
broad sectors) is the standard granularity for industry neutralization and the
DOF-safest on small cross-sections, so it is the default; L2/L3 remain available.

Tests (223 passed): industry_level config (default L1, accepts L1/L2/L3, rejects
invalid, phase2 config = L1), level-selective feed parse (L1/L2/L3), pipeline
passes the configured level, report discloses the level. Docs (BIAS_AUDIT
regenerated, RUNBOOK, TEST_REPORT, CLAUDE, AGENTS) carry the corrected taxonomy-
vs-granularity finding.
… default L1'

Comment/docstring-only (no behavior): feed module docstring (default L2 -> L1),
pit_industry module docstring (SW-L1 / old pit_sw_l1_intervals -> SW level-configurable /
pit_sw_intervals), test comments (default L2 -> L1), and phase2 coverage comments
(PIT SW-L1 -> PIT SW industry coverage). The runtime already passes the configured
industry_level (default L1).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant