feat(validation): P3-4 robustness matrix (universes x windows OOS cells)

[StackOverFlow11]

Summary

P3-4 — robustness matrix: longer history + wider universe. Re-runs the P3-3 OOS stability check on every (universe × window) cell to answer whether the P3-2/P3-3 conclusions were an SSE50 small-sample accident. No new factor, no new alpha, no portfolio / execution / factor-math change — a report-only validation layer on top of the P3-3 framework.

Implementation

• Runner qt/robustness.py + CLI run-phase3-robustness: every cell reuses the P3-3 cell core verbatim — _run_oos_cell was refactored out of run_phase3_oos (single-run behaviour unchanged, its 16 tests untouched), so every cell inherits holding-window performance slicing, realization-date IC slicing, walk-forward weights, and the shared guards (real source / oos section / ic_weighted fake-comparison guard).
• Cell derivation: per-cell configs = the base config with ONLY the cell identity swapped (universe.index_code, data window, split, per-cell output_name so panels never collide); the derived config re-runs full pydantic validation.
• Explicit coverage: robustness.skip_cells removes named cells (validated against declared universes/windows; ≥1 cell must remain) and every skip is disclosed in the report — coverage is never silently reduced. This run skips CSI300×2020-2022 (runtime budget; a CSI300 2y cell alone is ~88 min through the rate-limited SDK).
• Aggregation (summarize_matrix): strictly per cell (no pooling/mixing — locked by tests): per-series #cells with positive test IC + train→test sign consistency + test IC by cell; #cells where ic_weighted beats equal_weight on test annual.

Real matrix (3 cells, 2h: 960s + 934s + 5301s; CSI300 = 399 historical names)

P3-3 no-drift smoke ✓: the SSE50|2022-2024 cell reproduces the P3-3 report number-for-number (eq train −11.92% / test −5.27%, ic train −8.31% / test −2.70%, boundary 2023-06-30 excluded + disclosed).

Cross-cell stability summary:

Series	test IC > 0	train→test sign consistent
momentum_20	3/3	1/3
roe	2/3	0/3
netprofit_yoy	2/3	0/3
combo_equal_weight	1/3	1/3
combo_ic_weighted	3/3	3/3 (but IC ≈ 0.004–0.008, tiny)

Per-cell test-period annual (eq / ic): SSE50|2020-2022 −8.09% / −7.85% · SSE50|2022-2024 −5.27% / −2.70% · CSI300|2022-2024 −5.14% / −17.74%. ic_weighted beats equal_weight in 2/3 cells — but see finding 3.

Findings (the honest read)

1. Raw-factor IC sign flips are pervasive across cells (momentum 1/3, roe 0/3, np_yoy 0/3 consistent) — the P3-3 instability finding was NOT an SSE50 accident.
2. combo_ic_weighted is the only cross-cell-consistent series — but at IC ≈ 0.004–0.008, far below tradable strength.
3. Key negative finding: ic_weighted portfolio performance does NOT generalize. The train→test collapse shape appears in 2 of 3 cells: SSE50|2020-2022 train +7.09% → test −7.85% and CSI300|2022-2024 train +11.07% → test −17.74% (sharpe +0.59 → −1.01; 1.2–1.6 turnover adds cost drag on the wider universe). Whenever ic_weighted is strongly positive in-sample, it flips negative out-of-sample — the classic overfit signature. The P3-2 one-year edge does not extrapolate. Weak positive IC ≠ nav win. NOT a return claim, not a tuned result.

Quality gates

Gate	Result
pytest -p no:cacheprovider	299 passed (was 285: +14 matrix)
ruff check .	clean
validate-config ×7 (all old + robustness_matrix)	OK
run-phase0 demo	ic 0.96 / annual 0.84 unchanged
P3-3 no-drift	smoke cell reproduces P3-3 numbers exactly
Real matrix	3 cells run, 1 skipped + disclosed, runtimes in report
Secret scan	report + tree contain no token / .config.json content
Hygiene	artifacts git-ignored, no conflict markers

Tests (network-free)

test_robustness_matrix.py (14): matrix config validation (split-inside-window, unknown-skip, all-skipped), cell enumeration excludes skips in order, cell derivation swaps only the cell identity + unique output_name + passes shared preconditions, strictly-per-cell aggregation (no mixing, no dilution), runner guards (demo / missing section / non-ic_weighted), report render (cell labels, skipped-cell disclosure, boundary/fallback/sign-consistency, summary, caveat, no secret). P3-3's 16 tests pass unchanged on the refactored core.

Docs

RUNBOOK (P3-4 section), TEST_REPORT (299 + breakdown), CLAUDE.md / AGENTS.md (verbatim copy) progress incl. the negative finding and the roadmap implication (the current factor set lacks tradable signal strength — iterate the factor set inside this validation framework, or move to minute-level).

• MEDIUM fix (0fcfd58): the matrix DOWNGRADES section no longer copies only the first cell's disclosures — it opens with a MATRIX SCOPE line (run cells / skipped cells / universes / windows) and renders the union of ALL cells' disclosures (each universe's PIT-membership line present); the "one index, two years" caveat is reworded to per-cell-accurate phrasing. Locked by a render test asserting BOTH universes appear and the old wording does not. The artifacts report is git-ignored; the next run emits the corrected disclosures.