feat(analytics): P2-4 alphalens + quantstats report-only cross-check - #7
Merged
Conversation
…ck (P2-4) Add thin, report-only adapters over alphalens-reloaded and quantstats so the report shows the standard-tool metrics alongside the existing simple ones. This is analytics/reporting only: no new factor, no parameter tuning, and selection / alpha / portfolio / runtime / fills / universe are untouched — the backtest trading numbers are unchanged. - analytics/quantstats_adapter.py: quantstats_performance(returns, periods_per_year) -> CAGR / Sharpe / max-drawdown / volatility, tagged with the backend used. - analytics/alphalens_adapter.py: alphalens_factor_metrics(factor, prices, quantiles, period) -> IC mean / IC-IR / per-quantile mean returns; suppresses alphalens' stdout banner / warnings. - Both use a `_import_*` indirection so the ImportError path is unit-testable, and on ImportError / runtime error they return backend=unavailable/error (exception TYPE only, no message) and carry the simple-pandas fallback — never a silent fake (INV-007). - qt/pipeline.py `_standard_analytics`: computes both from the already-built nav / factor (report-only); Phase0Result/Phase2Result gain std_performance/std_factor; `_collect_downgrades` rewrites the IC/perf items to "simple authoritative + standard cross-check, backend disclosed". - qt/reports.py: a "Standard analytics" section in the phase0 and phase2 reports (added to the phase2 required-section contract). Tests (234 passed, network-free): adapter metrics + unavailable/error fallback + stdout suppression; phase0 run proves the standard analytics are additive, not replacing (simple perf keeps its keys, report flags them authoritative), and no secret leak. Empirically alphalens IC == simple IC on the demo (0.96) and the demo trading numbers (ic 0.96 / annual 0.84) are unchanged. Docs (RUNBOOK, TEST_REPORT, CLAUDE, AGENTS) updated; BIAS_AUDIT carries no analytics-tool claim (biases only).
… fix doc drift Review LOW fixes on PR #7: - tests: pytest.importorskip(quantstats/alphalens) on the 3 success-path adapter tests so a clean .[dev]-only env skips (disclosed) instead of failing; fallback/disclosure tests still monkeypatch and run everywhere. Verified: with both libs shadowed, 5 passed / 3 skipped; with quant_mf, 234 passed unchanged. - analytics/factor.py + performance.py: stale 'may be wired in later' docstrings -> simple metrics remain AUTHORITATIVE; standard adapters live in analytics/*_adapter.py as report-only cross-check (P2-4). - CLAUDE.md / AGENTS.md progress: P2-4 is now 'PR #7 OPEN/MERGEABLE, awaiting acceptance/merge' (was stale 'no PR yet'). - TEST_REPORT.md: document the analytics optional extra and the skip behaviour for the full 234-passed run.
StackOverFlow11
added a commit
that referenced
this pull request
Jun 10, 2026
docs: mark PR #7 (P2-4) merged in progress docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P2-4 wires alphalens-reloaded + quantstats into the reports as a report-only cross-check. No new factor, no parameter tuning; selection / alpha / portfolio / runtime / fills / universe are untouched, so the backtest trading numbers are unchanged. The simple numpy/pandas metrics remain the AUTHORITATIVE result and drive the run; the standard tools are shown alongside.
What changed
analytics/quantstats_adapter.py(new) —quantstats_performance(returns, periods_per_year)→ CAGR / Sharpe / max-drawdown / volatility.analytics/alphalens_adapter.py(new) —alphalens_factor_metrics(factor, prices, quantiles, period)→ IC mean / IC-IR / per-quantile mean returns; suppresses alphalens' stdout banner / warnings._import_*indirection (so the ImportError path is unit-testable) and, onImportError/ runtime error, returnbackend = unavailable / error(exception TYPE only, never a message) and carry the simple-pandas fallback — never a silent fake (INV-007).qt/pipeline.py_standard_analytics: computes both from the already-built nav / factor (report-only);Phase0Result/Phase2Resultgainstd_performance/std_factor;_collect_downgradesrewrites the IC/perf item to "simple authoritative + standard cross-check, backend disclosed".qt/reports.py: a "Standard analytics" section in the phase0 and phase2 reports (added to the phase2 required-section contract).RUNBOOK.md,TEST_REPORT.md,CLAUDE.md,AGENTS.mdupdated (the "simple, not alphalens/quantstats" downgrade is now "implemented as a report-only cross-check").BIAS_AUDIT.mdis unchanged — it documents biases, not analytics tooling.A real bug caught + fixed before this PR
The first real run surfaced two quantstats-usage bugs (absurd values): CAGR −97% (quantstats
cagrdefaults toperiods=252, treating monthly returns as daily) and maxDD 0.00% (quantstatsmax_drawdownwants the RETURNS series, not a prebuilt nav). Fixed: passperiods=periods_per_yearto CAGR and the returns series tomax_drawdown, plus a non-finite guard. alphalens was correct throughout.Behaviour invariance + cross-check (verified)
run-phase0ic 0.96 / annual 0.84; real SSE50 baseline annual −10.19% (identical to P2-3) — the standard analytics never alter the run.Real baseline (SSE50, manual, not in CI)
run-phase2-baselineOK (~14.6 min), backendsquantstats+alphalensboth ran. Authoritative (simple): annual −10.19%, maxDD −16.52%, vol 16.59%, Sharpe −0.57. quantstats cross-check: CAGR −15.6%, Sharpe −0.93, maxDD −21.9%, vol 16.8% (differs by convention — calendar-day CAGR, equity-curve-anchored maxDD — which is the point of a cross-check). alphalens: IC 0.0083, IC-IR 0.027, quantile means small/monotone-ish.Quality gates
pytest -p no:cacheproviderruff check .validate-config× (example / example_tushare / phase2_real_baseline)run-phase0(demo)Hygiene: no tushare token, conflict markers, or
artifacts//*.parquettracked.Tests (network-free)
test_quantstats_adapter.py(4) +test_alphalens_adapter.py(4): metrics on synthetic input, unavailable / error fallback disclosure (no silent fake), alphalens stdout suppression.test_phase0_pipeline.py(+2): standard analytics are additive-not-replacing and leak no secret.test_phase2_baseline.py(+1): the cross-check section renders with named backends.Review fixes (ae0674f)
Three LOW findings from review, all fixed:
pytest.importorskipquantstats/alphalens — a clean.[dev]-only env skips (disclosed) instead of failing; the fallback/disclosure tests monkeypatch the import and run everywhere. Verified both ways: libs shadowed → 5 passed / 3 skipped;quant_mf→ 234 passed unchanged.TEST_REPORT.mddocuments the.[analytics]extra.analytics/factor.py/analytics/performance.pystill said alphalens/quantstats "may be wired in later" — now state the simple metrics remain AUTHORITATIVE and the standard adapters live inanalytics/*_adapter.pyas a report-only cross-check.