Skip to content

feat(analytics): P2-4 alphalens + quantstats report-only cross-check - #7

Merged
StackOverFlow11 merged 2 commits into
mainfrom
p2-standard-analytics
Jun 10, 2026
Merged

feat(analytics): P2-4 alphalens + quantstats report-only cross-check#7
StackOverFlow11 merged 2 commits into
mainfrom
p2-standard-analytics

Conversation

@StackOverFlow11

@StackOverFlow11 StackOverFlow11 commented Jun 9, 2026

Copy link
Copy Markdown
Owner

Summary

P2-4 wires alphalens-reloaded + quantstats into the reports as a report-only cross-check. No new factor, no parameter tuning; selection / alpha / portfolio / runtime / fills / universe are untouched, so the backtest trading numbers are unchanged. The simple numpy/pandas metrics remain the AUTHORITATIVE result and drive the run; the standard tools are shown alongside.

What changed

  • analytics/quantstats_adapter.py (new) — quantstats_performance(returns, periods_per_year) → CAGR / Sharpe / max-drawdown / volatility.
  • analytics/alphalens_adapter.py (new) — alphalens_factor_metrics(factor, prices, quantiles, period) → IC mean / IC-IR / per-quantile mean returns; suppresses alphalens' stdout banner / warnings.
  • Both use a _import_* indirection (so the ImportError path is unit-testable) and, on ImportError / runtime error, return backend = unavailable / error (exception TYPE only, never a message) and carry the simple-pandas fallback — never a silent fake (INV-007).
  • qt/pipeline.py _standard_analytics: computes both from the already-built nav / factor (report-only); Phase0Result / Phase2Result gain std_performance / std_factor; _collect_downgrades rewrites the IC/perf item to "simple authoritative + standard cross-check, backend disclosed".
  • qt/reports.py: a "Standard analytics" section in the phase0 and phase2 reports (added to the phase2 required-section contract).
  • Docs: RUNBOOK.md, TEST_REPORT.md, CLAUDE.md, AGENTS.md updated (the "simple, not alphalens/quantstats" downgrade is now "implemented as a report-only cross-check"). BIAS_AUDIT.md is unchanged — it documents biases, not analytics tooling.

A real bug caught + fixed before this PR

The first real run surfaced two quantstats-usage bugs (absurd values): CAGR −97% (quantstats cagr defaults to periods=252, treating monthly returns as daily) and maxDD 0.00% (quantstats max_drawdown wants the RETURNS series, not a prebuilt nav). Fixed: pass periods=periods_per_year to CAGR and the returns series to max_drawdown, plus a non-finite guard. alphalens was correct throughout.

Behaviour invariance + cross-check (verified)

  • Trading numbers unchanged: demo run-phase0 ic 0.96 / annual 0.84; real SSE50 baseline annual −10.19% (identical to P2-3) — the standard analytics never alter the run.
  • alphalens IC == simple IC exactly (0.96 demo; 0.0083 real SSE50) — a strong cross-check that the standard tool and our simple impl agree.

Real baseline (SSE50, manual, not in CI)

run-phase2-baseline OK (~14.6 min), backends quantstats + alphalens both ran. Authoritative (simple): annual −10.19%, maxDD −16.52%, vol 16.59%, Sharpe −0.57. quantstats cross-check: CAGR −15.6%, Sharpe −0.93, maxDD −21.9%, vol 16.8% (differs by convention — calendar-day CAGR, equity-curve-anchored maxDD — which is the point of a cross-check). alphalens: IC 0.0083, IC-IR 0.027, quantile means small/monotone-ish.

Quality gates

Gate Result
pytest -p no:cacheprovider 234 passed, 0 failed (P2-4 adds adapter 8 + phase0 additive/no-leak 2 + phase2 render 1)
ruff check . clean
validate-config × (example / example_tushare / phase2_real_baseline) all OK
run-phase0 (demo) OK (ic 0.96, unchanged)

Hygiene: no tushare token, conflict markers, or artifacts//*.parquet tracked.

Tests (network-free)

test_quantstats_adapter.py (4) + test_alphalens_adapter.py (4): metrics on synthetic input, unavailable / error fallback disclosure (no silent fake), alphalens stdout suppression. test_phase0_pipeline.py (+2): standard analytics are additive-not-replacing and leak no secret. test_phase2_baseline.py (+1): the cross-check section renders with named backends.

Review fixes (ae0674f)

Three LOW findings from review, all fixed:

  • Optional-extras test robustness: the 3 success-path adapter tests now pytest.importorskip quantstats/alphalens — a clean .[dev]-only env skips (disclosed) instead of failing; the fallback/disclosure tests monkeypatch the import and run everywhere. Verified both ways: libs shadowed → 5 passed / 3 skipped; quant_mf → 234 passed unchanged. TEST_REPORT.md documents the .[analytics] extra.
  • Docstring drift: analytics/factor.py / analytics/performance.py still said alphalens/quantstats "may be wired in later" — now state the simple metrics remain AUTHORITATIVE and the standard adapters live in analytics/*_adapter.py as a report-only cross-check.
  • Progress-line drift: CLAUDE.md / AGENTS.md P2-4 entries updated from "no PR yet" to "PR feat(analytics): P2-4 alphalens + quantstats report-only cross-check #7 OPEN, awaiting acceptance/merge".

…ck (P2-4)

Add thin, report-only adapters over alphalens-reloaded and quantstats so the
report shows the standard-tool metrics alongside the existing simple ones. This
is analytics/reporting only: no new factor, no parameter tuning, and selection /
alpha / portfolio / runtime / fills / universe are untouched — the backtest
trading numbers are unchanged.

- analytics/quantstats_adapter.py: quantstats_performance(returns, periods_per_year)
  -> CAGR / Sharpe / max-drawdown / volatility, tagged with the backend used.
- analytics/alphalens_adapter.py: alphalens_factor_metrics(factor, prices, quantiles,
  period) -> IC mean / IC-IR / per-quantile mean returns; suppresses alphalens'
  stdout banner / warnings.
- Both use a `_import_*` indirection so the ImportError path is unit-testable, and
  on ImportError / runtime error they return backend=unavailable/error (exception
  TYPE only, no message) and carry the simple-pandas fallback — never a silent fake
  (INV-007).
- qt/pipeline.py `_standard_analytics`: computes both from the already-built nav /
  factor (report-only); Phase0Result/Phase2Result gain std_performance/std_factor;
  `_collect_downgrades` rewrites the IC/perf items to "simple authoritative +
  standard cross-check, backend disclosed".
- qt/reports.py: a "Standard analytics" section in the phase0 and phase2 reports
  (added to the phase2 required-section contract).

Tests (234 passed, network-free): adapter metrics + unavailable/error fallback +
stdout suppression; phase0 run proves the standard analytics are additive, not
replacing (simple perf keeps its keys, report flags them authoritative), and no
secret leak. Empirically alphalens IC == simple IC on the demo (0.96) and the demo
trading numbers (ic 0.96 / annual 0.84) are unchanged. Docs (RUNBOOK, TEST_REPORT,
CLAUDE, AGENTS) updated; BIAS_AUDIT carries no analytics-tool claim (biases only).
… fix doc drift

Review LOW fixes on PR #7:

- tests: pytest.importorskip(quantstats/alphalens) on the 3 success-path
  adapter tests so a clean .[dev]-only env skips (disclosed) instead of
  failing; fallback/disclosure tests still monkeypatch and run everywhere.
  Verified: with both libs shadowed, 5 passed / 3 skipped; with quant_mf,
  234 passed unchanged.
- analytics/factor.py + performance.py: stale 'may be wired in later'
  docstrings -> simple metrics remain AUTHORITATIVE; standard adapters
  live in analytics/*_adapter.py as report-only cross-check (P2-4).
- CLAUDE.md / AGENTS.md progress: P2-4 is now 'PR #7 OPEN/MERGEABLE,
  awaiting acceptance/merge' (was stale 'no PR yet').
- TEST_REPORT.md: document the analytics optional extra and the skip
  behaviour for the full 234-passed run.
@StackOverFlow11
StackOverFlow11 merged commit 1ac3ccb into main Jun 10, 2026
StackOverFlow11 added a commit that referenced this pull request Jun 10, 2026
docs: mark PR #7 (P2-4) merged in progress docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant