feat(factors): D2 — minute primitives, ops operators, 11-factor migration (bit-identical to the frozen baseline) - #89
Merged
Conversation
…y factor rolling on them D2 commit A (design v3.2 §3.2): factors/ops/timeseries.py is the one home of the per-symbol lag/rolling machinery the daily factors used to inline. Three conventions locked by tests: strictly per-symbol grouping, full-window leading-NaN semantics, purity. Only the operators the rewrite consumes exist (ts_lag / ts_window_return / ts_pct_change / ts_std / ts_mean / ts_sum / log_positive) — no speculative cs_rank/median, no string-dispatch registry. momentum / volatility / liquidity / overnight_mom computes now call the ops; reversal still delegates to momentum. Definitions unchanged: every rewritten factor is locked bit-identical to an independent naive oracle in tests/test_factors_ops.py, and the pre-existing factor tests pass unchanged. pytest 1917 passed (baseline 1902 + 15), ruff clean.
…compute/minute
D2 commits B+C (design v3.2 §3.2 / §6.4 / R14): the single definition point of
every minute factor's math moves to factors/compute/minute — one factor per
file, each file carrying BOTH the compute_* math (rewritten on the shared
primitives layer) and its D1 Factor surface class (spec declarations carried
over unchanged).
- factors/compute/minute/primitives.py: peak_mask_for_symbol +
prepare_visible_minute_bars move house VERBATIM (they were already
author-once; R1 — relocation, not repair), plus the shared PIT front half
(visible_minute_frame), amplitude guard, positive-trade mask, valid-day
rolling and pooled-trailing-reduce forms, per-symbol split, and the
VOLUME_PRV_* classification constants. Per-factor definition parameters
(min_valley_bars=20 / min_ridge_bars=10 / counted-AFTER-the-guard, ...)
stay explicit parameters — parameterized, never unified.
- data/clean/intraday_schema.py: DATE_LEVEL / DAILY_INDEX_NAMES /
DEFAULT_DECISION_TIME / DEFAULT_SESSION_OPEN move here (single home both
the data-layer generic core and factors.compute.minute can import without
a cycle); intraday_aggregate re-exports them unchanged.
- data/clean/intraday_aggregate.py: MIXED module per R14 — generic core
(asof_daily_features / resample_intraday_bars / feature keys) keeps real
code; the MMP and jump factor math migrated to factors.compute.minute.{mmp,
jump_amount_corr} and is re-exported (qt/intraday_tail_framework.py:38 and
the eval runners keep working with ZERO changes).
- 10 pure data/clean factor modules + factors/compute/intraday_derived.py are
now import-only re-export shims (deleted in D6d); locked by an AST purity
test scoped EXACTLY to the fully-migratable set (R14) plus re-export
identity tests (shim name IS the new object).
- factors/registry/builtin.py imports the classes/constants from their new
homes; dispatch semantics unchanged.
- The 11 per-factor property-test files migrate their imports to the new
modules (性质测试搬迁 — same properties, now proven directly on the new
engine; the PR-M teeth tests keep their monkeypatch seams, now patching
factors.compute.minute.peak_ridge_amount_ratio).
Factor definitions and float operation order preserved verbatim; the D1
frozen-baseline cell-by-cell reconciliation follows in this branch.
pytest 1941 passed (1917 + 24 shim tests), ruff clean, validate-config 31/31,
phase0 anchor ic 0.9600 / annual 0.8408 unchanged.
…R-D ex-date measurement debt D2 commit C - the NET-NEW P10 property the D0 catalogue stamped for this stage. tests/test_overnight_boundary_declarations.py implements D0 s1.2's reading of the declaration probe: a UNIFORM basis rescale of the symbol's entire < d price history (lambda=0.5, a power of two, so 'none' checks assert bit-equality). Declared none -> day-d value must not move (ridge_minute_return / intraday_amp_cut / jump_amount_corr minute-side, momentum through the real front_adjust chain daily-side); declared crossed_disclosed -> the value MUST move (valley_price_quantile + minute_ideal_amplitude positive controls, the s3.5 pair-migration rule). Two calibration notes recorded in the module docstring: (1) a first draft also rescaled 'amount' and the jump factor's cross-day pooled correlation rightly moved - that simulated a rescale no real ex-date performs (amount is traded RMB value); the D0 wording, prices only, is the faithful probe; (2) the first ridge data set had no ridge at the day's first bar, so the cross-day-lag mutation could not fire - the data now places a ridge run at slots (0,1) and the mutation evidence was RE-RUN and hits (cross-day lag: rc=1; restored: rc=0). minute_ideal_amplitude's D0-note-1 measurement debt is PAID: measured against the D1 frozen CSI500 panel + adj_factor cache, 4.58% of finite values (52,876 / 1,155,283) carry a true ex-date strictly inside their 10-day pooling window; on a seeded 120-value sample (each pinned value re-derived from raw 1min cache and matched 120/120), the ranking-key re-basing intervention moves the factor in 70% of windows (|delta| median 1.07e-05 / p90 1.79e-04 / max 2.91e-03 vs panel std 6.0e-04; top/bottom-k membership shifts median 2.96% / p90 45.8% of pooled bars). All three CROSSED_DISCLOSED requirements now met; the numbers live in the spec docstring, and the D1 debt-tracker test (test_minute_ideal_amplitude_docstring_tracks_the_d2_measurement_debt) now pins the SETTLED disclosure - the docstring must carry the measured numbers and must no longer claim the obligation open (an authorized tracker update, not a weakened test: the tracked debt was paid).
…tooling D2 commits F/G tooling (design v3.2 s5 legs 2+4, R11/R12). qt/panel_reconcile.py rebuilds all 14 closing-factor RAW panels on the current tree (the panel_freeze recipes route through the D2 shims, so they exercise the migrated factors/compute/minute math automatically), writes them to the SEPARATE artifacts/refactor_baseline/panels_d2/ directory (the frozen D1 baseline is never overwritten), and compares cell by cell: identical index, identical NaN sets, relative diff <= 1e-12 on jointly-finite cells. Anti-empty -reconciliation provenance: the frozen side is hash-verified against the D1 manifest BEFORE any comparison is trusted, and both sides are read from disk through independent file reads. tests/test_panel_reconcile_d2.py feeds the comparator engineered defects (value drift beyond budget, NaN-set change in both directions, index mismatch, near-zero fabrication) and asserts each one CONVICTS - a comparator that cannot fail is the compare_postmerge.py failure mode. qt/hand_anchors_d2.py + qt/hand_anchor_rows.py hand-recompute stratified anchor rows (R12: warm-up end / guard boundary / ex-date window / seeded interior) from the raw cache parquets in plain numpy+pandas arithmetic. The module NEVER imports factors.* or data.clean.* (runtime guard raises if the engine sneaks into sys.modules); engine values come from the panels_d2 files (disk reads). qt/hand_anchors_engine_values.py is the one companion allowed to import the engine - it fills in the engine side for the four ops-rewritten daily factors that have no frozen panel (momentum/reversal/liquidity/ overnight_mom). Two engine-semantics facts the hand check surfaced and now encodes (the pre-flight spot-check convicted the first drafts, which is the point of an independent reimplementation): (1) PR-C jump_amount_corr computes on FULL-day bars - its engine has no 14:50 cutoff (day-level PIT only; PR-C predates the cutoff convention); (2) PR-E resamples the FULL day to 5min FIRST and then PIT-filters the DERIVED bars by their own available_time = max(source)+1min, so a bucket with any post-14:49 constituent is dropped WHOLE. With both encoded, the 24-row pre-flight spot-check against the frozen panels passes 0-failure (20 exact, worst rel 5.8e-15, all within the 1e-12 float budget). tests/test_provisional_anchor_d2.py freezes the PROVISIONAL full-precision phase0 demo anchor (R10 timeline; promoted in D6): ic_mean 0.9600438863169586 / annual_return 0.8407986461861146 / sharpe 10.304260918159038 / plus turnover/cost/vol/maxDD, exact-equality contract, with the coarse 0.9600/0.8408 reading locked as a corollary.
Three corrections the frozen-panel spot-check convicted (the reason the hand reimplementation exists): (1) the minute-cache path gains its freq=1min partition level; (2) PR-C jump reads FULL-day bars (its engine has no 14:50 cutoff - day-level PIT only, a PR-C-predates-the-convention fact now recorded at the call site); (3) PR-E resamples the FULL day to 5min FIRST and PIT- filters the DERIVED bar by available_time = max(source)+1min, dropping partial buckets WHOLE. Post-fix spot-check: 24/24 rows pass (20 exact, worst rel 5.8e-15). Also widens the peak-family hand read windows to 200 calendar days (baseline-truncation safety), trims the valley-quantile cross-section read to 130 days and amortizes its five anchor rows over three distinct dates (each date costs a full-universe hand cross-section; disclosed).
momentum/reversal/overnight: warmup + ex-date + 3 random; liquidity (amount channel, ex-date class not applicable - disclosed, not faked): warmup + 4 random.
The first G run convicted every early-window row (warmup/ex-date anchors in 2021-07/08) while every mid-panel row was exact: the minute cache holds BACKFILLED bars from before 2021-07-01, and the hand reads were pulling that richer history in, while the frozen panels were computed on store.read_range(cfg.data.start=2021-07-01, ...). read_minutes now clips its lower bound to the plane start. _pick_warmup gains a disclosed fallback for factors that only emit rows once the accumulation floor is crossed (amp_cut's combine drops non-finite stat pairs, so its panel has no leading-NaN ramp): the symbol's first emitted row IS the warm-up end.
A missing guard_boundary row in hand_anchors_d2.json was previously indistinguishable from a forgotten stratification class (review D2, LOW). The scan now records an explicit guard_boundary_skipped row with the scan size, so the JSON is self-describing. Current run artifacts predate this fix; the D5 re-run will carry the disclosure.
StackOverFlow11
added a commit
that referenced
this pull request
Jul 24, 2026
docs: record refactor step D2 (PR #89)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Step D2 of the factor-layer refactor (v3.2 §3.2/§九) — the first math-touching step. Identity: implementation moves, definitions do not — and the migration turned out bit-identical.
What (9 commits)
factors/ops/— 7 time-series operators (only actually-consumed ones); momentum/volatility/liquidity/overnight_mom rewritten on them, locked by naive-oracle equivalence tests.factors/compute/minute/—primitives.py(absorbspeak_mask_for_symbol— the taxonomy's single authored definition — plus visible-bar prep, VWAP identities, rolling baselines, valid-day guards, all parameterized per-factor) + 11 one-file-per-factor modules (math + surface class + D1 spec declarations).intraday_derived.pybecome AST-locked re-export shims;intraday_aggregatekeeps its generic core as real code and re-exports MMP/jump;intraday_schemahosts shared constants (adjudicated: leaving them in aggregate forms a real import cycle, live-tested; documented for D6d).Acceptance evidence
max_rel_diff = 0.0, canonical hashes equal — bit-identical. Comparator itself has 6 teeth tests; frozen panels never touched (panels_d2/separate).final_nav=1.019318; I5d quintiles + Spearman +0.9000; I5e quintiles + Spearman −0.5000 (cache-only, zero live calls).Review
Adversarial review: APPROVE-WITH-NITS (0 CRIT/HIGH/MED, 1 LOW). Reviewer went beyond reading: per-function text diffs of migrated math vs
git show main:(only docstring/rename deltas), live identity checks proving the old import paths serve the new engine objects (dead-code illusion excluded), a live import-cycle test validating the constants-home adjudication (JC1), two mutations re-run live, and independent re-derivation of the measurement-debt numbers. The 1 LOW (silently skipped guard_boundary anchor class for value_ep/value_bp left no machine-readable trace) is fixed in this PR (guard_boundary_skippeddisclosure rows; current JSON predates the fix, the D5 re-run will carry it).Lead-independent: gates re-run, 3
panels_d2hashes recomputed ≡ D1 manifest, shim import chains live-verified tofactors.compute.minute.*.Notes carried to D6: shim-period
aggregate → factors.minuteimport direction (disclosed exception to red line #10, cycle-free, sunset D6d);asof_daily_features' mmp_ew hook needs a new home when shims are deleted.