From 6a32bbbbec1492910b25c9a4ea4445cdc82c9dc7 Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Sat, 25 Jul 2026 16:28:02 -0700 Subject: [PATCH 1/7] fix(factors): truncate jump_amount_corr at the 14:50 decision cutoff CORRECTNESS FIX, not a refactor step. `compute_jump_amount_corr` was the one minute factor of the eleven with NO intraday PIT truncation at all: its `decision_time` / `prepare_visible_minute_bars` reference count was 0 while the other ten were 4-10. It was built before the 2026-07-19 standing authorization to truncate every full-day minute factor at 14:50 and never got it applied. Under the 14:51-VWAP exec_to_exec basis (PR #79) this is a STRICT LOOKAHEAD: the value at d pooled that day's 14:50-15:00 bars, so the information set crossed its own entry anchor by ten minutes, in the most volume-concentrated window of the A-share session. The published exec artifacts declare `spec.decision_cutoff="14:50:00"` -- a check the factor did not perform. The fix uses the SAME machinery as its siblings (`visible_minute_frame`, shared with ideal-amplitude / amp-anomaly / amp-cut). No new mechanism, and the correlation math is untouched -- only the set of bars fed into it. The two row filters commute, so the surviving set does not depend on their order. Blast radius (measured, not assumed): * old eval runner + `qt.panel_freeze` -> defective (the published artifacts); * D4 materializer DECISION view -> ALREADY correct: it applies `minute_decision_cutoff`, the same predicate, before the binding, so the double filter is idempotent and those values are bit-unchanged; * D4 materializer CLOSE view -> was defective, now consistent. `binding.py` already documented "the compute functions run with their DEFAULT decision_time as a redundant-but-consistent internal cutoff" -- true for ten of eleven. It is now true for all eleven. Spec version 1.0 -> 1.1 and the description carries the correction, so every report / dashboard self-identifies which definition produced it. NET-NEW GUARD (`tests/test_minute_decision_cutoff_leakage.py`, 40 tests): every existing leakage test guards the CROSS-DAY boundary, and their fixtures are five-bar morning sessions with no 14:50 in them -- the intraday boundary was untested everywhere, which is why nothing went red. The new guard covers ALL eleven minute Factor subclasses (discovered by walking the package, so a new factor file cannot be added without being covered) plus the `mmp_ew` intraday aggregate feature, under two knives: every bar the visibility rule excludes, and the sharper `bar_end >= 14:51` (the execution bar itself and the nine after it). Each perturbation asserts BEFORE use that it moved every selected row and no unselected row, and each factor must emit a finite value, so a green run cannot come from an empty mutation or from NaN == NaN. Mutation evidence (run, not claimed): * pre-fix, jump only: both knives RED -- `invisible` moved 400/400 finite values (max|diff| 0.453), `exec_and_after` moved the NaN mask itself; the other ten factors and mmp_ew were GREEN on both. Post-fix: 40 passed. * dropping one factor from the coverage dict -> the closure test goes RED. * making the perturbation a no-op -> 26 tests go RED (the preconditions are inside the invariance tests, so a vacuous mutation cannot pass quietly). `qt/hand_anchors_d2.py` read FULL-day bars for this factor and said so in a comment -- a hand check that faithfully mirrored the defect. It now truncates like every sibling. --- config/phase_c_jump_amount_corr.yaml | 9 + factors/compute/minute/jump_amount_corr.py | 57 ++- qt/eval_jump_amount_corr.py | 6 + qt/hand_anchors_d2.py | 17 +- tests/test_minute_decision_cutoff_leakage.py | 447 +++++++++++++++++++ 5 files changed, 517 insertions(+), 19 deletions(-) create mode 100644 tests/test_minute_decision_cutoff_leakage.py diff --git a/config/phase_c_jump_amount_corr.yaml b/config/phase_c_jump_amount_corr.yaml index cd8971a..725adfe 100644 --- a/config/phase_c_jump_amount_corr.yaml +++ b/config/phase_c_jump_amount_corr.yaml @@ -6,6 +6,15 @@ # CACHE-ONLY: the minute read is provably live-call-free; daily / universe / # covariate endpoints go through the read-through cache (warm -> 0 gap fetches). # +# PIT TRUNCATION (read this before comparing any run against a pre-2026-07-25 artifact): +# the factor pools only the 1min bars visible at 14:50 (available_time <= that bar's +# trade_date + 14:50), like every sibling minute factor. It did NOT always: this factor +# was built before the standing 14:50-truncation authorization and ran on FULL-day bars, +# so its value at d read that day's 14:50-15:00 window -- POST-ENTRY information under the +# 14:51-VWAP exec_to_exec basis. Artifacts produced before the fix are superseded and are +# kept, labelled, under artifacts/reports/pr_c_untruncated_close_to_close/. The factor +# spec version distinguishes them: v1.0 = untruncated, v1.1 = truncated. +# # Universe = CSI500 (000905.SH), PIT membership; window = 2021-07-01 .. 2026-06-30 # (the project's minute-coverage window; the report notes the factor is stronger in # mid/small caps). Daily rebalance (project + contract default). The evaluator runs diff --git a/factors/compute/minute/jump_amount_corr.py b/factors/compute/minute/jump_amount_corr.py index 5bb2dca..a59c00e 100644 --- a/factors/compute/minute/jump_amount_corr.py +++ b/factors/compute/minute/jump_amount_corr.py @@ -4,10 +4,21 @@ -10.23%): the trailing-``lookback_days``-trading-day lagged Pearson correlation between the traded ``amount`` at price-JUMP minutes and the amount at the STRICTLY-next minute. The daily value at ``(date d, symbol s)`` uses ONLY bars -at dates ``<= d`` (the trailing window ending at d), so a factor value never -sees a future bar (invariant #1); it is meant to trade close-to-close from d+1. +whose ``available_time`` is at or before ``d + decision_time`` within the +trailing window ending at d, so a factor value never sees a bar it could not +know at the decision moment (invariant #1). Definition (LOCKED, per bar of one (symbol, day) session): + * PIT truncation (standing authorization, 2026-07-19): each day keeps only the + 1min bars with ``available_time <= (that bar's trade_date + decision_time)`` + (default 14:50) — history days AND the signal day are truncated identically, + exactly as in the ten sibling minute factors. CORRECTNESS FIX (post-#79): + this factor was built BEFORE that authorization and was the one of the eleven + with no truncation at all, so under the 14:51-VWAP exec_to_exec basis its + value at d read d's 14:50-15:00 bars — i.e. the execution minute itself and + the nine after it. The published pre-fix values are superseded; see + ``tests/test_minute_decision_cutoff_leakage.py`` for the guard that now + covers the whole minute-factor surface; * ``amplitude = (high - low) / open`` (guard open > 0, amount finite); * ``jump`` = within-(symbol, day) amplitude z-score (ddof=1) ``> jump_z``; * pair each jump minute ``t`` with the STRICTLY-next minute (same session, @@ -32,6 +43,7 @@ from data.availability_policy import STK_MINS_1MIN from data.clean.intraday_schema import ( DATE_LEVEL, + DEFAULT_DECISION_TIME, SYMBOL_LEVEL, validate_intraday_bars, ) @@ -39,6 +51,7 @@ from factors.compute.minute.primitives import ( ONE_MINUTE_SECONDS, empty_factor_series, + visible_minute_frame, ) from factors.spec import FactorSpec, PanelField @@ -63,6 +76,7 @@ def compute_jump_amount_corr( lookback_days: int = JUMP_LOOKBACK_DAYS, min_pairs: int = JUMP_MIN_PAIRS, jump_z: float = JUMP_Z, + decision_time: str = DEFAULT_DECISION_TIME, name: str = "jump_amount_corr", ) -> pd.Series: """PIT-safe daily "price-jump turnover correlation" factor from 1min ``bars``. @@ -76,6 +90,7 @@ def compute_jump_amount_corr( lookback_days: trailing trading-day window length (part of the definition). min_pairs: minimum jump-pairs in the window for a finite value. jump_z: within-day amplitude z-score threshold defining a jump minute. + decision_time: per-bar PIT cutoff time-of-day (default 14:50:00). name: the returned Series name (the factor-panel column name). Returns: @@ -91,15 +106,24 @@ def compute_jump_amount_corr( if len(bars) == 0: return empty_factor_series(name) - work = bars.reset_index()[ - [SYMBOL_LEVEL, "bar_end", "open", "high", "low", "amount"] - ].copy() - # Guard bad rows BEFORE anything else: a non-positive open makes the amplitude - # meaningless and a non-finite amount would poison the correlation. + # PIT truncation FIRST, on the SAME shared helper the sibling amplitude family + # uses (jump / ideal-amplitude / amp-anomaly / amp-cut): keep only the bars whose + # own ``available_time`` is at or before their ``trade_date + decision_time``. + # ``trade_date`` is this factor's date axis, so it is renamed to DATE_LEVEL and + # every downstream step below is unchanged. + work = visible_minute_frame( + bars, + columns=("open", "high", "low", "amount"), + decision_time=decision_time, + ).rename(columns={"trade_date": DATE_LEVEL}) + if work.empty: + return empty_factor_series(name) + # Guard bad rows: a non-positive open makes the amplitude meaningless and a + # non-finite amount would poison the correlation. (Row filters commute, so the + # surviving set is the same whichever of the two masks is applied first.) work = work[(work["open"] > 0.0) & np.isfinite(work["amount"].to_numpy(dtype=float))] if work.empty: return empty_factor_series(name) - work[DATE_LEVEL] = work["bar_end"].dt.normalize() # Sort so the "strictly next minute" shift and the per-symbol trading-day # rolling both see bars in chronological order within each (symbol, day). work = work.sort_values([SYMBOL_LEVEL, "bar_end"], kind="mergesort") @@ -238,14 +262,23 @@ def spec(self) -> FactorSpec: """ return FactorSpec( factor_id=self.name, - version="1.0", + # 1.1: the intraday-cutoff correctness fix. The DEFINITION changed, so + # every artifact self-identifies which one produced it (the version is + # printed in the report title and on the dashboard). + version="1.1", description=( - f"Price-jump turnover correlation (Kaiyuan report §6): trailing " + f"Price-jump turnover correlation (Kaiyuan report §6): 1min bars " + f"PIT-truncated at 14:50 per bar, then the trailing " f"{self._lookback_days}-trading-day lagged Pearson corr between the " f"traded amount at price-JUMP minutes (within-day amplitude z-score " f">1) and the amount at the strictly-next minute. Derived from 1min " - f"bars but a DAILY signal traded close-to-close; >= {JUMP_MIN_PAIRS} " - f"jump-pairs required else NaN." + f"bars but a DAILY signal; >= {JUMP_MIN_PAIRS} jump-pairs required " + f"else NaN. CORRECTION (v1.0 -> v1.1): values published before this " + f"fix were computed WITHOUT the 14:50 truncation, so a value at d " + f"included that day's 14:50-15:00 bars — i.e. POST-ENTRY information " + f"under the 14:51-VWAP exec_to_exec basis (the execution minute " + f"itself and the nine after it). Those values are superseded; this " + f"run is the truncated recomputation." ), expected_ic_sign=-1, is_intraday=False, diff --git a/qt/eval_jump_amount_corr.py b/qt/eval_jump_amount_corr.py index c2f0850..3acd572 100644 --- a/qt/eval_jump_amount_corr.py +++ b/qt/eval_jump_amount_corr.py @@ -104,6 +104,12 @@ def _load_jump_factor_panel( root = cfg.data.cache.root_dir store = IntradayParquetStore(root) start = pd.Timestamp(cfg.data.start).normalize() + # Whole-day READ window; the 14:50 PIT truncation is applied inside + # ``compute_jump_amount_corr`` (as in every sibling minute factor), so the + # post-cutoff bars this read returns are dropped before any pooling. This + # read is NOT the cutoff — a reader who assumes it is will conclude the + # factor is truncated when the compute is not, which is exactly how the + # missing truncation survived review here. end = pd.Timestamp(cfg.data.end).normalize() + pd.Timedelta("23:59:59") series: list[pd.Series] = [] diff --git a/qt/hand_anchors_d2.py b/qt/hand_anchors_d2.py index 116b467..8756933 100644 --- a/qt/hand_anchors_d2.py +++ b/qt/hand_anchors_d2.py @@ -83,10 +83,11 @@ def read_minutes( ``pit=True`` keeps only the 14:50-visible bars (``bar_end + 1min <= trade_date + 14:50`` — the runners' ``data_lag='1min'``). ``pit=False`` - returns the FULL day: two factors need it — PR-C (jump) predates the - cutoff convention and its engine computes on full-day bars (day-level PIT - only), and PR-E (amp anomaly) resamples the FULL day to 5min FIRST and - PIT-filters the DERIVED bars by their own available_time. + returns the FULL day: PR-E (amp anomaly) needs it because it resamples the + FULL day to 5min FIRST and PIT-filters the DERIVED bars by their own + available_time. (PR-C / jump used to need it too — its engine had no + intraday cutoff at all. That was a lookahead defect, not a convention: it + is fixed, and this hand check now truncates like every sibling.) """ # Clip to the evaluation plane's start: the cache holds BACKFILLED bars # from before 2021-07-01, but the frozen panels were computed on @@ -377,9 +378,11 @@ def hand_peak_ridge_amount_ratio(symbol: str, d: pd.Timestamp) -> float: def hand_jump_amount_corr(symbol: str, d: pd.Timestamp) -> float: - # FULL-day bars: the engine's compute has NO 14:50 cutoff (PR-C predates - # the cutoff convention; its PIT contract is day-level, dates <= d only). - bars = read_minutes(symbol, d - pd.Timedelta(days=90), d, pit=False) + # 14:50-visible bars only (pit=True, the default): the engine truncates each + # day at the decision time like every sibling minute factor. Before the + # truncation fix this read full days to mirror an engine that had no + # intraday cutoff — mirroring a defect is how a hand check blesses one. + bars = read_minutes(symbol, d - pd.Timedelta(days=90), d) ok = (bars["open"].to_numpy(float) > 0) & np.isfinite(bars["amount"].to_numpy(float)) work = bars[ok].sort_values("bar_end").reset_index(drop=True) days = sorted(work["trade_date"].unique()) diff --git a/tests/test_minute_decision_cutoff_leakage.py b/tests/test_minute_decision_cutoff_leakage.py new file mode 100644 index 0000000..db6b84e --- /dev/null +++ b/tests/test_minute_decision_cutoff_leakage.py @@ -0,0 +1,447 @@ +"""INTRADAY PIT boundary: no minute factor may read a post-cutoff bar. + +Why this file exists (the blind spot it closes) +----------------------------------------------- +Every minute factor already had a leakage test — but all of them guard the +CROSS-DAY boundary ("perturbing day d+1 must not move day d"). The INTRA-day +boundary was untested: the fixtures of those tests are five-or-six-bar morning +sessions that do not even contain a 14:50, so the assertion "the 14:50 cutoff is +applied" was never expressed anywhere. + +That boundary became load-bearing when PR #79 moved the entry anchor from +``close(d)`` to the 14:51 execution bar: under the exec_to_exec basis a factor +value at d that reads d's 14:50-15:00 bars is strictly forward-looking — the +information set crosses its own entry anchor, in the single most +volume-concentrated window of the A-share session. ``jump_amount_corr`` was the +one factor of the eleven with no truncation at all (its ``decision_time`` / +``prepare_visible_minute_bars`` reference count was 0 while the other ten were +4-10), and nothing went red. + +So the guard here is deliberately GENERIC: it covers every ``Factor`` subclass +defined under ``factors/compute/minute`` (discovered by walking the package, not +by a hand-maintained list — §六.8, and the same lesson D4b learned when dropping +a factor from a binding table left the suite green) plus the ``mmp_ew`` intraday +aggregate feature, which is a minute signal reached through +:func:`data.clean.intraday_aggregate.asof_daily_features` rather than a Factor +class. + +The two knives +-------------- +1. ``INVISIBLE`` — every bar the visibility rule excludes + (``available_time > trade_date + decision_time``; with the 1-minute data lag + that is ``bar_end >= 14:50``). This is the full set a PIT-truncating factor + must be blind to. +2. ``EXEC_AND_AFTER`` — only ``bar_end >= 14:51``, i.e. the execution bar itself + and the nine minutes after it. A strict subset of (1), and the sharper + statement: a factor that moves under this perturbation has seen the very + minute it is modelled as trading in. + +Both must leave every factor value BIT-identical (NaN mask included). + +Not-vacuous, by assertion rather than by hope +--------------------------------------------- +Each perturbation is checked BEFORE it is used: + +* the mask selects at least one bar (and, for the factors, at least one bar on + every fixture day); +* the perturbed frame really differs from the base frame on EVERY selected row; +* the perturbed frame is byte-identical to the base frame on EVERY unselected + row (so a "pass" cannot come from a mutation that changed nothing, or from one + that changed the visible half too). + +And each factor must emit at least one finite value on the base fixture, +otherwise "unchanged" would be the trivially-true statement that NaN == NaN. +""" + +from __future__ import annotations + +import importlib +import inspect +import pathlib + +import numpy as np +import pandas as pd +import pytest + +from data.clean.intraday_aggregate import asof_daily_features +from data.clean.intraday_schema import ( + DEFAULT_DECISION_TIME, + SYMBOL_LEVEL, + TIME_LEVEL, + normalize_intraday_bars, +) +from factors.base import Factor +from factors.compute.minute import binding as binding_module +from factors.compute.minute.amp_marginal_anomaly_vol import ( + AmpMarginalAnomalyVolFactor, + compute_amp_marginal_anomaly_vol, +) +from factors.compute.minute.intraday_amp_cut import ( + IntradayAmpCutFactor, + compute_intraday_amp_cut, +) +from factors.compute.minute.jump_amount_corr import ( + JumpAmountCorrFactor, + compute_jump_amount_corr, +) +from factors.compute.minute.minute_ideal_amplitude import ( + MinuteIdealAmplitudeFactor, + compute_minute_ideal_amplitude, +) +from factors.compute.minute.peak_interval_kurtosis import ( + PeakIntervalKurtosisFactor, + compute_peak_interval_kurtosis, +) +from factors.compute.minute.peak_ridge_amount_ratio import ( + PeakRidgeAmountRatioFactor, + compute_peak_ridge_amount_ratio, +) +from factors.compute.minute.ridge_minute_return import ( + RidgeMinuteReturnFactor, + compute_ridge_minute_return, +) +from factors.compute.minute.valley_price_quantile import ( + ValleyPriceQuantileFactor, + compute_valley_price_quantile, +) +from factors.compute.minute.valley_relative_vwap import ( + ValleyRelativeVwapFactor, + compute_valley_relative_vwap, +) +from factors.compute.minute.valley_ridge_vwap_ratio import ( + ValleyRidgeVwapRatioFactor, + compute_valley_ridge_vwap_ratio, +) +from factors.compute.minute.volume_peak_count import ( + VolumePeakCountFactor, + compute_volume_peak_count, +) + +# --------------------------------------------------------------------------- # +# Fixture: a realistic A-share session that STRADDLES the cutoff +# --------------------------------------------------------------------------- # +# Morning 09:31..11:30 (120 bars) + afternoon 13:01..15:00 (120 bars) = 240/day. +# The afternoon deliberately runs to 15:00 so bars exist on BOTH sides of 14:50. +_N_DAYS = 40 # >= baseline_days (20) + min_valid_days (10) for the peak family +_N_SYMBOLS = 10 # >= AMP_CUT_MIN_CROSS_SECTION / VALLEY_QUANTILE_MIN_CROSS_SECTION +_SYMBOLS = tuple(f"{600000 + i}.SH" for i in range(_N_SYMBOLS)) +_CUTOFF = DEFAULT_DECISION_TIME # "14:50:00" +_EXEC_MINUTE = "14:51:00" + + +def _session_minutes(day: pd.Timestamp) -> pd.DatetimeIndex: + morning = pd.date_range(day + pd.Timedelta("09:31:00"), periods=120, freq="1min") + afternoon = pd.date_range(day + pd.Timedelta("13:01:00"), periods=120, freq="1min") + return morning.append(afternoon) + + +def _build_bars() -> pd.DataFrame: + """Deterministic pseudo-random 1min bars for ``_N_SYMBOLS`` x ``_N_DAYS``.""" + rng = np.random.default_rng(20260725) + days = pd.bdate_range("2021-07-01", periods=_N_DAYS) + frames = [] + for s_i, sym in enumerate(_SYMBOLS): + for day in days: + times = _session_minutes(day) + n = len(times) + # Random-walk close around a per-symbol level; high/low bracket it. + drift = rng.normal(0.0, 0.0009, n).cumsum() + close = (10.0 + s_i) * np.exp(drift) + spread = np.abs(rng.normal(0.0018, 0.0011, n)) + 1e-4 + high = close * (1.0 + spread) + low = close * (1.0 - spread) + open_ = np.concatenate([[close[0]], close[:-1]]) + volume = rng.lognormal(9.0, 0.8, n) + frames.append( + pd.DataFrame( + { + "time": times, + "symbol": sym, + "open": open_, + "high": high, + "low": low, + "close": close, + "volume": volume, + "amount": volume * close, + } + ) + ) + return normalize_intraday_bars(pd.concat(frames, ignore_index=True), freq="1min") + + +BARS = _build_bars() +#: Stand-in for the runner's DAILY front-adjusted close panel (MultiIndex(date, +#: symbol) with a ``close`` column) that ``valley_price_quantile`` neutralizes +#: against. It is a DAILY input, so it is built once from the base bars and held +#: FIXED across every perturbation below. +_DAILY_CLOSE = ( + BARS.reset_index() + .assign(date=lambda f: f[TIME_LEVEL].dt.normalize()) + .groupby(["date", SYMBOL_LEVEL], sort=True)["close"] + .last() + .to_frame("close") +) + +# Perturbed columns: every numeric channel any minute factor reads. +_PERTURBED_COLUMNS = ("open", "high", "low", "close", "volume", "amount") + + +def _invisible_mask(bars: pd.DataFrame) -> np.ndarray: + """Bars the visibility rule excludes: ``available_time > trade_date + cutoff``.""" + trade_date = bars["bar_end"].dt.normalize() + return (bars["available_time"] > trade_date + pd.Timedelta(_CUTOFF)).to_numpy() + + +def _exec_and_after_mask(bars: pd.DataFrame) -> np.ndarray: + """The execution bar itself and every bar after it (``bar_end >= 14:51``).""" + trade_date = bars["bar_end"].dt.normalize() + return (bars["bar_end"] >= trade_date + pd.Timedelta(_EXEC_MINUTE)).to_numpy() + + +_MASKS = {"invisible": _invisible_mask, "exec_and_after": _exec_and_after_mask} + + +def _perturb(bars: pd.DataFrame, mask: np.ndarray) -> pd.DataFrame: + """Wildly move every perturbed channel on the masked rows only. + + The price channels stay internally consistent (``low <= close <= high``, + all strictly positive) so the perturbed frame still passes + ``validate_intraday_bars`` — a factor must be blind to these bars because + they are in the future, not because they are malformed. + """ + out = bars.copy(deep=True) + scale = 137.0 # far outside any plausible per-minute move + out.loc[mask, "high"] = bars.loc[mask, "high"].to_numpy() * scale + out.loc[mask, "low"] = bars.loc[mask, "low"].to_numpy() / scale + out.loc[mask, "close"] = bars.loc[mask, "close"].to_numpy() * 7.0 + out.loc[mask, "open"] = bars.loc[mask, "open"].to_numpy() / 5.0 + out.loc[mask, "volume"] = bars.loc[mask, "volume"].to_numpy() * 9.0e5 + 1.0 + out.loc[mask, "amount"] = bars.loc[mask, "amount"].to_numpy() * 9.0e5 + 1.0 + return out + + +def _assert_mutation_is_real_and_targeted( + base: pd.DataFrame, perturbed: pd.DataFrame, mask: np.ndarray, label: str +) -> None: + """The mutation moved EVERY selected row and NO unselected row. + + Without this, a green invariance test proves nothing: the project has six + recorded instances of a mutation that changed nothing while the test that + depended on it stayed green. + """ + assert mask.any(), f"{label}: the mask selected no bar — the fixture is vacuous" + cols = list(_PERTURBED_COLUMNS) + changed = (base.loc[mask, cols].to_numpy() != perturbed.loc[mask, cols].to_numpy()).any( + axis=1 + ) + assert changed.all(), ( + f"{label}: {int((~changed).sum())} selected bar(s) were NOT changed by the " + f"perturbation — the mutation does not reach what this test claims it does" + ) + pd.testing.assert_frame_equal(base.loc[~mask], perturbed.loc[~mask]) + + +# --------------------------------------------------------------------------- # +# The covered surface: every minute Factor class + the mmp_ew aggregate feature +# --------------------------------------------------------------------------- # +def _jump(bars: pd.DataFrame) -> pd.Series: + return compute_jump_amount_corr(bars) + + +def _ideal_amp(bars: pd.DataFrame) -> pd.Series: + return compute_minute_ideal_amplitude(bars) + + +def _amp_anomaly(bars: pd.DataFrame) -> pd.Series: + return compute_amp_marginal_anomaly_vol(bars) + + +def _peak_count(bars: pd.DataFrame) -> pd.Series: + return compute_volume_peak_count(bars) + + +def _amp_cut(bars: pd.DataFrame) -> pd.Series: + return compute_intraday_amp_cut(bars) + + +def _peak_interval(bars: pd.DataFrame) -> pd.Series: + return compute_peak_interval_kurtosis(bars) + + +def _valley_vwap(bars: pd.DataFrame) -> pd.Series: + return compute_valley_relative_vwap(bars) + + +def _valley_ridge(bars: pd.DataFrame) -> pd.Series: + return compute_valley_ridge_vwap_ratio(bars) + + +def _ridge_return(bars: pd.DataFrame) -> pd.Series: + return compute_ridge_minute_return(bars) + + +def _peak_ridge_amount(bars: pd.DataFrame) -> pd.Series: + return compute_peak_ridge_amount_ratio(bars) + + +def _valley_quantile(bars: pd.DataFrame) -> pd.Series: + # The daily close panel is a DAILY input, not a minute one: it is held FIXED + # across the perturbation on purpose, so the only thing that can move the + # factor is the minute channel this test is about. + return compute_valley_price_quantile(bars, _DAILY_CLOSE) + + +def _mmp_ew(bars: pd.DataFrame) -> pd.Series: + frame = asof_daily_features(bars, features=["mmp_ew"]) + return frame[frame.columns[0]] + + +#: minute Factor class -> the free compute that produces its raw daily series. +_FACTOR_COMPUTES: dict[type[Factor], object] = { + JumpAmountCorrFactor: _jump, + MinuteIdealAmplitudeFactor: _ideal_amp, + AmpMarginalAnomalyVolFactor: _amp_anomaly, + VolumePeakCountFactor: _peak_count, + IntradayAmpCutFactor: _amp_cut, + PeakIntervalKurtosisFactor: _peak_interval, + ValleyRelativeVwapFactor: _valley_vwap, + ValleyRidgeVwapRatioFactor: _valley_ridge, + RidgeMinuteReturnFactor: _ridge_return, + PeakRidgeAmountRatioFactor: _peak_ridge_amount, + ValleyPriceQuantileFactor: _valley_quantile, +} + +#: The full covered surface: the eleven factors keyed by name, plus mmp_ew, which +#: is a minute signal WITHOUT a Factor class (it is an intraday-aggregate feature +#: consumed by the I5c/I5d runners) and would otherwise slip through a walk that +#: only knows about Factor subclasses. +_CASES: dict[str, object] = { + **{cls().name: fn for cls, fn in _FACTOR_COMPUTES.items()}, + "mmp_ew": _mmp_ew, +} + + +def _minute_factor_classes() -> dict[type, str]: + """Every ``Factor`` subclass DEFINED under ``factors/compute/minute``. + + The authoritative surface (same walk as the D4b binding-closure test): a new + minute factor file cannot be added without either being covered here or + turning this file red. + """ + package = pathlib.Path(binding_module.__file__).parent + defined: dict[type, str] = {} + for path in sorted(package.glob("*.py")): + if path.name == "__init__.py": + continue + module = importlib.import_module(f"factors.compute.minute.{path.stem}") + for obj in vars(module).values(): + if ( + inspect.isclass(obj) + and issubclass(obj, Factor) + and obj is not Factor + and obj.__module__ == module.__name__ + ): + defined[obj] = path.name + return defined + + +def test_every_minute_factor_class_is_covered_by_a_cutoff_case(): + """A minute factor absent from ``_FACTOR_COMPUTES`` fails HERE, not silently. + + Deriving the parametrize list from a hand-written dict only guards the + direction "listed -> tested". The other direction — a factor that exists but + is in no list — is exactly how a leakage guard goes quietly vacuous, so the + source of truth is the package walk, not the dict. + """ + defined = _minute_factor_classes() + assert defined, "found no minute factor classes — the walk itself is broken" + missing = {cls.__name__: defined[cls] for cls in defined if cls not in _FACTOR_COMPUTES} + assert not missing, ( + f"minute factor(s) with no decision-cutoff leakage case: {missing} — they " + f"would silently keep the right to read post-14:50 bars" + ) + stale = {cls.__name__ for cls in _FACTOR_COMPUTES if cls not in defined} + assert not stale, f"leakage cases for classes no longer defined: {stale}" + + +# --------------------------------------------------------------------------- # +# The perturbations themselves are real and targeted (pre-conditions) +# --------------------------------------------------------------------------- # +@pytest.mark.parametrize("mask_name", sorted(_MASKS)) +def test_perturbation_is_real_and_targeted(mask_name): + mask = _MASKS[mask_name](BARS) + _assert_mutation_is_real_and_targeted(BARS, _perturb(BARS, mask), mask, mask_name) + + +def test_masks_are_nested_and_both_non_trivial(): + """``exec_and_after`` is a STRICT subset of ``invisible`` on this fixture. + + Pins the relationship the two knives are supposed to have: the sharper one + must select fewer bars (it starts one minute later), and both must leave + the visible half of every day untouched. + """ + invisible = _invisible_mask(BARS) + exec_after = _exec_and_after_mask(BARS) + assert exec_after.sum() > 0 + assert int(invisible.sum()) - int(exec_after.sum()) == _N_DAYS * _N_SYMBOLS, ( + "expected exactly the 14:50 bar per (day, symbol) to separate the two masks" + ) + assert not (exec_after & ~invisible).any(), "exec_and_after must be inside invisible" + # The visible half is a real, large majority — a cutoff that ate the session + # would make every "unchanged" assertion below trivially true. + assert (~invisible).sum() > 0.9 * invisible.sum() * 10 + + +# --------------------------------------------------------------------------- # +# The guard +# --------------------------------------------------------------------------- # +@pytest.fixture(scope="module") +def base_values() -> dict[str, pd.Series]: + return {name: fn(BARS) for name, fn in _CASES.items()} + + +@pytest.mark.parametrize("case", sorted(_CASES)) +def test_base_fixture_produces_finite_values(case, base_values): + """Non-vacuity: "unchanged" must not mean "NaN both times".""" + series = base_values[case] + assert int(np.isfinite(series.to_numpy(dtype=float)).sum()) > 0, ( + f"{case}: the base fixture produced no finite value, so the invariance " + f"assertions below would be vacuous" + ) + + +@pytest.mark.parametrize("case", sorted(_CASES)) +@pytest.mark.parametrize("mask_name", sorted(_MASKS)) +def test_post_cutoff_bars_cannot_move_a_minute_factor(case, mask_name, base_values): + """Perturbing bars the factor must not see leaves its values BIT-identical. + + ``jump_amount_corr`` failed both knives before the truncation fix (measured: + every emitted cell moved, max|diff| > 1.0 on a real-cache sample) while the + other ten were already exactly 0 — the asymmetry is the whole point of + running this over the full surface rather than over the factor that was + known to be broken. + """ + mask = _MASKS[mask_name](BARS) + perturbed = _perturb(BARS, mask) + _assert_mutation_is_real_and_targeted(BARS, perturbed, mask, mask_name) + + got = _CASES[case](perturbed) + want = base_values[case] + + assert got.index.equals(want.index), ( + f"{case} / {mask_name}: the emitted (date, symbol) grid moved — a " + f"post-cutoff bar changed WHICH values exist" + ) + g = got.to_numpy(dtype=float) + w = want.to_numpy(dtype=float) + assert np.array_equal(np.isnan(g), np.isnan(w)), ( + f"{case} / {mask_name}: the NaN mask moved" + ) + finite = ~np.isnan(w) + if finite.any(): + worst = float(np.max(np.abs(g[finite] - w[finite]))) + assert worst == 0.0, ( + f"{case} / {mask_name}: {int((g[finite] != w[finite]).sum())} of " + f"{int(finite.sum())} finite values moved (max|diff| = {worst:g}) — the " + f"factor reads bars it cannot know at the decision time" + ) From 10b6ef0530d4e6b7dcc0aca6b7ca08d3b5f0dfcb Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Sat, 25 Jul 2026 16:28:32 -0700 Subject: [PATCH 2/7] feat(freeze): opt-in --only selection + PR-C reference-panel provenance The D1 frozen panel `artifacts/refactor_baseline/panels/jump_amount_corr_20.parquet` was produced by the runner loader, so it carries the untruncated definition. It is left BYTE-UNTOUCHED -- it is the faithful record of what was published, and D2's bit-for-bit reconciliation still points at it. What D5's panel leg needs is a SECOND, truncated reference for this one factor. `--only` (default None = all 14, byte-identical) makes the existing freeze tool produce it, instead of writing a second freeze implementation for one factor or re-reading the 279M-row minute cache for the thirteen that did not change. The shared data plane is still built from the FULL book, so a filtered run computes on the same daily panel as a full run: the flag decides what is FROZEN, never what is LOADED. Selection is validated BEFORE the expensive data plane, and an unknown id is a readable error -- "froze 0 panels" reported as success is the empty-reconciliation shape this repo has already recorded once. Why "old engine + truncated input" is a clean refactor-only control: the fix is a pure INPUT truncation; every line from `amp` to Pearson's closed form is unchanged. Re-running the SAME pre-D5 runner path therefore isolates exactly one variable, so D5 can still attribute any residual difference to load geometry / saturation depth / anchor truncation rather than to a definition change. `docs/factors/pr_c_cutoff_fix_reference_panel.md` carries the provenance: what was wrong, which value paths were affected (the D4 decision view was NOT -- it already applies the same predicate), which panel to use where, the exact rerun command, the old-vs-new evaluation table, and the explicit warning not to treat the two panels as two measurements of one quantity. The D1 manifest doc gets a pointer at the jump row; its numbers are unchanged. --- docs/factors/d1_panel_freeze_manifest.md | 7 + .../pr_c_cutoff_fix_reference_panel.md | 152 ++++++++++++++++++ qt/panel_freeze.py | 98 +++++++++-- tests/test_panel_freeze.py | 38 +++++ 4 files changed, 284 insertions(+), 11 deletions(-) create mode 100644 docs/factors/pr_c_cutoff_fix_reference_panel.md diff --git a/docs/factors/d1_panel_freeze_manifest.md b/docs/factors/d1_panel_freeze_manifest.md index 04f9c66..03e7609 100644 --- a/docs/factors/d1_panel_freeze_manifest.md +++ b/docs/factors/d1_panel_freeze_manifest.md @@ -93,6 +93,13 @@ hash（哈希与首段实时监控记录逐一一致），3 个双跑对象另 （mean/std 为 float64 全精度 `Series.mean()` / `std(ddof=1)`，NaN 跳过；canonical hash 为权威。） +> ⚠️ **`jump_amount_corr_20` 一行的取值口径此后被判定为缺陷**（该因子缺 14:50 日内截断， +> 在 14:51 VWAP exec-to-exec 基准下构成前视）。本表**一个字节未改**——它是已发布内容的 +> 忠实记录，D2 逐位对账的历史参照仍然是它。**D5 面板腿对该因子请改用**截断后的参照面板 +> `artifacts/refactor_baseline/pr_c_cutoff_fix/`，provenance 与「为什么两份并存」见 +> [`pr_c_cutoff_fix_reference_panel.md`](pr_c_cutoff_fix_reference_panel.md)。 +> 其余 13 行不受影响。 + | factor_id | kind | rows | date_min | date_max | n_symbols | n_nan | mean | std | canonical_sha256 | file_sha256 | |---|---|---|---|---|---|---|---|---|---|---| | value_ep | book | 1158912 | 2021-07-01 | 2026-06-30 | 996 | 141362 | 0.04807913635037822 | 0.050405247296850246 | 1404a68fc88778e78d47da1ed6375c39abec36244a29961c3316a74f0c042e76 | 3b3a8970fcfe51d9b221da79a8f001a03d622d37592fdd76d19275a5159164d4 | diff --git a/docs/factors/pr_c_cutoff_fix_reference_panel.md b/docs/factors/pr_c_cutoff_fix_reference_panel.md new file mode 100644 index 0000000..03f41fc --- /dev/null +++ b/docs/factors/pr_c_cutoff_fix_reference_panel.md @@ -0,0 +1,152 @@ +# PR-C `jump_amount_corr_20` 截断修正 —— 两份冻结面板的 provenance + +> **状态**：correctness fix 交付物（**不属于任何重构步骤**，见下「为什么单独成 PR」）。 +> 本文是入 git 的权威 provenance；两份 bulk 面板都在 `artifacts/refactor_baseline/` +> （gitignored）。 + +--- + +## 一、缺陷（一句话） + +`compute_jump_amount_corr` 是十一个分钟因子里**唯一没有 14:50 日内截断**的一个：它建于 +2026-07-19「全日数据因子一律截断到 14:50」这条常设授权**之前**，此后没有被补上。它在 +日 `d` 的取值因此吃进当天 14:50–15:00 的 1min bar。 + +在 PR #79 把入场锚从 `close(d)` 移到 **14:51 VWAP** 之后，这不再只是「定义与声明不符」， +而是**严格前视**：因子的信息集越过了它自己的入场锚十分钟，且正是 A 股全天成交最集中的 +窗口。已发布的 exec artifact 里 `spec.decision_cutoff="14:50:00"` —— 报告声明了一个它 +没有执行的检查。 + +修法：改用兄弟因子（ideal-amplitude / amp-anomaly / amp-cut）**完全相同**的 +`factors.compute.minute.primitives.visible_minute_frame`，不新造机制、不动相关系数数学。 + +## 二、爆炸半径：哪条取值路径受影响，哪条不受 + +| 取值路径 | 修正前 | 说明 | +|---|---|---| +| 旧 eval runner `qt/eval_jump_amount_corr.py` | **有缺陷** | `end = 当日 + 23:59:59` 读整日 bar，直接喂 `compute_*`，全程无补救。**已发布的 22 份 artifact 走的就是这条**。 | +| `qt/panel_freeze.py`（D1 冻结） | **有缺陷** | 它调的就是 runner 的 `_load_jump_factor_panel`，所以 `artifacts/refactor_baseline/panels/jump_amount_corr_20.parquet` 继承同一缺陷。 | +| D4 materializer **decision 视图** | **本来就对** | `materialize.py` 在调 binding 前先跑 `minute_decision_cutoff`，与 `visible_minute_frame` 是**同一个谓词**（`available_time <= trade_date + decision_time`），因此双重过滤幂等 —— 这条路径的取值**逐位不变**。 | +| D4 materializer **close 视图** | 有缺陷 | close 视图不做日内截断，靠因子自己截。修正后与 decision 视图对齐（十个兄弟本来就是这样）。 | + +`factors/compute/minute/binding.py` 的模块 docstring 早就写着「compute 函数以其 +**默认 decision_time（14:50）作为冗余但一致的内部截断**」—— 这句话对十一个因子里的十个 +成立，对 jump **不成立**。修正后它对全部十一个成立。 + +## 三、两份面板：各自是什么、该用在哪 + +| | `panels/jump_amount_corr_20.parquet` | `pr_c_cutoff_fix/panels/jump_amount_corr_20.parquet` | +|---|---|---| +| 定义 | 因子 **v1.0**，无 14:50 截断 | 因子 **v1.1**，截断后 | +| 产出 | D1 冻结 run（`main` @ `3669c90`），见 `d1_panel_freeze_manifest.md` | 见下 §四 | +| 引擎 | pre-D2 runner loader | **同一条 runner loader**（`qt.panel_freeze` + `--only`） | +| 用途 | **已发布内容的忠实记录**；D2 逐位对账的历史参照 | **D5 面板腿对该因子的有效参照** | +| 是否可被覆盖 | **否**（本次改动只新增，一个字节没动） | 可再生（命令在 §四） | + +**为什么「旧引擎 + 截断输入」是干净的 refactor-only 对照**：本次改动是**纯输入截断**—— +被喂进相关系数计算的 bar 少了一批，`compute_jump_amount_corr` 里从 `amp` 到 Pearson +闭式解的每一行代码一字未动。所以在**同一条 pre-D5 runner 取值路径**上重跑，得到的面板与 +原基线之间只差「截断」这一个变量；D5 拿它对账新路径时，任何差异都仍然只能归因于加载几何 / +饱和深度 / anchor 截断，而不会混进一个定义变更。 + +**不要**把两份面板放进同一次对账去「取平均」或「看哪个更接近」：它们是两个不同定义的因子值， +不是同一个量的两次测量。 + +## 四、新参照面板的产生方式（可复跑） + +``` +cd # 缓存根 artifacts/cache/tushare/v1 就位 +/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.panel_freeze \ + --config config/phase_c_jump_amount_corr.yaml \ + --output-root artifacts/refactor_baseline/pr_c_cutoff_fix \ + --only jump_amount_corr_20 +``` + +- `--only` 是本次为此新增的 **additive** 参数（默认 `None` = 全部 14 个因子，行为逐字节 + 不变）。它存在的理由：**不为一个因子重造第二套冻结实现**，也不为十三个没变的因子重读 + 2.79 亿行分钟缓存。选择在昂贵的数据面之前就被校验，未知 id 是可读报错 —— 「froze 0 + panels」被当成功报出去，正是本仓记录在案的空对账形态。 +- 共享数据面（universe / 日频面板 / 富化）**照旧用完整因子簿**构建，所以过滤 run 看到的 + 日频面板与整跑完全相同；`--only` 只决定**冻什么**，不决定**加载什么**。 +- 该 run 的 manifest 自报 `selected_factors`，与原基线 manifest（`selected_factors: all`） + 一眼可分。 +- ⚠️ 与原基线的 provenance 规则**刻意不同**：D1 基线要求「只许从钉住的 pre-D2 SHA + checkout 重跑」；本参照面板**必须**在含有截断修正的树上跑（这正是它要记录的东西）。 + 它的 `producing_git_sha` 记在自己的 manifest header 里。 + +## 五、实测结果（本次 run 填入） + +见 `artifacts/refactor_baseline/pr_c_cutoff_fix/manifest.md` / `manifest.json`。要点： + +- `stk_mins_live_calls = 0`（cache-only，与 D1 冻结同约束）； +- 写盘**前**先过 `_process_factors` + 与新 eval artifact 的 `data_coverage` 对账 + （`panel_rows` / `evaluation_periods` / `symbols_evaluated` / + `universe_symbols_declared` / `dropped_symbols_count` / `factor_nan_rate`）， + 对不上就 raise、不落盘； +- 确定性双跑（`jump_amount_corr_20` 是 `DETERMINISM_FACTORS` 成员）跨进程重建一次， + canonical hash 必须相等。 + +## 六、重述后的评估结果（CSI500 2021-07-01..2026-06-30，cache-only，`stk_mins_live_calls=0`） + +**唯一变量是截断**：同一条旧 runner、同一配置、同一缓存、同一评估器；另外只有两处 +**非数值**元数据随定义变更（`spec.version` 1.0→1.1、`spec.description` 增加更正段）， +它们进报告标题/dashboard 文本，不进任何计算。 + +**operative basis = `exec_to_exec`（14:51 VWAP）**： + +| 指标 | old v1.0（未截断，取自冻结副本） | new v1.1（截断后） | +|---|---|---| +| IC mean | −0.030840 | −0.030539 | +| ICIR | −0.425348 | −0.426002 | +| ICIR 95% CI（N_eff） | [−0.485384, −0.365312] | [−0.485645, −0.366360] | +| N_eff | 1162.21 | **1177.89** | +| NW-t | −15.191 | −15.260 | +| win rate | 0.66915 | 0.67246 | +| 按日单调 | −0.059222，CI [−0.093645, −0.024799] | −0.059636，CI [−0.093840, −0.025432] | +| 换手（多空腿） | 0.414632 | 0.417067 | +| 净多空 1× | −0.000834 | −0.000843 | +| 增量 ICIR（with_book） | −0.300601，CI [−0.363276, −0.237926] | −0.299010，CI [−0.361400, −0.236619] | +| **verdict no_book / with_book** | **Watch / Watch** | **Watch / Watch**（三轴逐一相同） | + +`close_to_close` 并排对照（保留作控制，正在退出评估契约）：IC mean −0.029845 → −0.029548； +ICIR −0.400976 → −0.401519；win rate 0.66584 → 0.66088；增量 ICIR −0.290002 → −0.288083； +verdict 同样 **Watch / Watch** 不变。 + +⚠️ **不许把「聚合指标只动了一点」读成「缺陷无害」。** 这两件事互相独立： +- **每一个** emitted cell 都被污染过（实测：真实缓存样本 1,477/1,477 与 1,373/1,373 全动， + max|diff| 1.28 / 1.05；只扰动 `bar_end >= 14:51` 时 1,363/1,363 动、max|diff| 1.30）。 +- 聚合的秩 IC 变化小，是因为被污染的 bar 只占 20 日池化窗口的约 4.5%，且秩 IC 逐日封顶 —— + **这是稀释，不是无害**。而且这个「小」是**跑出来的**，不是从截断前后值的 pearson r=0.9993 + 推出来的：相关系数高与 verdict 是否存活没有推理关系，所以必须重跑。 +- verdict 不变**不追认旧 artifact**：旧值在 exec 基准下是前视产物，无论它当时给出什么标签， + 都不构成一个诚实的结论。现在这一版才是。 + +## 七、落地清单（合并顺序敏感） + +并行的 D5 分支新增了一条**有意的响亮阻断**：`analytics/eval` 的 `subject_view(factor_id)` +从事实派生信息集视图，事实源是 `factors/compute/minute/binding.py` 的 deny list +`NOT_DECISION_CUTOFF_SAFE`；`jump_amount_corr_20` 在表里 ⇒ 派生出 (close, exec_to_exec) +非法配对 ⇒ `EvalConfig` 拒绝 ⇒ 它的 exec 评估 loud raise。在截断修正之前这是对的（不存在 +它的诚实 exec artifact，阻断优于假声明）。 + +**本修正正是解除条件**，因此： + +1. **合并顺序上靠后的那个 PR 负责删除 `NOT_DECISION_CUTOFF_SAFE` 里的 `jump_amount_corr` + 一条。** 本 PR 基于 `main@d806e70`，那时 deny list、`subject_view` 与 + `tests/test_decision_cutoff_visibility.py` **都还不存在**（已核实：全仓零命中），所以本 + 分支上没有可删的东西；若 D5 先合并，本分支 rebase 后即删。 +2. **不要让它依赖谁记得**：deny list 应当**断言等于**可见性测量出来的不安全集合，而不是 + 手工维护 —— 那样 `jump_amount_corr` 一旦变安全，测试会自己变红逼出删除。否则两边都合并 + 而无人删除时，这个因子会**永远**评不出 exec 结果，而且从不跑该评估的人看不见 —— + 正是本仓记录在案的「行为变了而措辞没变」家族。 +3. **两处测量应合一**：本 PR 的 `tests/test_minute_decision_cutoff_leakage.py` 对全部 11 个 + 分钟因子 + `mmp_ew` 测的就是同一个性质。两边合并后会存在两处事实源，应由靠后的那个 PR + 收敛成一处（author-once），而不是并存。 + +## 八、为什么单独成 PR（不许夹带进 D5） + +这条截断本身是**研究侧**常设授权的补用（用户 2026-07-19：「全日数据因子：一律截断到 +14:50…授权直接截断，不必逐因子再问」），**不是新的研究决策**。但从**重构视角**它仍然是一次 +**定义变更**（已发布因子的取值变了），而设计 §〇 明令重构不改定义。两者不矛盾 —— 授权来自 +研究侧，不来自重构。所以它自己一个 PR、显式标注 correctness fix，D5 的任何 commit 都不许 +夹带它。 diff --git a/qt/panel_freeze.py b/qt/panel_freeze.py index 2099c78..bb32ef0 100644 --- a/qt/panel_freeze.py +++ b/qt/panel_freeze.py @@ -58,6 +58,7 @@ import os import subprocess import time +from collections.abc import Sequence from dataclasses import dataclass from pathlib import Path from typing import Callable @@ -435,6 +436,45 @@ def add(module, factor, build) -> None: return recipes +def freezable_factor_ids() -> frozenset[str]: + """Every factor id this freeze can produce (3 book + 11 minute). + + Derived from the recipes and the runners' own book builder, so it cannot + drift from what the freeze actually writes. + """ + from qt.eval_jump_amount_corr import _build_book_factors + + return frozenset( + {f.name for f in _build_book_factors()} + | {r.factor_id for r in minute_recipes()} + ) + + +def resolve_selection(only: Sequence[str] | None) -> tuple[str, ...] | None: + """Validate a ``--only`` selection against the ids the freeze can produce. + + ``None`` (freeze everything) passes straight through — the default path is + untouched. A named id that this freeze does not produce is a readable error + rather than a silently empty output root: "froze 0 panels" reported as + success is how a correctness rerun quietly produces nothing. + """ + if only is None: + return None + selected = tuple(only) + if not selected: + raise ValueError( + "--only was given an empty selection; omit it to freeze all factors." + ) + known = freezable_factor_ids() + unknown = [fid for fid in selected if fid not in known] + if unknown: + raise ValueError( + f"--only names factor id(s) this freeze does not produce: {unknown}; " + f"known ids are {sorted(known)}." + ) + return selected + + # --------------------------------------------------------------------------- # # Artifact reconciliation (against the shipped eval JSONs) # --------------------------------------------------------------------------- # @@ -533,9 +573,18 @@ def run_panel_freeze( output_root: str | Path = DEFAULT_OUTPUT_ROOT, *, resume: bool = False, + only: Sequence[str] | None = None, ) -> FreezeResult: """Freeze all 14 RAW factor panels + verify determinism + reconcile artifacts. + ``only`` restricts the freeze to the named factor ids (default ``None`` = + all 14, byte-identical to before). It exists so a SINGLE factor whose + definition was corrected can be re-frozen into its OWN output root as a + reference panel, without a second freeze implementation and without + re-reading the 279M-row minute cache for the thirteen that did not change. + The shared data plane is built from the FULL book either way, so the daily + panel a filtered run computes on is identical to the full run's. + ``resume=True`` completes an interrupted freeze: a minute factor whose panel file already exists is NOT rebuilt from the minute cache — its RAW series is read back FROM the frozen file and then pushed through the SAME processing + @@ -561,6 +610,9 @@ def run_panel_freeze( ) started = time.monotonic() + # Validate the selection BEFORE the expensive data plane: a typo must cost a + # readable error, not an hour of cache reads followed by an empty freeze. + selected = resolve_selection(only) cfg = load_config(config_path) _check_preconditions(cfg) # tushare + cache-only + PIT index + neutralize, as the runners # Freeze-specific PanelStore name: identical content, never clobbers an @@ -592,11 +644,19 @@ def run_panel_freeze( canonical_by_id: dict[str, str] = {} live_calls_total = 0 + def _keep(factor_id: str) -> bool: + return selected is None or factor_id in selected + # -- book factors (raw, pre-processing) --------------------------------- # - book_raw = pd.concat( - [factor.compute(panel).rename(factor.name) for factor in book_factors], axis=1 + kept_book = [factor for factor in book_factors if _keep(factor.name)] + book_raw = ( + pd.concat( + [factor.compute(panel).rename(factor.name) for factor in kept_book], axis=1 + ) + if kept_book + else pd.DataFrame() ) - for factor in book_factors: + for factor in kept_book: raw = book_raw[factor.name].rename(factor.name) canonical = canonical_content_hash(raw) target = panels_dir / f"{factor.name}.parquet" @@ -607,7 +667,8 @@ def run_panel_freeze( # -- minute factors (raw, the runners' own loader chains) ---------------- # resumed: list[str] = [] - for recipe in minute_recipes(): + kept_recipes = [r for r in minute_recipes() if _keep(r.factor_id)] + for recipe in kept_recipes: target = panels_dir / f"{recipe.factor_id}.parquet" if resume and target.exists(): raw = read_frozen_panel(target, recipe.factor_id) @@ -650,7 +711,8 @@ def run_panel_freeze( # -- determinism double-run (2 minute + 1 book) -------------------------- # determinism: dict[str, dict] = {} - for factor_id in DETERMINISM_FACTORS: + determinism_subjects = tuple(fid for fid in DETERMINISM_FACTORS if _keep(fid)) + for factor_id in determinism_subjects: rebuilt = _rebuild_for_determinism( factor_id, cfg, symbols, panel, panel_dates, logger ) @@ -675,17 +737,19 @@ def run_panel_freeze( "universe_symbols": int(len(symbols)), "stk_mins_live_calls": int(live_calls_total), "resumed_factors": ",".join(resumed) if resumed else "none", + "selected_factors": "all" if selected is None else ",".join(selected), "elapsed_seconds": round(time.monotonic() - started, 1), "generated_utc": pd.Timestamp.now(tz="UTC").strftime("%Y-%m-%d %H:%M:%S"), "canonical_hash_version": CANONICAL_HASH_VERSION, "determinism_double_run": "; ".join( f"{fid}:{'ok' if determinism[fid]['ok'] else 'FAIL'}" - for fid in DETERMINISM_FACTORS - ), + for fid in determinism_subjects + ) + or "none (no determinism subject selected)", "artifact_reconciliation": ( - f"{len(reconciliations)}/11 minute factors reconciled against " - "eval_*_no_book.json data_coverage (book factors carry no coverage " - "fields in the eval artifacts — disclosed, not skipped silently)" + f"{len(reconciliations)}/{len(kept_recipes)} minute factors reconciled " + "against eval_*_no_book.json data_coverage (book factors carry no " + "coverage fields in the eval artifacts — disclosed, not skipped silently)" ), } @@ -757,8 +821,20 @@ def main(argv: list[str] | None = None) -> int: "reconciliation) instead of being rebuilt from the minute cache" ), ) + parser.add_argument( + "--only", + default=None, + help=( + "comma-separated factor ids to freeze (default: all 14). Used to " + "re-freeze a SINGLE corrected factor into its own --output-root; an " + "unknown id is a readable error, never a silently empty freeze" + ), + ) args = parser.parse_args(argv) - result = run_panel_freeze(args.config, args.output_root, resume=args.resume) + only = None if args.only is None else tuple(x for x in args.only.split(",") if x) + result = run_panel_freeze( + args.config, args.output_root, resume=args.resume, only=only + ) print(f"panels frozen: {len(result.rows)} -> {result.output_root / 'panels'}") print(f"manifest: {result.manifest_json}") print(f"stk_mins_live_calls: {result.header['stk_mins_live_calls']}") diff --git a/tests/test_panel_freeze.py b/tests/test_panel_freeze.py index 124f853..2bed9dc 100644 --- a/tests/test_panel_freeze.py +++ b/tests/test_panel_freeze.py @@ -33,10 +33,12 @@ atomic_write_parquet, canonical_content_hash, file_sha256, + freezable_factor_ids, manifest_row, read_frozen_panel, reconcile_with_eval_artifact, render_manifest_markdown, + resolve_selection, ) @@ -313,3 +315,39 @@ def test_reconcile_missing_payload_is_loud(tmp_path: Path): processed = _panel([1.0, 2.0, 3.0, 4.0], KEYS) with pytest.raises(ValueError, match="data_coverage"): reconcile_with_eval_artifact("eval_x", processed, ["000001.SZ"], tmp_path) + + +# --------------------------------------------------------------------------- # +# --only selection (added for the PR-C intraday-cutoff correctness re-freeze) +# --------------------------------------------------------------------------- # +def test_selection_default_is_none_so_the_full_freeze_is_untouched(): + assert resolve_selection(None) is None + + +def test_selection_keeps_the_given_order_and_accepts_book_and_minute_ids(): + assert resolve_selection(["jump_amount_corr_20", "value_ep"]) == ( + "jump_amount_corr_20", + "value_ep", + ) + + +def test_selection_rejects_an_unknown_factor_id(): + """A typo must cost a readable error, not an empty output root reported as + success — "froze 0 panels" is how a correctness re-freeze silently produces + nothing at all.""" + with pytest.raises(ValueError, match="does not produce"): + resolve_selection(["jump_amount_corr"]) # missing the _20 window suffix + + +def test_selection_rejects_an_empty_selection(): + with pytest.raises(ValueError, match="empty selection"): + resolve_selection([]) + + +def test_freezable_ids_are_the_14_the_freeze_writes(): + ids = freezable_factor_ids() + assert len(ids) == 14 + assert {"value_ep", "value_bp", "volatility_20"} <= ids + # Every determinism subject must be freezable, or --only could silently drop + # the double-run subject the manifest still claims to have checked. + assert set(DETERMINISM_FACTORS) <= ids From 199a152c93f8f94248227d201d06d635705ae6d9 Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Sat, 25 Jul 2026 16:36:25 -0700 Subject: [PATCH 3/7] fix(eval): lift the jump_amount_corr exec-evaluation block, now that it is fixed D5b added a deliberate loud block: `subject_view` derives a factor's information set from `NOT_DECISION_CUTOFF_SAFE`, `jump_amount_corr_20` was on it, so (close, exec_to_exec) was refused and its exec evaluation raised. That was correct -- before the truncation fix there was no honest exec artifact of it, and a block beats a false `view=decision` claim. This branch's first commit is the release condition, so the entry is removed HERE rather than left for a later PR: the intermediate state "factor fixed, evaluation still blocked" never exists on main. INDEPENDENT CORROBORATION, and it is stronger than this branch's own assertion. `tests/test_decision_cutoff_visibility.py` was written by another author with its own fixture (12 names x 40 days x 240 bars). Run on the merged tree BEFORE removing the entry, it failed with exactly the two instructions carried out here: jump_amount_corr_20 no longer depends on post-14:50:00 bars. If that was deliberate, this factor's PUBLISHED values changed: drop it from factors.compute.minute.binding.NOT_DECISION_CUTOFF_SAFE, and re-state its verdict rather than letting the artifacts drift silently. The factor now sits in that file's clean parametrization (all ten bars-bound minute factors, 0 cells moved), so a regression re-introducing the leak is red again. The two follow-on sites are updated to statements that are still TRUE rather than weakened: * `test_the_known_exception_is_still_exactly_one_factor_and_still_leaks` -> `test_the_deny_list_is_empty_and_that_emptiness_is_a_measurement`. It asserts BOTH that the list is empty AND that the measured set equals the bound set: "empty" alone would also pass for a list nobody ever measured. * `test_a_factor_that_is_not_cutoff_safe_cannot_get_an_exec_identity` keeps its property by INJECTING an offender (monkeypatched deny list) instead of naming a factor that is no longer broken. Deleting the test would drop the guard on the day the next offender appears; leaving the old name in place would be the stale-wording failure this repo keeps re-learning. * the D5 runner-difference catalogue entry becomes "released, in this PR", and points the C5 panel leg at the truncated reference panel. The D5b review NIT (the block fires late in the runner, after the expensive per-symbol minute aggregation and after both close_to_close reports are written) disappears with the entry and needs no separate handling. STILL OPEN, recorded rather than papered over: two places now measure the same property -- that file and this branch's `test_minute_decision_cutoff_leakage.py` (11 Factor subclasses + mmp_ew, two knives, coverage closure). Both being true is useful today; author-once consolidation is a follow-up, and the leakage file's coverage is the strict superset. --- .../factors/d5_runner_difference_catalogue.md | 8 ++- .../pr_c_cutoff_fix_reference_panel.md | 60 +++++++++++----- factors/compute/minute/binding.py | 9 ++- tests/test_decision_cutoff_visibility.py | 70 +++++++++++-------- tests/test_eval_contract_v1.py | 31 ++++++-- 5 files changed, 121 insertions(+), 57 deletions(-) diff --git a/docs/factors/d5_runner_difference_catalogue.md b/docs/factors/d5_runner_difference_catalogue.md index 35bf1ed..e342940 100644 --- a/docs/factors/d5_runner_difference_catalogue.md +++ b/docs/factors/d5_runner_difference_catalogue.md @@ -280,7 +280,13 @@ docstring，按 #74 先例）。它改变 artifact 的**三处且仅三处**， ⚠️ **`book_view` 使 exec 侧 no_book 与 with_book 两份 artifact 的 `eval_config` 块首次不同**（`null` vs `"close"`）。这是**有意**的：一次 with-book 评估携带**两个**信息集（候选因子的与因子簿的），一个 `view` 字段表达不了；设计 §1.1 记录的因子簿 close-view 活缺陷要到 D7 才关，在那之前诚实的 artifact 就该写 `view=decision, book_view=close`。对账时**不要**把这一处不同当作 no_book/with_book 之间的回归。 -⚠️ **`jump_amount_corr_20` 的 exec 评估现在会 loud raise**（`qt.exec_basis_eval.exec_identity`）：它的值是 close 视图（compute 无 14:50 截断，实测），(close, exec_to_exec) 非法配对。这是**故意的阻断**——在该因子被截断修正之前，它没有诚实的 exec artifact；而写一份声明 `view=decision` 的假 artifact 正是本字段要防的事。修正由**独立的 correctness-fix PR** 处理；它落地时从 `factors.compute.minute.binding.NOT_DECISION_CUTOFF_SAFE` 移除该条即可解除阻断。 +✅ **已解除（解除于截断修正 PR，本节保留作记录）**：`jump_amount_corr_20` 的 exec 评估曾经 loud raise（`qt.exec_basis_eval.exec_identity`）——它的值是 close 视图（compute 无 14:50 截断，实测），(close, exec_to_exec) 非法配对。那是**故意的阻断**：在该因子被截断修正之前，它没有诚实的 exec artifact，而写一份声明 `view=decision` 的假 artifact 正是本字段要防的事。 + +独立的 correctness-fix PR 已把该因子截断到 14:50（用与十个兄弟相同的 `visible_minute_frame`），并在**同一个 PR 里**从 `factors.compute.minute.binding.NOT_DECISION_CUTOFF_SAFE` 移除该条 + 用旧 runner 重跑重述了它的 exec verdict（Watch/Watch 不变，数字见 `docs/factors/pr_c_cutoff_fix_reference_panel.md` §六），所以不存在「因子已修好、评估仍被拦」的中间态。`NOT_DECISION_CUTOFF_SAFE` 现在为空集，且这个空集是**被测量的**：`tests/test_decision_cutoff_visibility.py` 把十个 bars-bound 因子全部纳入 clean 参数化——**它正是逼出这次删除的东西**（截断落地后它红了，并在失败信息里给出「drop it from … and re-state its verdict」）。 + +⚠️ 连带影响（D5b 评审的 NIT，自动消失）：那个阻断点在 runner 里偏晚（跑完逐票分钟聚合、写完两份 close_to_close 报告之后才 raise）。deny list 条目移除后该路径不再 raise，此 NIT 无需单独处理。 + +⚠️ **C5 面板腿注意**：该因子的 D1 冻结面板（`artifacts/refactor_baseline/panels/jump_amount_corr_20.parquet`）是用**未截断**定义冻的，仍按原样保留不动；对账请改用 `artifacts/refactor_baseline/pr_c_cutoff_fix/`（同一条旧 runner 取值路径 + 截断输入）。provenance 与「为什么两份并存」见 `docs/factors/pr_c_cutoff_fix_reference_panel.md`。 **没有变的**（`tests/test_eval_contract_v1.py` 逐值钉住）：`VerdictThresholds` 的**每一个** 默认门（`min_abs_icir=0.30` / `min_incremental_abs_icir=0.15` / diff --git a/docs/factors/pr_c_cutoff_fix_reference_panel.md b/docs/factors/pr_c_cutoff_fix_reference_panel.md index 03f41fc..2fd68ee 100644 --- a/docs/factors/pr_c_cutoff_fix_reference_panel.md +++ b/docs/factors/pr_c_cutoff_fix_reference_panel.md @@ -121,27 +121,51 @@ verdict 同样 **Watch / Watch** 不变。 - verdict 不变**不追认旧 artifact**：旧值在 exec 基准下是前视产物，无论它当时给出什么标签， 都不构成一个诚实的结论。现在这一版才是。 -## 七、落地清单（合并顺序敏感） +## 七、deny list 的解除（本 PR 内完成） -并行的 D5 分支新增了一条**有意的响亮阻断**：`analytics/eval` 的 `subject_view(factor_id)` +D5b（PR #99）新增了一条**有意的响亮阻断**：`qt.exec_basis_eval.subject_view(factor_id)` 从事实派生信息集视图，事实源是 `factors/compute/minute/binding.py` 的 deny list `NOT_DECISION_CUTOFF_SAFE`；`jump_amount_corr_20` 在表里 ⇒ 派生出 (close, exec_to_exec) -非法配对 ⇒ `EvalConfig` 拒绝 ⇒ 它的 exec 评估 loud raise。在截断修正之前这是对的（不存在 -它的诚实 exec artifact，阻断优于假声明）。 - -**本修正正是解除条件**，因此： - -1. **合并顺序上靠后的那个 PR 负责删除 `NOT_DECISION_CUTOFF_SAFE` 里的 `jump_amount_corr` - 一条。** 本 PR 基于 `main@d806e70`，那时 deny list、`subject_view` 与 - `tests/test_decision_cutoff_visibility.py` **都还不存在**（已核实：全仓零命中），所以本 - 分支上没有可删的东西；若 D5 先合并，本分支 rebase 后即删。 -2. **不要让它依赖谁记得**：deny list 应当**断言等于**可见性测量出来的不安全集合，而不是 - 手工维护 —— 那样 `jump_amount_corr` 一旦变安全，测试会自己变红逼出删除。否则两边都合并 - 而无人删除时，这个因子会**永远**评不出 exec 结果，而且从不跑该评估的人看不见 —— - 正是本仓记录在案的「行为变了而措辞没变」家族。 -3. **两处测量应合一**：本 PR 的 `tests/test_minute_decision_cutoff_leakage.py` 对全部 11 个 - 分钟因子 + `mmp_ew` 测的就是同一个性质。两边合并后会存在两处事实源，应由靠后的那个 PR - 收敛成一处（author-once），而不是并存。 +非法配对 ⇒ `EvalConfig` 拒绝 ⇒ 它的 exec 评估 loud raise。在截断修正之前这是对的：不存在 +它的诚实 exec artifact，阻断优于假声明（红线 #9）。 + +**本修正正是解除条件**，且解除与重述在**同一个 PR** 内完成，不留「因子已修好、评估仍被拦」 +的中间态：本分支切自 `main@d806e70`（deny list 尚不存在，全仓零命中），合入 `main@435a798` +后删除该条，`NOT_DECISION_CUTOFF_SAFE` 变为空集。 + +**独立佐证（比本 PR 自己的断言更有说服力）**：D5b 的 +`tests/test_decision_cutoff_visibility.py` 是**另一个作者、另一套 fixture**（12 名 × 40 日 × +240 bar 随机游走）独立写的测量。合入后**先不删 deny list 直接跑**，它给出： + +``` +FAILED test_the_known_exception_is_still_exactly_one_factor_and_still_leaks +AssertionError: jump_amount_corr_20 no longer depends on post-14:50:00 bars. + If that was deliberate, this factor's PUBLISHED values changed: drop it from + factors.compute.minute.binding.NOT_DECISION_CUTOFF_SAFE, and re-state its + verdict rather than letting the artifacts drift silently. +``` + +—— 它测量到截断确实生效，并逐字给出了本 PR 随后执行的两个动作。删除后该因子进入 clean +参数化（十个 bars-bound 因子全测、全 0 cell 移动）。 + +配套改动（都不是削弱，是把主张改成仍然为真的那一个）： + +- `test_the_known_exception_is_still_exactly_one_factor_and_still_leaks` → + `test_the_deny_list_is_empty_and_that_emptiness_is_a_measurement`：断言**两件事**—— + deny list 为空 **且** 被测量的集合恰等于 bars-bound 集合。「空」与「测过」是不同的主张， + 只断言前者会让「从没测过」也通过。新 offender 仍被 clean 参数化抓住。 +- `tests/test_eval_contract_v1.py::test_a_factor_that_is_not_cutoff_safe_cannot_get_an_exec_identity`： + 该拒绝路径不再有真实 offender，于是**注入**一个（monkeypatch 把一个真因子类放进 deny + list）而不是删掉测试 —— 删掉会在下一个 offender 出现的那天没有守卫；继续点名一个已经修好 + 的因子则正是本仓反复吃亏的 stale-wording。 +- `docs/factors/d5_runner_difference_catalogue.md` §七之二 的「现在会 loud raise」条目改为 + 「已解除，解除于本 PR」。D5b 评审留的那条 NIT（阻断点在 runner 里偏晚）随之自动消失。 + +**仍待收敛（留给后续，本 PR 不做）**：现在有**两处**测量同一性质 —— +D5b 的 `tests/test_decision_cutoff_visibility.py`（10 个 bars-bound 因子，随机游走 fixture） +与本 PR 的 `tests/test_minute_decision_cutoff_leakage.py`（11 个 Factor 子类 + `mmp_ew`， +两把刀，带覆盖闭包）。二者互为独立复核，现在**同时为真**是好事；但长期应 author-once +收敛成一处（本 PR 的覆盖是严格超集）。此处如实记录，不假装已经合并。 ## 八、为什么单独成 PR（不许夹带进 D5） diff --git a/factors/compute/minute/binding.py b/factors/compute/minute/binding.py index 13f02b0..7fad248 100644 --- a/factors/compute/minute/binding.py +++ b/factors/compute/minute/binding.py @@ -404,7 +404,14 @@ def is_valid_day_pooled(factor: Factor) -> bool: #: #: Declared HERE, next to the other minute-factor partitions, because it is a fact #: about a factor's values — not about the evaluation path that has to consult it. -NOT_DECISION_CUTOFF_SAFE: frozenset[type[Factor]] = frozenset({JumpAmountCorrFactor}) +#: +#: CURRENTLY EMPTY, and that emptiness is MEASURED rather than assumed: the sole +#: entry was ``JumpAmountCorrFactor``, whose compute applied no truncation at all; +#: it now truncates at 14:50 like its ten siblings, and the measuring test moved it +#: into the clean parametrization (where a regression makes it red again). An empty +#: set still means "measured clean OR never measured" for anything outside +#: ``_MINUTE_STREAM_BINDINGS`` — the measuring test pins the bound ten explicitly. +NOT_DECISION_CUTOFF_SAFE: frozenset[type[Factor]] = frozenset() def is_decision_cutoff_safe(factor: Factor) -> bool: diff --git a/tests/test_decision_cutoff_visibility.py b/tests/test_decision_cutoff_visibility.py index ba0f980..8c9f761 100644 --- a/tests/test_decision_cutoff_visibility.py +++ b/tests/test_decision_cutoff_visibility.py @@ -15,17 +15,21 @@ the project already uses, aimed at the WITHIN-DAY boundary rather than the across-day one. -RESULT, ENCODED RATHER THAN DESCRIBED. Nine of the ten bars-bound minute factors are -clean. ``jump_amount_corr_20`` is not: its compute is the ONLY one of the eleven that -applies no decision-time truncation of its own (grep: zero ``decision_time`` / -``prepare_visible_minute_bars`` references in its module), so under the old runners -it saw 09:30-15:00 of day d. - -That is recorded as a KNOWN, NAMED exception rather than fixed. Truncating it would -change a published factor's values and could move its verdict — a definition --affecting research decision, not a refactor's business (design v3.2 §〇). The -exception is asserted POSITIVELY, so the day someone changes it, this test goes red -and the change has to be made deliberately instead of noticed later. +RESULT, ENCODED RATHER THAN DESCRIBED. All ten bars-bound minute factors are now +clean. ``jump_amount_corr_20`` was NOT when this file was written: its compute was +the ONLY one of the eleven that applied no decision-time truncation of its own +(grep: zero ``decision_time`` / ``prepare_visible_minute_bars`` references in its +module), so under the old runners it saw 09:30-15:00 of day d. + +It was recorded here as a KNOWN, NAMED exception rather than fixed, because +truncating it changes a published factor's values and could move its verdict — a +definition-affecting research decision, not a refactor's business (design v3.2 §〇). +The exception was asserted POSITIVELY, and that is precisely how the fix was forced +to be deliberate: the separate correctness-fix PR truncated the factor, THIS test +went red with the instruction to drop the entry and re-state the verdict, and the +entry was dropped in the same PR that re-stated it. The factor has moved into the +clean parametrization below, so a regression that re-introduces the leak is red +again — the positive assertion did its job and is not needed a second time. THE LIST IS PRODUCTION STATE, NOT TEST STATE. It lives in ``factors.compute.minute.binding.NOT_DECISION_CUTOFF_SAFE`` because the exec path @@ -45,6 +49,7 @@ Measured on the real cache the same way (12 CSI500 names, 2021-07..2021-12, 361,500 bars, 4.6% of them post-cutoff): nine factors moved 0 cells; ``jump_amount_corr_20`` moved 1,477 of 1,477 with max |diff| = 1.276 on a correlation bounded in [-1, 1]. +After the truncation fix it moves 0 of its cells here, like the other nine. """ from __future__ import annotations @@ -149,7 +154,11 @@ def _moved_cells(factor_id: str) -> tuple[int, int, float]: @pytest.mark.parametrize("factor_id", CLEAN_FACTOR_IDS) def test_value_does_not_depend_on_bars_after_the_decision_cutoff(factor_id): - """The nine that truncate for themselves: perturbing the future moves nothing.""" + """Every bars-bound minute factor: perturbing the future moves nothing. + + ``jump_amount_corr_20`` joined this parametrization when its truncation was + fixed; before the fix it moved 1,477/1,477 cells here. + """ cells, moved, worst = _moved_cells(factor_id) assert cells > 0, f"{factor_id}: no comparable cells — the test would be vacuous" assert moved == 0, ( @@ -159,27 +168,26 @@ def test_value_does_not_depend_on_bars_after_the_decision_cutoff(factor_id): ) -def test_the_known_exception_is_still_exactly_one_factor_and_still_leaks(): - """jump_amount_corr_20: recorded as a defect, asserted positively. +def test_the_deny_list_is_empty_and_that_emptiness_is_a_measurement(): + """No known exception remains — and the claim is backed, not merely absent. - Asserted in the POSITIVE direction on purpose. A test that merely tolerated the - exception would stay green whether it was fixed, worsened, or spread to a second - factor. This one goes red on any of those, which is what forces the decision to - be taken deliberately — and because the exec path DERIVES its right to declare - ``view=decision`` from the same list, a stale entry here is not cosmetic. + "Empty deny list" and "every factor measured clean" are DIFFERENT statements: + a list can be empty because nothing was ever measured. So this asserts both, + and that the measured set is exactly the bound set — the emptiness is only + worth anything while the parametrization above covers every bound factor. + + The old positive assertion (jump_amount_corr_20 is on the list AND still + leaks) is gone because its subject is gone, not because it was inconvenient: + it fired, red, on the branch that fixed the factor, and its message is what + told that branch to drop the entry and re-state the verdict. A new offender + is still caught — by the clean parametrization, which is where it belongs. """ - assert KNOWN_POST_CUTOFF_DEPENDENT_IDS == {"jump_amount_corr_20"} - for factor_id in sorted(KNOWN_POST_CUTOFF_DEPENDENT_IDS): - assert factor_id in BOUND_FACTOR_IDS - cells, moved, worst = _moved_cells(factor_id) - assert cells > 0 - assert moved > 0, ( - f"{factor_id} no longer depends on post-{CUTOFF} bars. If that was " - f"deliberate, this factor's PUBLISHED values changed: drop it from " - f"factors.compute.minute.binding.NOT_DECISION_CUTOFF_SAFE, and re-state " - f"its verdict rather than letting the artifacts drift silently." - ) - assert worst > 0.0 + assert KNOWN_POST_CUTOFF_DEPENDENT_IDS == frozenset() + assert set(CLEAN_FACTOR_IDS) == set(BOUND_FACTOR_IDS) + assert len(BOUND_FACTOR_IDS) == 10, ( + "the bound minute-factor set changed; the emptiness above only covers " + "what this file measures, so re-check the new one before trusting it" + ) def test_every_clean_factor_is_absent_from_the_production_deny_list(): diff --git a/tests/test_eval_contract_v1.py b/tests/test_eval_contract_v1.py index c899dcc..79ba864 100644 --- a/tests/test_eval_contract_v1.py +++ b/tests/test_eval_contract_v1.py @@ -96,19 +96,38 @@ def test_the_exec_basis_module_restates_the_identity_it_scores_on(): } -def test_a_factor_that_is_not_cutoff_safe_cannot_get_an_exec_identity(): +def test_a_factor_that_is_not_cutoff_safe_cannot_get_an_exec_identity(monkeypatch): """The derivation refuses rather than declaring a view the values do not have. - ``jump_amount_corr_20``'s compute applies no 14:50 truncation, so its values are - close-view; (close, exec_to_exec) is not a legal pairing, and that refusal is - the correct outcome — there is no honest exec artifact of it until it is fixed. + ``jump_amount_corr_20`` used to BE the live example: its compute applied no + 14:50 truncation, so its values were close-view and (close, exec_to_exec) is + not a legal pairing. It was fixed and left ``NOT_DECISION_CUTOFF_SAFE``, so + the deny list is now empty and no real factor exercises this path. + + The refusal is still the property under test, so the offender is INJECTED + rather than the test deleted: a real factor class is put on the deny list for + the duration, and the derivation must refuse it. Keeping a fixed factor named + here as though it were still broken would be the stale-wording failure this + repo keeps re-learning; deleting the test would drop the guard on the day the + next offender appears. """ + from factors.compute.minute import binding as binding_module + from factors.compute.minute.volume_peak_count import VolumePeakCountFactor from qt.exec_basis_eval import exec_identity, subject_view - assert subject_view("jump_amount_corr_20") == "close" + # Nothing real is on the list any more: every bound factor derives decision-view. + assert binding_module.NOT_DECISION_CUTOFF_SAFE == frozenset() assert subject_view("volume_peak_count_20") == "decision" + assert subject_view("jump_amount_corr_20") == "decision" + + monkeypatch.setattr( + binding_module, + "NOT_DECISION_CUTOFF_SAFE", + frozenset({VolumePeakCountFactor}), + ) + assert subject_view("volume_peak_count_20") == "close" with pytest.raises(ValueError, match="no 14:50 truncation of its own"): - exec_identity(_cfg(), factor_id="jump_amount_corr_20", book_view="close") + exec_identity(_cfg(), factor_id="volume_peak_count_20", book_view="close") def test_the_no_book_and_with_book_runs_do_not_share_one_book_view(): From ffe40d13d685544d6bca44529cc3b54e8f2198da Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Sat, 25 Jul 2026 16:58:12 -0700 Subject: [PATCH 4/7] docs(factors): record the PR-C reference-panel manifest and the restated verdicts The bulk panel and its machine-readable manifest are gitignored, so the authoritative numbers live in the doc (same reason the D1 manifest doc carries its full table). Recorded: producing SHA, canonical/file sha256, row and NaN counts against the D1 baseline (same grid, 11 more honest NaNs), and the fact that the panel was frozen twice -- once on the uncommitted tree, once on the committed one -- with byte-identical hashes. The first freeze is not kept: its producing_git_sha pointed at the branch base, which does NOT contain the fix. A provenance line that names a tree the artifact did not come from is worse than no line, so it was re-run rather than annotated. --- .../pr_c_cutoff_fix_reference_panel.md | 44 ++++++++++++++----- 1 file changed, 33 insertions(+), 11 deletions(-) diff --git a/docs/factors/pr_c_cutoff_fix_reference_panel.md b/docs/factors/pr_c_cutoff_fix_reference_panel.md index 2fd68ee..a93a727 100644 --- a/docs/factors/pr_c_cutoff_fix_reference_panel.md +++ b/docs/factors/pr_c_cutoff_fix_reference_panel.md @@ -74,17 +74,39 @@ cd # 缓存根 artifacts/cache/tushare/v1 就位 checkout 重跑」；本参照面板**必须**在含有截断修正的树上跑（这正是它要记录的东西）。 它的 `producing_git_sha` 记在自己的 manifest header 里。 -## 五、实测结果（本次 run 填入） - -见 `artifacts/refactor_baseline/pr_c_cutoff_fix/manifest.md` / `manifest.json`。要点： - -- `stk_mins_live_calls = 0`（cache-only，与 D1 冻结同约束）； -- 写盘**前**先过 `_process_factors` + 与新 eval artifact 的 `data_coverage` 对账 - （`panel_rows` / `evaluation_periods` / `symbols_evaluated` / - `universe_symbols_declared` / `dropped_symbols_count` / `factor_nan_rate`）， - 对不上就 raise、不落盘； -- 确定性双跑（`jump_amount_corr_20` 是 `DETERMINISM_FACTORS` 成员）跨进程重建一次， - canonical hash 必须相等。 +## 五、实测结果（bulk artifact gitignored，故权威数字记在这里） + +机器可读版在 `artifacts/refactor_baseline/pr_c_cutoff_fix/manifest.{json,md}`。 + +| 项 | 值 | +|---|---| +| producing_git_sha | `199a152c93f8f94248227d201d06d635705ae6d9`（含截断修正的本分支 commit） | +| selected_factors | `jump_amount_corr_20` | +| universe / 窗口 | `index:000905.SH` / 2021-07-01..2026-06-30（996 成分、1210 交易日、面板 1,158,912 行） | +| `stk_mins_live_calls` | **0**（cache-only，与 D1 冻结同约束） | +| determinism_double_run | `jump_amount_corr_20:ok` | +| artifact_reconciliation | 1/1（对新 eval artifact 的 `data_coverage`） | +| elapsed | 729.8 s | + +| factor_id | rows | n_symbols | n_nan | mean | std | canonical_sha256 | +|---|---|---|---|---|---|---| +| jump_amount_corr_20 | 1159263 | 995 | **891** | **0.5929255105118916** | **0.1575466344089912** | `fafda48514d6e99055bd070a420250e388655fe11c519a9905961e71cffb2707` | + +对照 D1 原基线同一因子（**未截断**）：`n_nan` 880 / mean 0.5912439770493371 / +std 0.15727054230823656 / canonical `b6359f12…`。行数、日期范围、symbol 数完全相同 —— +变的是值本身与 891−880 = **11 个新增 NaU**（截断后某些日的 jump-pair 数掉到 `min_pairs` +以下，honest missing）。 + +**免费拿到的确定性证据**：本参照面板被冻结过**两次**——一次在未提交的工作树上、一次在 +提交后的 `199a152` 上，两次跑在不同进程、不同时刻，`canonical_sha256` 与 `file_sha256` +**逐字节相同**。第一次的 manifest 因此被第二次覆盖（它的 `producing_git_sha` 指向 +`d806e70`，即**不含**修正的分支基点 —— 那是一句会误导人的 provenance，所以重跑而不是 +留着解释）。 + +写盘**前**先过 `_process_factors` + 与新 eval artifact 的 `data_coverage` 对账 +（`panel_rows` / `evaluation_periods` / `symbols_evaluated` / +`universe_symbols_declared` / `dropped_symbols_count` / `factor_nan_rate`）， +对不上就 raise、不落盘。 ## 六、重述后的评估结果（CSI500 2021-07-01..2026-06-30，cache-only，`stk_mins_live_calls=0`） From 890bbeccb3c6213ab3677b0b859ed3ec92da08d4 Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Sat, 25 Jul 2026 18:11:20 -0700 Subject: [PATCH 5/7] fix(eval): carry the correction as structured data, not as capped prose SELF-INFLICTED, CAUGHT BY REVIEW. The first version of this branch wrote the "these values supersede previously published ones" disclosure as prose appended to `FactorSpec.description`, and the report claimed it reached the Markdown, the JSON and the dashboard. Two of those three were false: * JSON -- `sanitize_payload` caps every exported string at MAX_VALUE_CHARS (200) and appends `...[truncated]`. The disclosure started at character 353, so the MACHINE-READABLE copy -- the one a summary layer opens -- contained none of it. * PNG -- the longer description grew the FACTOR DEFINITION band from 3 lines to 5, and the 5th line printed on the same baseline as the metadata row. A report that fails to state the thing about its own provenance that it must state is the same shape as the factor defect that produced the correction. It landing exactly where the report claimed success is the part worth remembering. THE CAP IS GENERIC, NOT A QUIRK OF ONE PATH. Measured over the 44 shipped eval JSONs: 44/44 carry a truncated `spec.description`. So `description` is not a usable carrier for anything load-bearing, for any factor. CONTRACT v1.1: the correction becomes `FactorSpec.corrections`, a tuple of frozen `FactorCorrection` (from_version / to_version / date / defect / effect / superseded). Every field is required, and OVER-LENGTH RAISES at construction rather than trimming -- a carrier that can silently lose its content would rebuild the defect it exists to fix. `to_version` must equal the spec's own version, so `spec.version` alone tells a reader which side of the correction a stored artifact is on. Exported as a top-level `corrections` key via `corrections_record`, outside the capped path (still redacted). The Markdown provenance row and the dashboard marker derive from the SAME tuple, so the three surfaces cannot disagree. `description` goes back to being a definition. The census guard immediately earned itself: with corrections still a spec field, `vars(spec)` put a SECOND copy into the JSON -- a dataclass repr, cut at 200. Two carriers for one fact, one of them broken. `corrections` is now excluded from the spec dump. DASHBOARD OVERLAP -- fixed generically, and it is NOT this branch's regression. `_definition_band` anchors the description at y=0.62 and the metadata at y=0.20/0.10 with nothing in between, so a long description is simply drawn over the metadata. `analytics/eval/figures.py` has ONE commit in its history: every dashboard ever produced was drawn this way, including the 22 frozen ones. Measured on the real geometry, 7 of the 11 minute factors overlap (valley_price_quantile_20 by 296 px, 25 wrapped lines). `DEFINITION_MAX_LINES = 5` is MEASURED, not chosen: the gap falls 15 px per line and goes negative at six (1..5 give +63.8/+48.8/+33.8/+18.8/+3.8). Five keeps every factor that fits today rendering in FULL -- including the two 5-line ones a tidier bound of 4 would have needlessly clipped -- and gives the seven that overflow an explicit elision marker pointing at the Markdown, which carries the description whole. Elision is disclosed; overprinting loses both texts silently. No factor's description text is touched: that would be the scope explosion. GUARDS (three files, none of them a substring match on rendered text -- D5b just lost a guard to exactly that): * tests/test_factor_correction_carrier.py (15) -- round-trips the DECLARED object through the real `report_to_dict` + json dumps/loads and asserts EQUALITY, plus pins that the fields really exceed the cap, so "it survived" is only provable while the carrier is outside the capped path. It also demonstrates the original mistake directly: prose past the cap inside `description` is shown being cut on export. * tests/test_definition_band_layout.py (25) -- renders the band on the REAL dashboard figsize/GridSpec and compares matplotlib window extents. Geometric, not visual: the failure being guarded is pixels landing on pixels. * tests/test_report_json_truncation_census.py (4) -- reads the JSON FROM DISK and asserts the WHOLE truncated-field set, not the one field just fixed (#82: a guard that scans only what you already fixed can only confirm what you already know). Mutation evidence, all run: * DEFINITION_MAX_LINES -> 999: 7 factors go red with measured overruns (11.2 / 56.2 / 71.2 / 128.7->191.2 / 206.2 / 296.2 px). * corrections=() on the shipped factor: 3 tests red across two files. * the census guard failed on the real corpus at `spec.corrections` before the double-carrier fix, and passes after regenerating. INHERITED, RECORDED, NOT FIXED HERE: six fields are truncated across the corpus (spec.description 44/44; monotonicity_spearman_by_date_ci_note 44/44; monotonicity_reversed_status 44/44; multiple_testing_note 44/44; vif_status 22/44; forward_return_source 20/44). All six are explanatory caveats cut mid-sentence -- monotonicity_reversed_status stops at "This evaluator scores ONE cell and cannot ...", i.e. the half stating what it CANNOT do is the half that is gone. They predate this branch (the frozen #79 baseline has them), so the set is RECORDED and asserted exactly rather than fixed under a correctness PR: a seventh cannot join quietly, and fixing one forces the record to be updated. Ruled out while measuring: verdict reasons and section notes go through `sanitize_text` only and are NOT capped -- proved on disk by a 401-char reason and a 1263-char note surviving whole. The genuine near-miss is `coverage_bias_bad_vwap` at 158 of 200. --- analytics/eval/contract.py | 36 ++- analytics/eval/figures.py | 75 ++++- analytics/eval/render.py | 58 +++- .../factors/d5_runner_difference_catalogue.md | 20 ++ .../pr_c_cutoff_fix_reference_panel.md | 73 ++++- factors/compute/minute/jump_amount_corr.py | 47 ++- factors/spec.py | 131 ++++++++ tests/test_definition_band_layout.py | 172 +++++++++++ tests/test_eval_contract_v1.py | 10 +- tests/test_factor_correction_carrier.py | 281 ++++++++++++++++++ tests/test_report_json_truncation_census.py | 188 ++++++++++++ 11 files changed, 1078 insertions(+), 13 deletions(-) create mode 100644 tests/test_definition_band_layout.py create mode 100644 tests/test_factor_correction_carrier.py create mode 100644 tests/test_report_json_truncation_census.py diff --git a/analytics/eval/contract.py b/analytics/eval/contract.py index 050e5ac..b144a8c 100644 --- a/analytics/eval/contract.py +++ b/analytics/eval/contract.py @@ -43,13 +43,47 @@ (``eval_contract_version``) in every report JSON, plus four new provenance rows in the Markdown. No metric, no axis input and no threshold moves. The D5 C5 reconciliation registers this as a named, expected artifact drift. + +CONTRACT v1.1 — WHAT CHANGED AND WHY +------------------------------------ +v1.0 could state which information set a factor's values came from. It could not +state that those values SUPERSEDE previously published ones — and the obvious +place to write that, ``FactorSpec.description``, turned out not to work. + +``sanitize_payload`` caps every exported string at :data:`MAX_VALUE_CHARS` (200) +and appends ``...[truncated]``. That cap is correct for arbitrary payload values +and it is generic, not path-specific: measured across the 44 shipped eval JSONs, +**44/44** carry a truncated ``spec.description`` (218 ``[truncated]`` markers in +all, three of them methodological notes in every artifact). So a correction +written as prose in the description reaches the Markdown and the dashboard and is +CUT OUT of the JSON — the copy a summary layer reads. A machine consumer would +see restated numbers with no sign that they replaced a defective run: a report +failing to say the thing about its own provenance that it must say, which is the +same shape as the factor defect that produced the correction in the first place. + +v1.1 adds ONE top-level key, ``corrections``, built from the new structured +``FactorSpec.corrections`` (a tuple of ``FactorCorrection``). It is emitted OUTSIDE +the capped path (``analytics.eval.render.corrections_record``) — redacted, never +truncated — and over-length RAISES at spec construction instead of trimming, so +the carrier cannot silently lose the thing it exists to carry. The same tuple +renders one Markdown provenance row per correction and a marker on the dashboard, +so the three surfaces cannot disagree. + +An empty tuple is the default and means "no correction has been DECLARED" — never +"this factor is known to be correct". + +WHAT DID NOT CHANGE (v1.1) +-------------------------- +Every decision rule, every threshold, every metric, and the 200-char cap itself +(it is right for arbitrary payload values; the fix is to stop routing a +load-bearing disclosure through it). """ from __future__ import annotations #: The evaluation contract version this package implements. Bumped ONLY with a #: written statement of what changed (this module's docstring is that statement). -EVAL_CONTRACT_VERSION = "1.0" +EVAL_CONTRACT_VERSION = "1.1" #: The identity fields a v1.0 report must be able to state about itself. Named #: here once so the config validator, the renderer and the cross-basis summary diff --git a/analytics/eval/figures.py b/analytics/eval/figures.py index 9aee10f..09d6c42 100644 --- a/analytics/eval/figures.py +++ b/analytics/eval/figures.py @@ -218,6 +218,70 @@ def _definition_block_line(spec: FactorSpec) -> str | None: ) +#: Wrap width of the description inside the FACTOR DEFINITION band. +DEFINITION_WRAP_WIDTH = 150 + +#: How many wrapped lines the band's description slot can hold before the text +#: collides with the metadata row below it. NOT a style preference — the slot is +#: bounded by fixed axes-fraction anchors (description at 0.62 va="top", metadata +#: at 0.20/0.10 va="bottom"), so a longer description silently DRAWS OVER the +#: metadata. +#: +#: MEASURED, not chosen, on the real dashboard geometry (figsize 15x21.5, the +#: GridSpec of ``_render``): the gap between the two boxes falls 15 px per line and +#: goes negative at SIX lines — 1..5 lines give +63.8/+48.8/+33.8/+18.8/+3.8 px, +#: 6 gives -11.2. Five is therefore the capacity, and it is deliberately not +#: reduced for comfort: every factor whose description fits today keeps rendering +#: in full, and only the seven that genuinely overflow get an elision marker. +#: +#: The slack at five lines is thin (~4 px, a quarter of a line). That is a fact +#: about the layout, not a hidden risk: ``tests/test_definition_band_layout.py`` +#: re-measures every shipped spec against the real geometry, so a font or +#: matplotlib change that eats the slack turns the guard RED instead of silently +#: overprinting. The number is checked, never trusted. +DEFINITION_MAX_LINES = 5 + + +def definition_description_lines(spec) -> list[str]: + """The description lines the band may draw, bounded so it cannot overlap. + + Six of the eleven shipped minute factors have descriptions long enough to + collide with the metadata row (measured with the real renderer: + ``valley_price_quantile_20`` overlaps by 291 px, and it is 25 wrapped lines + against a 4-line slot). The renderer has ONE commit in its history, so those + dashboards have always been drawn that way — this is a standing defect, not a + regression, and "both texts unreadable" is the worst of the options. + + Bounding it visibly is strictly better than overdrawing: an elided tail is + marked and points at the copy that carries the description in full (the + Markdown provenance box — the JSON's is capped at + ``analytics.eval.render.MAX_VALUE_CHARS``). Nothing is dropped without saying + so, which is the whole difference between this and what it replaces. + """ + lines = textwrap.wrap(spec.description, width=DEFINITION_WRAP_WIDTH) + if len(lines) <= DEFINITION_MAX_LINES: + return lines + elided = len(lines) - (DEFINITION_MAX_LINES - 1) + return lines[: DEFINITION_MAX_LINES - 1] + [ + f"... [{elided} more lines elided to fit — FULL definition in the " + f"Markdown report's provenance box]" + ] + + +def _correction_marker(spec) -> str: + """Compact "this version supersedes published values" marker, or ``""``. + + Empty when nothing is declared — an unconditional badge would say something + about every factor, and "no correction declared" is not "known correct". + """ + corrections = tuple(getattr(spec, "corrections", ()) or ()) + if not corrections: + return "" + first = corrections[0] + extra = f" (+{len(corrections) - 1})" if len(corrections) > 1 else "" + return f"CORRECTED v{first.from_version} -> v{first.to_version}{extra}" + + def _definition_band(ax, data: DashboardData) -> None: """MANDATORY panel: how the factor is computed, from the FactorSpec. @@ -235,7 +299,16 @@ def _definition_band(ax, data: DashboardData) -> None: ax.text(0.185, 0.90, f"{spec.factor_id} (v{spec.version})", fontsize=9.5, fontweight="bold", color="#111111", transform=ax.transAxes, va="top", family="DejaVu Sans Mono") - ax.text(0.012, 0.62, textwrap.fill(spec.description, width=150), fontsize=9.2, + # Corrected factors say so ON the picture. The dashboard is the surface a + # reader is most likely to look at alone, and "this version supersedes + # published values" is not something they should have to open the JSON for. + # Sourced from the SAME structured tuple as the JSON block and the Markdown + # row — never re-worded here. + marker = _correction_marker(spec) + if marker: + ax.text(0.60, 0.90, marker, fontsize=9.0, fontweight="bold", color="#A33A00", + transform=ax.transAxes, va="top", family="DejaVu Sans Mono") + ax.text(0.012, 0.62, "\n".join(definition_description_lines(spec)), fontsize=9.2, color="#222222", transform=ax.transAxes, va="top") block = _definition_block_line(spec) meta_y = 0.20 if block is not None else 0.10 diff --git a/analytics/eval/render.py b/analytics/eval/render.py index e1ed9a2..4b4e937 100644 --- a/analytics/eval/render.py +++ b/analytics/eval/render.py @@ -87,6 +87,29 @@ def sanitize_payload(value: object) -> object: return cleaned +def corrections_record(spec) -> list[dict[str, str]]: + """``spec.corrections`` as plain dicts — REDACTED but never truncated. + + Deliberately NOT routed through :func:`sanitize_payload`. That function caps a + string at :data:`MAX_VALUE_CHARS` and appends ``...[truncated]``, which is right + for arbitrary payload values (a payload can accidentally hold a whole panel's + repr) and WRONG here: measured on the 44 shipped eval JSONs, 44/44 had a + truncated ``spec.description``, so a correction written as prose in the + description reached the Markdown and the dashboard and was cut out of the JSON — + the one copy a machine consumer reads. A record that drops the disclosure of its + own provenance is the same shape as the defect this project keeps re-learning. + + Safe to skip the cap because these are not arbitrary values: every field is an + authored, fixed-shape string that ``FactorCorrection`` already bounds at + construction, and over-length there RAISES rather than trimming. Redaction still + runs — dropping the cap must not drop the secret guard. + """ + return [ + {k: sanitize_text(str(v)) for k, v in c.as_dict().items()} + for c in (getattr(spec, "corrections", ()) or ()) + ] + + def format_value(value: object) -> str: """Compact deterministic rendering of one payload value.""" if isinstance(value, dict): @@ -150,6 +173,11 @@ def report_to_dict(report: FactorEvalReport) -> dict: entry["note"] = section.note sections.append(dict(sorted(entry.items()))) return { + # Contract v1.1: the factor's STRUCTURED correction statements, top level + # (sibling of eval_contract_version) rather than inside ``spec``, because + # the ``spec`` block goes through the 200-char payload cap and a superseded + # -values disclosure must never be the thing that gets trimmed. + "corrections": corrections_record(report.spec), "criteria_source": report.criteria_source, # Contract v1.0 (analytics/eval/contract.py): stated in the machine-readable # record so a stored verdict can be matched to the contract that produced it @@ -158,7 +186,15 @@ def report_to_dict(report: FactorEvalReport) -> dict: "eval_config": sanitize_payload(vars(report.cfg)), "schema_version": report.SCHEMA_VERSION, "sections": sections, - "spec": sanitize_payload(vars(report.spec)), + # ``corrections`` is dropped from the spec dump: it is exported whole at + # top level, and leaving it here too would put a SECOND copy in the record + # — one rendered by ``clean_value``'s ``str()`` fallback (a dataclass repr) + # and then cut at MAX_VALUE_CHARS. Two carriers for one fact, of which the + # in-spec one is broken, is worse than the single-carrier problem this + # whole field was added to fix. + "spec": sanitize_payload( + {k: v for k, v in vars(report.spec).items() if k != "corrections"} + ), # the ACTUAL criteria values used, whatever their source (design §6, v0.6). "thresholds": sanitize_payload(vars(report.thresholds)), # The DERIVED deployment label + the three axis-verdicts it was derived @@ -239,11 +275,31 @@ def _requires_row(spec) -> str: return ", ".join(sorted(f"{f.source}.{f.field}" for f in fields)) +def _correction_rows(spec) -> list[tuple[str, str]]: + """One provenance row per declared correction (none = no rows, not a blank one). + + A factor with nothing to correct must not grow an empty "corrections:" row — + "no correction has been DECLARED" and "this factor is known to be right" are + different statements, and a blank row invites reading the second one. + """ + return [ + ( + f"⚠️ CORRECTION (v{c['from_version']} -> v{c['to_version']}, {c['date']})", + f"{c['defect']} {c['effect']} {c['superseded']}", + ) + for c in corrections_record(spec) + ] + + def _provenance_rows(report: FactorEvalReport) -> list[tuple[str, str]]: spec, cfg = report.spec, report.cfg rows: list[tuple[str, object]] = [ ("factor_id", spec.factor_id), ("description", spec.description), + # Rendered from the SAME structured tuple the JSON's top-level + # ``corrections`` block is built from, so the human copy and the machine + # copy cannot disagree about whether this factor's values were superseded. + *_correction_rows(spec), ("family", spec.family), ("hypothesis (expected IC sign, fixed pre-run)", f"{spec.expected_ic_sign:+d}"), ("horizon / return basis", f"h={spec.forward_return_horizon} / {spec.return_basis}"), diff --git a/docs/factors/d5_runner_difference_catalogue.md b/docs/factors/d5_runner_difference_catalogue.md index e342940..d6801fe 100644 --- a/docs/factors/d5_runner_difference_catalogue.md +++ b/docs/factors/d5_runner_difference_catalogue.md @@ -277,6 +277,26 @@ docstring，按 #74 先例）。它改变 artifact 的**三处且仅三处**， | 1 | JSON `eval_config` 块 | **+3 键** `view` / `return_basis` / `book_view` | `report_to_dict` 的 `sanitize_payload(vars(report.cfg))` 自动带上 `EvalConfig` 的新字段（与 §七 的 `spec` 16→20 完全同一机制） | | 2 | JSON 顶层 | **+1 键** `eval_contract_version` | 显式写入：一个 verdict 只有对着产生它的契约版本才可解释（#74 的教训） | | 3 | Markdown `## 0. Header & Provenance` | **+4 行** `evaluation contract` / `requires (endpoint inputs)` / `adjustment / overnight boundary` / `lookback depth ...` | R24 的身份字段 + D1 契约 v1.0/v1.1 的三个声明维，从 `vars(spec)` 的 repr 转述升级为具名行 | +| 4 | JSON 顶层 | **+1 键** `corrections`（契约 **v1.1**，见下） | 更正是**结构化事实**，必须在**有长度上限的自由文本通道之外**传达 | +| 5 | Markdown `## 0.` + dashboard PNG | 每条更正 **+1 行** provenance 行；PNG 头行 **+1 个** `CORRECTED vX -> vY` 标记（无更正的因子零新增） | 与 #4 同一结构化元组派生（author once），三个呈现面不可能互相矛盾 | +| 6 | dashboard PNG FACTOR DEFINITION 带 | 描述被**限行**（`DEFINITION_MAX_LINES=4`），超出部分标注省略并指向 Markdown | **修既有缺陷**，见下 | + +**#4/#5 的成因（契约 v1.1）**：`sanitize_payload` 对**任何**导出字符串封顶 200 字符 +（`MAX_VALUE_CHARS`）并追加 `...[truncated]`。这是**通用行为、非某条路径特有**——实测 +已发布的 **44/44** 份 eval JSON 的 `spec.description` 全部被截断（共 218 个 +`[truncated]` 标记）。所以写进 `description` 的更正只到得了 Markdown 与 PNG，在 JSON —— +**汇总层读的那一份**——里被切掉。v1.1 把更正改为 `FactorSpec.corrections` +（`FactorCorrection` 元组），经 `corrections_record` 在**封顶通道之外**导出（仍脱敏）， +且超长在 spec 构造期 **raise 而非裁剪**。 + +**#6 的成因（既有缺陷，非本次回归）**：FACTOR DEFINITION 带把描述锚在 y=0.62（va=top）、 +metadata 行锚在 y=0.20/0.10（va=bottom），**中间没有任何约束**，长描述直接**画在 metadata +上面**，两行都糊。真实渲染几何实测：**11 个分钟因子里 7 个重叠** +（`valley_price_quantile_20` 超出 **296 px** / 25 行对 4 行槽位）。 +`analytics/eval/figures.py` **在 git 里只有一个 commit**，所以这不是新引入的——**每一份已 +产出的 dashboard 都是这样画的**，包括冻结的 22 张。限行 + 显式省略标记严格优于覆盖绘制 +（后者两段文字同时不可读且不声明）。守卫 `tests/test_definition_band_layout.py` 用**真实 +dashboard 几何**渲染后比对 Text 的 bbox，**几何断言而非看图**。 ⚠️ **`book_view` 使 exec 侧 no_book 与 with_book 两份 artifact 的 `eval_config` 块首次不同**（`null` vs `"close"`）。这是**有意**的：一次 with-book 评估携带**两个**信息集（候选因子的与因子簿的），一个 `view` 字段表达不了；设计 §1.1 记录的因子簿 close-view 活缺陷要到 D7 才关，在那之前诚实的 artifact 就该写 `view=decision, book_view=close`。对账时**不要**把这一处不同当作 no_book/with_book 之间的回归。 diff --git a/docs/factors/pr_c_cutoff_fix_reference_panel.md b/docs/factors/pr_c_cutoff_fix_reference_panel.md index a93a727..cdced0b 100644 --- a/docs/factors/pr_c_cutoff_fix_reference_panel.md +++ b/docs/factors/pr_c_cutoff_fix_reference_panel.md @@ -68,8 +68,10 @@ cd # 缓存根 artifacts/cache/tushare/v1 就位 panels」被当成功报出去，正是本仓记录在案的空对账形态。 - 共享数据面（universe / 日频面板 / 富化）**照旧用完整因子簿**构建，所以过滤 run 看到的 日频面板与整跑完全相同；`--only` 只决定**冻什么**，不决定**加载什么**。 -- 该 run 的 manifest 自报 `selected_factors`，与原基线 manifest（`selected_factors: all`） - 一眼可分。 +- 该 run 的 manifest header 自报 `selected_factors: jump_amount_corr_20`。**原基线 + manifest 里根本没有这个键**——它产生于 `--only` 存在之前，所以两者的区别是「有该键 vs + 无该键」，不是「两个不同的值」。（初稿曾写成原基线写着 `selected_factors: all`，评审实测 + 证否；如实更正。） - ⚠️ 与原基线的 provenance 规则**刻意不同**：D1 基线要求「只许从钉住的 pre-D2 SHA checkout 重跑」；本参照面板**必须**在含有截断修正的树上跑（这正是它要记录的东西）。 它的 `producing_git_sha` 记在自己的 manifest header 里。 @@ -143,6 +145,73 @@ verdict 同样 **Watch / Watch** 不变。 - verdict 不变**不追认旧 artifact**：旧值在 exec 基准下是前视产物，无论它当时给出什么标签， 都不构成一个诚实的结论。现在这一版才是。 +## 六之二、更正声明放在哪里（本 PR 第二次自查纠错） + +**第一版把更正写成 `spec.description` 的一段散文，报告称「四份 artifact 各含 1 处、同时进 +JSON 与 dashboard」——三个说法里两个是错的**： + +| 承载 | 第一版实况 | +|---|---| +| Markdown | ✅ 完整 | +| JSON（**机器可读的那一份**） | ❌ `spec.description` 被 `sanitize_payload` 封顶 **200 字符**并追加 `...[truncated]`；更正从第 353 字符起，**整段不在里面** | +| dashboard PNG | ❌ 加长的 description 使 FACTOR DEFINITION 带从 3 行涨到 5 行，**与下面的 metadata 行叠字**，被叠掉的正是更正那几行 | + +即：**更正在两条路上都没到达，而恰恰是我报告说已经做到的地方**——与本 PR 正在修的缺陷同形 +（一份文档没有说出关于它自己出处的该说的话）。 + +**根因是通用的，不是 jump 特有**：实测已发布的 **44/44** 份 eval JSON 的 `spec.description` +全部被截断（共 218 个 `[truncated]` 标记，其中三处还是每份 artifact 里的方法学说明）。 + +**改法（评估契约 v1.1）**：更正成为**结构化字段** `FactorSpec.corrections` +（`FactorCorrection` 元组：`from_version`/`to_version`/`date`/`defect`/`effect`/`superseded`， +每项非空校验、**超长 raise 而非裁剪**、`to_version` 必须等于 spec 自身 version 以便 +`spec.version` 单独就能判别一份存档在更正的哪一侧）。JSON 顶层新增 `corrections` 键 +（与 `eval_contract_version` 同级），经 `corrections_record` 走**封顶通道之外**导出；Markdown +provenance 行与 PNG 头行标记**从同一元组派生**（author once）。`description` 恢复为纯定义。 + +**守卫（`tests/test_factor_correction_carrier.py` 15 项）不做渲染文本子串匹配**（D5b 刚因此 +吃过亏）：把**声明的对象**经真实 `report_to_dict` + `json.dumps/loads` 往返后与原对象**相等 +比对**；并钉住「这些字段确实超过封顶长度」，所以「往返完好」只有在承载真的在封顶通道之外时 +才成立。 + +## 六之三、dashboard 叠字（既有缺陷，本 PR 顺带修 + 加守卫） + +FACTOR DEFINITION 带把描述锚在 `y=0.62`、metadata 锚在 `y=0.20/0.10`，**中间无约束** ⇒ 长 +描述直接画在 metadata 上。真实 dashboard 几何实测：**11 个分钟因子里 7 个重叠** +（`valley_price_quantile_20` 超 **296 px**，25 行对 4 行槽位）。 +`analytics/eval/figures.py` **在 git 里只有一个 commit** ⇒ **不是本 PR 的回归，而是每一份已 +产出 dashboard 都有的既有缺陷**（含冻结的 22 张）。 + +修法**不动任何因子的描述文本**（那才是爆炸半径）：`definition_description_lines` 限行到 +`DEFINITION_MAX_LINES=4`，超出部分以**显式省略标记**收尾并指向 Markdown（那一份 description +是完整的）。省略优于覆盖：覆盖是两段文字同时不可读**且不声明**。 + +守卫 `tests/test_definition_band_layout.py`（25 项）用**真实 dashboard 的 figsize/GridSpec** +渲染后取 Text 的 window extent 比 bbox —— **几何断言，不是看图，也不是子串匹配**。 + +## 六之四、D2 手算锚重跑：让冻结面板的陈旧性显形 + +`qt/hand_anchors_d2.py::hand_jump_amount_corr` 原本注释明写「引擎无 14:50 截断」并据此读 +整日 bar —— 一个**忠实复现缺陷**的手算参照。改成 `pit=True` 后重跑（`python -m +qt.hand_anchors_d2`，~7min）： + +``` +frozen 14: 70 rows, 5 mismatches (RC=1) +FAIL jump_amount_corr_20 warmup_end 2021-07-02 000537.SZ hand=0.4613901757 engine=0.4730559418 rel=2.47e-02 +FAIL jump_amount_corr_20 ex_date_window 2021-07-21 000050.SZ hand=0.7755175868 engine=0.7776570622 rel=2.75e-03 +FAIL jump_amount_corr_20 random 2024-05-30 002690.SZ hand=0.3831735019 engine=0.4011519443 rel=4.48e-02 +FAIL jump_amount_corr_20 random 2023-05-18 600867.SH hand=0.5843712903 engine=0.5831966539 rel=2.01e-03 +FAIL jump_amount_corr_20 random 2026-01-30 002773.SZ hand=0.6210838723 engine=0.6210838723 rel=1.06e-03 +``` + +**这 5 个 FAIL 是正确结果，不是回归**：手算侧已截断，而 `engine` 读的是**冻结的** +`artifacts/refactor_baseline/panels_d2/jump_amount_corr_20.parquet`（未截断口径，按约定不动）。 +其余 **63 行全部 OK、无一其他因子移动** —— 差异被精确隔离在这一个因子上。 + +重跑**前**盘上的状态是 5 行 hand==engine **逐位相同**：两侧都复现同一个缺陷，所以看起来干净。 +这正是「手算参照必须独立于引擎」的意义 —— 它一旦跟着引擎一起错，就不再是参照。 +旧文件已备份到 scratchpad（gitignored，非 git 记录）。 + ## 七、deny list 的解除（本 PR 内完成） D5b（PR #99）新增了一条**有意的响亮阻断**：`qt.exec_basis_eval.subject_view(factor_id)` diff --git a/factors/compute/minute/jump_amount_corr.py b/factors/compute/minute/jump_amount_corr.py index a59c00e..7881ace 100644 --- a/factors/compute/minute/jump_amount_corr.py +++ b/factors/compute/minute/jump_amount_corr.py @@ -53,7 +53,7 @@ empty_factor_series, visible_minute_frame, ) -from factors.spec import FactorSpec, PanelField +from factors.spec import FactorCorrection, FactorSpec, PanelField # Factor DEFINITION constants (reproduced from the report; NOT tuned knobs). The # daily value is a trailing-``JUMP_LOOKBACK_DAYS``-trading-day lagged correlation @@ -64,6 +64,38 @@ JUMP_MIN_PAIRS = 10 JUMP_Z = 1.0 +#: The intraday-cutoff correctness fix, declared as a STRUCTURED fact so every +#: artifact carries it verbatim (see :class:`factors.spec.FactorCorrection` for why +#: this cannot live in ``description``). This factor is the only one of the eleven +#: that shipped without a 14:50 truncation, so it is the only one with an entry. +_CUTOFF_CORRECTION = FactorCorrection( + from_version="1.0", + to_version="1.1", + date="2026-07-25", + defect=( + "v1.0 applied NO intraday PIT truncation at all — this factor was built " + "before the standing 14:50-truncation authorization and was the one minute " + "factor of eleven that never got it, so a value at date d pooled that day's " + "full session including the 14:50-15:00 bars." + ), + effect=( + "Under the 14:51-VWAP exec_to_exec basis the entry anchor for d is d's own " + "14:51 execution bar, so those values used POST-ENTRY information: the " + "execution minute itself and the nine minutes after it. Measured on the " + "shipped v1.0 engine, perturbing only the excluded bars moved every emitted " + "cell (1,477/1,477 and 1,373/1,373 in two independent samples, max |diff| " + "1.28 and 1.05); the other ten minute factors moved zero cells." + ), + superseded=( + "Every jump_amount_corr_20 evaluation artifact produced before this date is " + "superseded, on BOTH bases. The close_to_close copies are kept, labelled, " + "under artifacts/reports/pr_c_untruncated_close_to_close/; the exec copies " + "remain in the frozen exec baseline as the byte record of what was " + "published. Aggregate metrics moved little and the Watch verdict survived, " + "but that was established by re-running, not inferred from the similarity." + ), +) + def _minute_requires(*fields: str) -> tuple[PanelField, ...]: """The stk_mins_1min requires tuple of a minute-derived factor (D1).""" @@ -273,13 +305,14 @@ def spec(self) -> FactorSpec: f"traded amount at price-JUMP minutes (within-day amplitude z-score " f">1) and the amount at the strictly-next minute. Derived from 1min " f"bars but a DAILY signal; >= {JUMP_MIN_PAIRS} jump-pairs required " - f"else NaN. CORRECTION (v1.0 -> v1.1): values published before this " - f"fix were computed WITHOUT the 14:50 truncation, so a value at d " - f"included that day's 14:50-15:00 bars — i.e. POST-ENTRY information " - f"under the 14:51-VWAP exec_to_exec basis (the execution minute " - f"itself and the nine after it). Those values are superseded; this " - f"run is the truncated recomputation." + f"else NaN." ), + # The correction is a STRUCTURED fact, not a sentence appended here: + # the report JSON caps every payload string at 200 chars, so a + # correction living in ``description`` reaches the Markdown and the + # dashboard and is cut out of the machine-readable copy (measured: + # 44/44 shipped eval JSONs have a truncated description). + corrections=(_CUTOFF_CORRECTION,), expected_ic_sign=-1, is_intraday=False, forward_return_horizon=1, diff --git a/factors/spec.py b/factors/spec.py index 4dafd4d..21aa431 100644 --- a/factors/spec.py +++ b/factors/spec.py @@ -52,13 +52,19 @@ from __future__ import annotations +import re from collections.abc import Sequence from dataclasses import dataclass from enum import StrEnum +from typing import ClassVar from data.availability_policy import Adjustment, OvernightBoundary from factors.requires import PanelField +#: ISO date shape for a correction's ``date`` (a real calendar check is not the +#: point; a free-form date string in a machine record is). +_ISO_DATE = re.compile(r"\d{4}-\d{2}-\d{2}") + # The ONLY basis an intraday factor may declare: the minute tail model's holding # period runs execution-anchor to execution-anchor, exec(T) -> exec(T_next), and # is NEVER close-to-close (I5a). Cross-checked in _check_intraday_block. @@ -96,6 +102,87 @@ _PRICE_CHANNEL_FIELDS: frozenset[str] = frozenset({"open", "high", "low", "close"}) +#: Per-field character bound on a :class:`FactorCorrection`. Over-length RAISES at +#: construction rather than being trimmed: this field exists precisely BECAUSE the +#: report JSON's generic payload cap (``analytics.eval.render.MAX_VALUE_CHARS``) +#: silently trims long strings, so a carrier that could itself be trimmed would +#: rebuild the defect it was created to fix. Loud at authoring time > quiet in the +#: artifact. +CORRECTION_FIELD_MAX_CHARS = 2000 + + +@dataclass(frozen=True) +class FactorCorrection: + """A STRUCTURED statement that a factor's PUBLISHED values were superseded. + + Why this is a field and not a sentence in ``description``. The machine-readable + report record runs every string through ``analytics.eval.render.sanitize_payload``, + which caps a value at 200 characters and appends ``...[truncated]``. Measured on + the 44 shipped eval JSONs: **44/44** have a truncated ``spec.description`` (plus + three truncated methodological notes each, 218 markers in all). So a correction + written as prose inside ``description`` reaches the Markdown and the dashboard + but is CUT OUT of the JSON — and the JSON is the copy a summary layer reads. A + report that silently drops the disclosure of its own provenance is the same + shape of defect as a factor that reads bars it should not: a document not saying + the thing about itself that it must say. + + Every field is required and non-empty, so a half-declared correction is a + construction error rather than a hole in an artifact: + + Attributes: + from_version: the ``FactorSpec.version`` whose values are superseded. + to_version: the version that supersedes them; MUST equal the spec's own + ``version``, so an artifact's ``spec.version`` alone discriminates. + date: ISO date the correction landed (``YYYY-MM-DD``). + defect: what was wrong with the superseded values. + effect: what the defect did to those values (the reader's "so what"). + superseded: which published artifacts the statement retires, and where the + historical copies live. + """ + + from_version: str + to_version: str + date: str + defect: str + effect: str + superseded: str + + _FIELDS: ClassVar[tuple[str, ...]] = ( + "from_version", + "to_version", + "date", + "defect", + "effect", + "superseded", + ) + + def __post_init__(self) -> None: + for name in self._FIELDS: + value = getattr(self, name) + if not isinstance(value, str) or not value.strip(): + raise ValueError( + f"FactorCorrection.{name} must be a non-empty string: a " + f"half-declared correction would leave a hole in every " + f"artifact that carries it. Got {value!r}." + ) + if len(value) > CORRECTION_FIELD_MAX_CHARS: + raise ValueError( + f"FactorCorrection.{name} is {len(value)} chars, over the " + f"{CORRECTION_FIELD_MAX_CHARS} bound. This RAISES instead of " + f"trimming on purpose — a correction that can be silently cut " + f"short is the defect this field exists to fix. Shorten it." + ) + if not _ISO_DATE.fullmatch(self.date): + raise ValueError( + f"FactorCorrection.date must be an ISO YYYY-MM-DD date; got " + f"{self.date!r}." + ) + + def as_dict(self) -> dict[str, str]: + """Ordered plain-dict form for the machine-readable record.""" + return {name: getattr(self, name) for name in self._FIELDS} + + def _coerce_taxonomy( enum_cls: type[StrEnum], value: object, field_name: str, factor_id: str ) -> StrEnum: @@ -186,6 +273,12 @@ class FactorSpec: # actually consumed — see ``factors.store.incremental.factor_lookback_depth``. # Validated to a positive int WHEN present. lookback_depth: int | None = None + # Contract v1.1 ADDITIVE: structured statements that this factor's PUBLISHED + # values were superseded. Default () = "nothing was ever corrected"; it is NOT + # a claim that the factor is right, only that no correction has been declared. + # See :class:`FactorCorrection` for why this is a field rather than prose in + # ``description`` (the JSON caps every payload string at 200 chars). + corrections: tuple[FactorCorrection, ...] = () decision_cutoff: str | None = None data_lag: str | None = None session_open: str | None = None @@ -198,6 +291,7 @@ def __post_init__(self) -> None: self._check_measurement() self._check_inputs() self._check_declarations() + self._check_corrections() self._check_intraday_block() # -- validators (enforcement layer #1) -------------------------------- @@ -295,6 +389,41 @@ def _check_inputs(self) -> None: # immutable + hashable. object.__setattr__(self, "input_fields", normalized) + def _check_corrections(self) -> None: + """Corrections must be a tuple of FactorCorrection landing ON this version. + + ``to_version == self.version`` is enforced so ``spec.version`` alone is a + sufficient discriminator in a stored artifact: a reader holding one JSON + can tell whether it is the corrected side without having to find the + other one. A correction pointing at some other version would be a fact + about a document this one is not. + """ + corrections = self.corrections + if isinstance(corrections, FactorCorrection) or not isinstance( + corrections, Sequence + ): + raise ValueError( + f"FactorSpec.corrections must be a sequence of FactorCorrection " + f"(not a bare one); got {corrections!r} for {self.factor_id!r}." + ) + normalized = tuple(corrections) + bad = [c for c in normalized if not isinstance(c, FactorCorrection)] + if bad: + raise ValueError( + f"FactorSpec.corrections entries must be FactorCorrection " + f"instances (a structured fact, never free text); got {bad!r} " + f"for {self.factor_id!r}." + ) + mismatched = [c for c in normalized if c.to_version != self.version] + if mismatched: + raise ValueError( + f"FactorSpec.corrections entries must have to_version == the " + f"spec's own version {self.version!r} so spec.version alone " + f"discriminates a stored artifact; got " + f"{[c.to_version for c in mismatched]!r} for {self.factor_id!r}." + ) + object.__setattr__(self, "corrections", normalized) + def _check_declarations(self) -> None: """Contract v1.0: requires / adjustment / overnight_boundary are mandatory.""" requires = self.requires @@ -422,6 +551,8 @@ def _check_intraday_block(self) -> None: __all__ = [ + "CORRECTION_FIELD_MAX_CHARS", + "FactorCorrection", "FactorSpec", "PanelField", # re-export: the requires entry type (single class, factors.requires) "RETURN_BASES", diff --git a/tests/test_definition_band_layout.py b/tests/test_definition_band_layout.py new file mode 100644 index 0000000..dcf40ae --- /dev/null +++ b/tests/test_definition_band_layout.py @@ -0,0 +1,172 @@ +"""The FACTOR DEFINITION band must not draw its text on top of itself. + +WHAT THIS CATCHES, AND WHY A HUMAN LOOKING AT THE PNG DOES NOT COUNT +-------------------------------------------------------------------- +The band anchors its pieces at FIXED axes fractions: the description at y=0.62 +(va="top") and the metadata row at y=0.20/0.10 (va="bottom"). Nothing bounds the +description, so a long one simply keeps wrapping downward and is DRAWN OVER the +metadata — both texts become unreadable, and the artifact silently loses content +it appears to show. Measured with the real renderer before the fix, six of the +eleven shipped minute factors overlapped (``valley_price_quantile_20`` by 291 px +at 25 wrapped lines against the slot). + +That is a standing defect rather than a regression: ``analytics/eval/figures.py`` +has exactly one commit in its history, so every dashboard ever produced was drawn +by this code. + +The check is geometric, not visual: render the band on the REAL dashboard +geometry, ask matplotlib for each Text's window extent, and assert the boxes do +not intersect. A person squinting at a PNG is not a guard, and neither is a +substring match on anything — the failure mode is pixels landing on pixels. +""" + +from __future__ import annotations + +import matplotlib + +matplotlib.use("Agg") + +import matplotlib.pyplot as plt # noqa: E402 +import pytest # noqa: E402 +from matplotlib.gridspec import GridSpec # noqa: E402 + +from analytics.eval.figures import ( # noqa: E402 + DEFINITION_MAX_LINES, + DEFINITION_WRAP_WIDTH, + _definition_band, + definition_description_lines, +) +from factors.compute.minute import binding as binding_module # noqa: E402 +from factors.registry import build # noqa: E402 + +#: Every minute factor that reaches a dashboard. Derived from the binding tables +#: (plus the deferred one) rather than listed, so a new factor is covered without +#: anyone remembering to add it here. +MINUTE_FACTOR_IDS = tuple( + sorted( + {cls().name for cls in binding_module._MINUTE_STREAM_BINDINGS} + | {"valley_price_quantile_20"} + ) +) + + +class _Data: + """Minimal stand-in for DashboardData: the band reads only ``spec``.""" + + def __init__(self, spec) -> None: + self.spec = spec + + +def _band_boxes(spec): + """(description bbox, lowest metadata bbox) on the REAL dashboard geometry. + + The figsize and GridSpec are copied from ``figures._render`` on purpose: a + layout assertion measured on some other canvas size would prove nothing about + the artifact that actually ships. + """ + fig = plt.figure(figsize=(15, 21.5)) + gs = GridSpec( + 7, 3, figure=fig, + height_ratios=[0.30, 0.38, 0.80, 1.5, 1.1, 1.0, 1.0], + hspace=0.62, wspace=0.28, + left=0.06, right=0.945, top=0.975, bottom=0.04, + ) + ax = fig.add_subplot(gs[2, :]) + _definition_band(ax, _Data(spec)) + fig.canvas.draw() + renderer = fig.canvas.get_renderer() + + description = None + metadata = [] + for text in ax.texts: + box = text.get_window_extent(renderer) + if text.get_va() == "top" and "\n" in text.get_text() or ( + text.get_va() == "top" and text.get_text() == spec.description + ): + description = box + elif text.get_va() == "bottom": + metadata.append(box) + if description is None: # single-line description (no newline to key on) + candidates = [ + t for t in ax.texts + if t.get_va() == "top" and t.get_text() not in ("FACTOR DEFINITION",) + ] + description = max(candidates, key=lambda t: t.get_window_extent(renderer).width + ).get_window_extent(renderer) + top_metadata = max(metadata, key=lambda b: b.y1) + plt.close(fig) + return description, top_metadata + + +@pytest.mark.parametrize("factor_id", MINUTE_FACTOR_IDS) +def test_the_description_never_overlaps_the_metadata_row(factor_id): + """The load-bearing assertion: the two boxes must not intersect. + + ``DEFINITION_MAX_LINES`` is set to the MEASURED capacity (5), which leaves + only ~4 px of slack — so this test is what makes that number safe to use. A + font or matplotlib change that eats the slack fails HERE, loudly, instead of + silently overprinting two texts in every dashboard. + """ + spec = build(factor_id).spec + description, metadata = _band_boxes(spec) + gap = description.y0 - metadata.y1 + assert gap >= 0.0, ( + f"{factor_id}: the description block overruns the metadata row by " + f"{-gap:.1f} px — both texts are drawn on top of each other and the " + f"dashboard silently loses content it appears to show. If the layout " + f"itself changed, re-measure DEFINITION_MAX_LINES rather than relaxing " + f"this assertion." + ) + + +@pytest.mark.parametrize("factor_id", MINUTE_FACTOR_IDS) +def test_the_drawn_description_stays_within_the_line_budget(factor_id): + """The mechanism behind the geometric assertion, pinned separately. + + Two assertions rather than one because they fail for different reasons: this + one goes red if the bounding stops being applied, the geometric one goes red + if the budget itself becomes wrong (a font or layout change). + """ + lines = definition_description_lines(build(factor_id).spec) + assert len(lines) <= DEFINITION_MAX_LINES + + +def test_an_over_long_description_is_marked_not_silently_dropped(): + """Elision must announce itself and say where the full text is. + + Silently showing the first few lines would be the same class of defect as the + JSON's silent truncation: a document not saying what it left out. + """ + spec = build("valley_price_quantile_20").spec + lines = definition_description_lines(spec) + assert len(lines) == DEFINITION_MAX_LINES + assert "elided" in lines[-1] + assert "Markdown" in lines[-1] + # the elided count is real, not decorative + import textwrap + + full = textwrap.wrap(spec.description, width=DEFINITION_WRAP_WIDTH) + assert str(len(full) - (DEFINITION_MAX_LINES - 1)) in lines[-1] + + +def test_a_short_description_is_untouched(): + """A factor that fits must be rendered verbatim — no marker, no elision.""" + import textwrap + + spec = build("jump_amount_corr_20").spec + lines = definition_description_lines(spec) + assert lines == textwrap.wrap(spec.description, width=DEFINITION_WRAP_WIDTH) + assert "elided" not in " ".join(lines) + + +def test_the_correction_marker_shares_the_header_row_and_does_not_wrap(): + """The correction marker is a short header-row badge, not a wrapping block. + + If it ever grows into prose it would start colliding the same way the + description did, so its shape is pinned here rather than left to taste. + """ + from analytics.eval.figures import _correction_marker + + marker = _correction_marker(build("jump_amount_corr_20").spec) + assert marker and "\n" not in marker + assert len(marker) < 60 diff --git a/tests/test_eval_contract_v1.py b/tests/test_eval_contract_v1.py index 79ba864..72430cf 100644 --- a/tests/test_eval_contract_v1.py +++ b/tests/test_eval_contract_v1.py @@ -25,6 +25,7 @@ render_verdict_summary, require_basis_columns, ) +from analytics.eval import contract as contract_module from analytics.eval.render import _requires_row from data.availability_policy import ReturnBasis, View @@ -289,7 +290,14 @@ def test_every_verdict_threshold_default_is_the_frozen_close_era_value(): def test_contract_version_is_stated(): - assert EVAL_CONTRACT_VERSION == "1.0" + """Pinned so a bump has to be deliberate — and it has to bring its statement. + + v1.0 -> v1.1 added the top-level ``corrections`` key; the module docstring is + that statement, and this assertion is what forces the next bump to write one + too. The thresholds above are unchanged by that bump (asserted separately). + """ + assert EVAL_CONTRACT_VERSION == "1.1" + assert "CONTRACT v1.1" in contract_module.__doc__ # --------------------------------------------------------------------------- # diff --git a/tests/test_factor_correction_carrier.py b/tests/test_factor_correction_carrier.py new file mode 100644 index 0000000..ee05e62 --- /dev/null +++ b/tests/test_factor_correction_carrier.py @@ -0,0 +1,281 @@ +"""A correction must survive INTO the machine-readable record, whole. + +WHAT WENT WRONG, AND WHY A TEST HAD TO EXIST +-------------------------------------------- +The intraday-cutoff correctness fix first wrote its "these values supersede +previously published ones" disclosure as prose appended to +``FactorSpec.description``. The Markdown carried it. The dashboard carried it. +The JSON did not: ``analytics.eval.render.sanitize_payload`` caps every exported +string at ``MAX_VALUE_CHARS`` (200) and appends ``...[truncated]``, and the +disclosure sat past character 200. + +That cap is GENERIC, not a quirk of one path — measured across the 44 shipped +eval JSONs, 44/44 carry a truncated ``spec.description``. So the JSON, the copy a +summary layer reads, showed restated numbers with no indication that they +replaced a defective run. A report that fails to state the thing about its own +provenance that it must state is the same shape as the factor defect that +produced the correction: hence a guard, not a fixed sentence. + +WHY THESE ASSERTIONS AND NOT SUBSTRING MATCHES +----------------------------------------------- +D5b lost a guard to exactly that: a test that matched tokens in a function's +SOURCE stayed green when the tokens were moved into a comment. So nothing here +greps rendered text for a phrase. Instead each test round-trips the DECLARED +structured object through the real export path and asserts EQUALITY with what the +factor declared — a lexical rewording cannot pass, and a truncation cannot pass, +because the compared thing is the object itself. + +The length assertions are the teeth for the specific failure: the fields are +deliberately longer than the cap, so "it round-tripped" is only provable while +the carrier is genuinely outside the capped path. +""" + +from __future__ import annotations + +import json + +import pytest + +from analytics.eval.figures import _correction_marker +from analytics.eval.render import ( + MAX_VALUE_CHARS, + corrections_record, + report_to_dict, + sanitize_payload, +) +from factors.compute.minute.jump_amount_corr import JumpAmountCorrFactor +from factors.registry import build as build_factor +from factors.spec import ( + CORRECTION_FIELD_MAX_CHARS, + FactorCorrection, + FactorSpec, + PanelField, +) + +_TRUNCATION_MARKER = "[truncated]" + + +def _long(prefix: str) -> str: + """A field comfortably past the payload cap (so the cap would show if hit).""" + return prefix + " " + "x" * (MAX_VALUE_CHARS + 50) + + +def _correction(**overrides) -> FactorCorrection: + base = dict( + from_version="1.0", + to_version="1.1", + date="2026-07-25", + defect=_long("the defect was"), + effect=_long("the effect was"), + superseded=_long("superseded artifacts are"), + ) + base.update(overrides) + return FactorCorrection(**base) + + +# --------------------------------------------------------------------------- # +# FactorCorrection: a structured fact, validated at authoring time +# --------------------------------------------------------------------------- # +def test_every_field_is_required(): + for name in FactorCorrection._FIELDS: + with pytest.raises(ValueError, match=name): + _correction(**{name: " "}) + + +def test_over_length_raises_rather_than_trimming(): + """The carrier must not be able to lose what it carries. + + Trimming here would rebuild the very defect this field exists to fix, so the + bound is enforced LOUDLY at construction instead of quietly at export. + """ + with pytest.raises(ValueError, match="RAISES instead of trimming"): + _correction(defect="y" * (CORRECTION_FIELD_MAX_CHARS + 1)) + + +def test_date_must_be_an_iso_date(): + with pytest.raises(ValueError, match="ISO YYYY-MM-DD"): + _correction(date="July 2026") + + +def test_spec_rejects_a_correction_that_lands_on_another_version(): + """``spec.version`` alone must discriminate a stored artifact. + + A reader holding ONE json should be able to tell which side of the correction + it is on without going to find the other one. + """ + with pytest.raises(ValueError, match="to_version"): + _spec(version="2.0", corrections=(_correction(to_version="1.1"),)) + + +def test_spec_rejects_free_text_in_place_of_a_correction(): + with pytest.raises(ValueError, match="FactorCorrection"): + _spec(corrections=("we fixed the cutoff",)) + + +def test_spec_rejects_a_bare_correction_not_in_a_tuple(): + with pytest.raises(ValueError, match="not a bare one"): + _spec(corrections=_correction()) + + +def _spec(**overrides) -> FactorSpec: + base = dict( + factor_id="fixture_factor", + version="1.1", + description="fixture", + expected_ic_sign=1, + is_intraday=False, + forward_return_horizon=1, + return_basis="close_to_close", + input_fields=("close",), + requires=(PanelField("close", source="market_daily"),), + adjustment="returns_invariant", + overnight_boundary="none", + ) + base.update(overrides) + return FactorSpec(**base) + + +# --------------------------------------------------------------------------- # +# The capped path is real (so the tests below are not shadow-boxing) +# --------------------------------------------------------------------------- # +def test_the_generic_payload_cap_would_have_eaten_this(): + """Pins the mechanism the carrier routes around. + + If this ever stops truncating, the correction field is no longer load-bearing + for THIS reason and the docstring above needs rewriting rather than the code. + """ + long_value = "z" * (MAX_VALUE_CHARS + 500) + got = sanitize_payload({"note": long_value})["note"] + assert got.endswith(_TRUNCATION_MARKER) + # the surviving payload is exactly the cap; the marker rides on top of it + assert got[: -len(_TRUNCATION_MARKER)].rstrip(".") == "z" * MAX_VALUE_CHARS + + +def test_a_correction_written_into_the_description_would_be_truncated(): + """The exact mistake this file exists to prevent, demonstrated. + + Shows the failure mode directly rather than describing it: prose past the cap + inside ``description`` does not survive ``vars(spec)`` export. + """ + spec = _spec(description="definition. " + _long("CORRECTION:")) + exported = sanitize_payload(vars(spec))["description"] + assert exported.endswith(_TRUNCATION_MARKER) + assert "CORRECTION" in spec.description # it IS in the object ... + assert exported.count("x") < spec.description.count("x") # ... and cut on export + + +# --------------------------------------------------------------------------- # +# Round-trip: declared object == what a machine consumer reads back +# --------------------------------------------------------------------------- # +def test_corrections_record_round_trips_the_declared_object_exactly(): + correction = _correction() + got = corrections_record(_spec(corrections=(correction,))) + assert got == [correction.as_dict()] + assert all(_TRUNCATION_MARKER not in v for v in got[0].values()) + + +def test_a_factor_with_no_corrections_exports_an_empty_list_not_a_blank_entry(): + """"No correction declared" and "known correct" are different statements.""" + assert corrections_record(_spec()) == [] + assert _correction_marker(_spec()) == "" + + +def _report_for(spec): + """A real, verdict-bearing report carrying ``spec`` — the actual export path. + + Reuses the contract suite's section fixtures rather than hand-rolling eight + mandatory sections here; the subject under test is the export, not assembly. + """ + from analytics.eval.report import FactorEvalReport + from tests.test_factor_eval_contract import _adopt_grade_sections + from tests.test_factor_eval_contract import _cfg as _contract_cfg + + return FactorEvalReport.assemble( + spec, _contract_cfg(), _adopt_grade_sections() + ).with_verdict() + + +def test_the_json_export_carries_the_declaration_at_top_level_untruncated(): + """The end-to-end property: parse the exported JSON, compare to the object. + + Uses the REAL ``report_to_dict`` + ``json.dumps``/``loads`` round trip on a + real report, so a change that reroutes corrections back through the capped + path — or drops the key — fails here. + """ + correction = _correction() + spec = _spec(corrections=(correction,)) + doc = json.loads(json.dumps(report_to_dict(_report_for(spec)))) + + assert doc["corrections"] == [correction.as_dict()] + for entry in doc["corrections"]: + for value in entry.values(): + assert _TRUNCATION_MARKER not in value + # ... and the teeth: the fields really are past the cap, so surviving whole + # is only possible outside the capped path. + assert max(len(v) for v in correction.as_dict().values()) > MAX_VALUE_CHARS + # The same report's spec block IS still capped — the cap was routed around, + # not removed (removing it would let a stray panel repr into the artifact). + assert doc["spec"]["description"].endswith(_TRUNCATION_MARKER) or len( + spec.description + ) <= MAX_VALUE_CHARS + + +def test_the_shipped_factors_declaration_survives_the_real_export(): + """The SHIPPED spec (not a fixture) round-trips through the real exporter.""" + spec = JumpAmountCorrFactor().spec + declared = [c.as_dict() for c in spec.corrections] + assert declared, "the shipped factor declares no correction — test is vacuous" + doc = json.loads(json.dumps(report_to_dict(_report_for(spec)))) + assert doc["corrections"] == declared + assert max(len(v) for v in declared[0].values()) > MAX_VALUE_CHARS + + +# --------------------------------------------------------------------------- # +# The shipped factor declares it, and the registry-built one agrees +# --------------------------------------------------------------------------- # +def test_the_corrected_factor_declares_its_correction_through_the_registry(): + """The path the runners actually use must carry it too. + + A declaration only reachable by importing the class directly would miss every + artifact, since the runners build through the registry. + """ + spec = build_factor("jump_amount_corr_20").spec + assert spec.version == "1.1" + assert len(spec.corrections) == 1 + correction = spec.corrections[0] + assert (correction.from_version, correction.to_version) == ("1.0", "1.1") + assert _correction_marker(spec) == "CORRECTED v1.0 -> v1.1" + + +def test_report_to_dict_emits_the_key_even_when_nothing_was_corrected(): + """Present on EVERY report, empty or not — an absent key is ambiguous. + + A consumer must be able to ask "was anything corrected?" without having to + distinguish "no corrections" from "an older contract that had no such key". + """ + doc = report_to_dict(_report_for(_spec())) + assert doc["corrections"] == [] + + +def test_the_markdown_and_the_json_cannot_disagree(): + """Both surfaces are built from the SAME tuple, so they move together. + + Asserted as a relationship between the two renderings of one object, not as + a fixed phrase in either: a reworded label stays green, a correction present + in one surface and missing from the other does not. + """ + correction = _correction() + report = _report_for(_spec(corrections=(correction,))) + doc = report_to_dict(report) + markdown = report.render() + + assert len(doc["corrections"]) == 1 + # every field of the declared correction appears whole in the rendered + # provenance box (the row composes them; it never re-words them) + for name in ("defect", "effect", "superseded"): + assert getattr(correction, name) in markdown + assert _TRUNCATION_MARKER not in markdown + + clean = _report_for(_spec()) + assert report_to_dict(clean)["corrections"] == [] + assert "CORRECTION" not in clean.render() diff --git a/tests/test_report_json_truncation_census.py b/tests/test_report_json_truncation_census.py new file mode 100644 index 0000000..75cce85 --- /dev/null +++ b/tests/test_report_json_truncation_census.py @@ -0,0 +1,188 @@ +"""Which exported fields lose text to the 200-char cap — asserted, not assumed. + +WHY A CENSUS AND NOT A SINGLE-FIELD ASSERTION +---------------------------------------------- +The intraday-cutoff correction was first written into ``FactorSpec.description`` +and was cut out of the JSON by ``sanitize_payload``'s ``MAX_VALUE_CHARS`` cap. The +obvious guard — "assert the correction survives" — would have confirmed only the +thing already known. This repo has the lesson written down from #82: *a guard that +only scans the file you already fixed can only confirm what you already know*, and +widening that scan immediately produced six more instances. + +So this file asserts the WHOLE truncated-field set of the exported record, not one +field. Anything new joining the set fails here; anything leaving it fails here too +(a fix must update the record, which is the point). + +WHAT IT FOUND +------------- +Six fields are truncated across the shipped corpus, and none of them is noise — +every one is an explanatory caveat cut mid-sentence: + +* ``spec.description`` — 44/44 artifacts; +* ``section[return_risk].payload.monotonicity_spearman_by_date_ci_note`` — 44/44; +* ``section[oos_generalization].payload.monotonicity_reversed_status`` — 44/44, + cut at "This evaluator scores ONE cell and cannot ...[truncated]", i.e. the half + that states what the evaluator CANNOT do is the half that is gone; +* ``section[caveats].payload.multiple_testing_note`` — 44/44; +* ``section[purity].payload.vif_status`` — 22/44; +* ``section[data_coverage].payload.forward_return_source`` — 20/44. + +This predates the correctness fix (the frozen #79 baseline has it too), so it is +recorded here rather than fixed here — see the PR notes for the cost comparison. +``corrections`` is the one field this branch is responsible for, and it must NEVER +appear in the set. + +WHAT IT ALSO RULED OUT +---------------------- +Not every long string is at risk. ``verdict.reasons``, the per-axis reasons and the +section ``note`` fields go through ``sanitize_text`` ONLY — no cap — as proved on +disk by a 401-char reason and a 1263-char note surviving whole. A "reasons are 16 +characters from the cap" alarm is therefore a false one: that path has no cap to +hit. The genuinely close call is ``coverage_bias_bad_vwap`` at 158 of 200. +""" + +from __future__ import annotations + +import json +import pathlib + +import pytest + +from analytics.eval.render import MAX_VALUE_CHARS + +MARKER = "...[truncated]" + +#: The truncated-field set of the shipped corpus, RECORDED so a seventh cannot +#: join quietly and a fix cannot land without updating this. Field names are the +#: normalized paths produced by :func:`_leaves`. +KNOWN_TRUNCATED_FIELDS: frozenset[str] = frozenset( + { + "spec.description", + "section[return_risk].payload.monotonicity_spearman_by_date_ci_note", + "section[oos_generalization].payload.monotonicity_reversed_status", + "section[caveats].payload.multiple_testing_note", + "section[purity].payload.vif_status", + "section[data_coverage].payload.forward_return_source", + } +) + +#: Fields this branch is responsible for. A correction that cannot survive export +#: is the exact defect this PR exists to fix, so it is called out separately from +#: the inherited set rather than folded into it. +MUST_NEVER_TRUNCATE: frozenset[str] = frozenset({"corrections"}) + +_REPORTS = pathlib.Path("artifacts/reports") + + +def _leaves(obj, path: str = ""): + """Every leaf string with a stable, index-free path. + + Section payloads are keyed by section NAME rather than list position, so the + recorded set does not shift when the canonical section order changes. + """ + if isinstance(obj, dict): + name = obj.get("name") + for key, value in obj.items(): + child = f"{path}.{key}" if path else str(key) + if name and key == "payload": + child = f"section[{name}].payload" + yield from _leaves(value, child) + elif isinstance(obj, list): + for value in obj: + yield from _leaves(value, path) + elif isinstance(obj, str): + yield path, obj + + +def _shipped_reports() -> list[pathlib.Path]: + return sorted(_REPORTS.glob("eval_*.json")) if _REPORTS.is_dir() else [] + + +def _census(paths) -> dict[str, int]: + counts: dict[str, int] = {} + for path in paths: + doc = json.loads(path.read_text(encoding="utf-8")) + hit = {field for field, value in _leaves(doc) if value.endswith(MARKER)} + for field in hit: + counts[field] = counts.get(field, 0) + 1 + return counts + + +requires_corpus = pytest.mark.skipif( + not _shipped_reports(), + reason="no eval_*.json on disk (artifacts/ is gitignored and run-generated)", +) + + +@requires_corpus +def test_the_truncated_field_set_is_exactly_the_recorded_one(): + """Reads the JSON FROM DISK — the exported bytes, not a rebuilt object. + + Asserting against a freshly constructed report would test the constructor and + could pass while the written file was wrong; the failure being guarded is + "what a machine consumer opens is missing text", which only the file can show. + """ + census = _census(_shipped_reports()) + got = frozenset(census) + unexpected = got - KNOWN_TRUNCATED_FIELDS + assert not unexpected, ( + f"NEW truncated field(s) in the exported record: {sorted(unexpected)} " + f"(counts {[(f, census[f]) for f in sorted(unexpected)]}). Something " + f"load-bearing is being cut at {MAX_VALUE_CHARS} chars." + ) + fixed = KNOWN_TRUNCATED_FIELDS - got + assert not fixed, ( + f"field(s) no longer truncated: {sorted(fixed)} — good, but update " + f"KNOWN_TRUNCATED_FIELDS so the record keeps matching reality." + ) + + +@requires_corpus +def test_the_correction_carrier_is_never_truncated_on_disk(): + """The one field this branch owns. Separate test, separate reason to fail.""" + census = _census(_shipped_reports()) + offending = MUST_NEVER_TRUNCATE & frozenset(census) + assert not offending, ( + f"{sorted(offending)} was truncated in the exported record — the " + f"superseded-values disclosure is exactly what must not be cut." + ) + + +@requires_corpus +def test_a_corrected_factors_artifact_carries_the_whole_declaration_on_disk(): + """End-to-end on the shipped bytes: parse the file, compare to the spec. + + The comparison is against the DECLARED object, so a reworded field passes and + a shortened one does not — the property is "whole", not "contains a phrase". + """ + from factors.registry import build + + declared = [c.as_dict() for c in build("jump_amount_corr_20").spec.corrections] + assert declared, "the factor declares no correction — this test is vacuous" + + checked = 0 + for path in _shipped_reports(): + if "jump_amount_corr" not in path.name: + continue + doc = json.loads(path.read_text(encoding="utf-8")) + assert doc["corrections"] == declared, f"{path.name} lost the declaration" + checked += 1 + assert checked >= 4, f"expected the 4 jump artifacts, checked {checked}" + + +@requires_corpus +def test_the_uncapped_paths_really_are_uncapped(): + """Rules out the false alarm: reasons/notes are not near any cap. + + ``verdict.reasons`` and section ``note`` go through ``sanitize_text`` only. + Proved on the corpus rather than by reading the code, because "which strings + are capped" is precisely what was wrong the first time. + """ + longest: dict[str, int] = {} + for path in _shipped_reports(): + doc = json.loads(path.read_text(encoding="utf-8")) + for field, value in _leaves(doc): + if not value.endswith(MARKER): + longest[field] = max(longest.get(field, 0), len(value)) + assert longest.get("verdict.reasons", 0) > MAX_VALUE_CHARS + assert longest.get("sections.note", 0) > MAX_VALUE_CHARS From 01f9714c4c3ad9d66cd33e63daf0a706d01d20bc Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Mon, 27 Jul 2026 01:56:53 -0700 Subject: [PATCH 6/7] fix(eval): size the definition band for the intraday branch the dashboard actually renders DEFINITION_MAX_LINES=5 was measured on the daily branch (metadata row at axes y=0.10), but the exec-basis dashboard of every shipped minute factor renders intraday_spec_variant (is_intraday=True), whose metadata row sits at 0.20 with the minute block below it. Measured on the real geometry (figsize 15x21.5, the _render GridSpec), the intraday branch holds only 4 wrapped lines (+1.5 px of slack; 5 lines overprints by -13.5 px), so 9 of the 11 exec-form dashboards drew the description over the metadata. Split the constant into DEFINITION_MAX_LINES_DAILY=5 / DEFINITION_MAX_LINES_INTRADAY=4, keyed on spec.is_intraday via definition_max_lines(), and extend the layout guard to BOTH forms: the geometric no-overlap assertion and the line-budget assertion now run against the close spec and the exec spec derived by the same intraday_spec_variant the exec path uses (ExecBasisParams sourced from IntradayCfg defaults so the test tracks project conventions). A boundary test pins the split in both directions on volume_peak_count_20, which wraps to exactly 5 lines: verbatim in the close form, elided to 4 with a marked tail in the exec form. Mutation evidence: setting the intraday budget back to 5 turns 11 tests red; reverting restores green. Also reconcile the numbers the docstrings stated (capacity 5 / 'seven that genuinely overflow' vs '4-line slot' / 'Six of the eleven' / '291 px'): both counts were right for different branches, which was the bug. Post-fix state, all measured: capacity 5 daily / 4 intraday, 7 of 11 minute factors overflow the daily budget and 9 of 11 the intraday one, and valley_price_quantile_20 unbounded overruns the metadata row by 296 px (daily) / 314 px (intraday). --- analytics/eval/figures.py | 85 ++++++++++---- tests/test_definition_band_layout.py | 166 +++++++++++++++++++++------ 2 files changed, 194 insertions(+), 57 deletions(-) diff --git a/analytics/eval/figures.py b/analytics/eval/figures.py index 09d6c42..20517a8 100644 --- a/analytics/eval/figures.py +++ b/analytics/eval/figures.py @@ -223,46 +223,81 @@ def _definition_block_line(spec: FactorSpec) -> str | None: #: How many wrapped lines the band's description slot can hold before the text #: collides with the metadata row below it. NOT a style preference — the slot is -#: bounded by fixed axes-fraction anchors (description at 0.62 va="top", metadata -#: at 0.20/0.10 va="bottom"), so a longer description silently DRAWS OVER the -#: metadata. +#: bounded by fixed axes-fraction anchors (description at 0.62 va="top"), and the +#: metadata row's anchor DEPENDS ON THE SPEC FORM: 0.10 for a daily spec, 0.20 +#: for an intraday one (which also draws the minute block at 0.04). A longer +#: description silently DRAWS OVER the metadata. +#: +#: The two forms therefore have DIFFERENT capacities, and sizing by the daily +#: branch alone is exactly the defect this split fixes: ``build(factor_id).spec`` +#: is daily (``is_intraday=False``) for every shipped minute factor, but the +#: exec-basis dashboard renders ``intraday_spec_variant`` (``is_intraday=True``), +#: so a single 5 measured on the daily branch overprints 9 of the 11 exec-form +#: dashboards. #: #: MEASURED, not chosen, on the real dashboard geometry (figsize 15x21.5, the -#: GridSpec of ``_render``): the gap between the two boxes falls 15 px per line and -#: goes negative at SIX lines — 1..5 lines give +63.8/+48.8/+33.8/+18.8/+3.8 px, -#: 6 gives -11.2. Five is therefore the capacity, and it is deliberately not -#: reduced for comfort: every factor whose description fits today keeps rendering -#: in full, and only the seven that genuinely overflow get an elision marker. +#: GridSpec of ``_render``); the gap between the two boxes falls 15 px per line: +#: +#: * daily branch (metadata at 0.10): 4 lines +18.8 px, 5 lines +3.8 px, +#: 6 lines -11.2 px -> capacity 5; +#: * intraday branch (metadata at 0.20): 4 lines +1.5 px, 5 lines -13.5 px +#: -> capacity 4. #: -#: The slack at five lines is thin (~4 px, a quarter of a line). That is a fact -#: about the layout, not a hidden risk: ``tests/test_definition_band_layout.py`` -#: re-measures every shipped spec against the real geometry, so a font or -#: matplotlib change that eats the slack turns the guard RED instead of silently -#: overprinting. The number is checked, never trusted. -DEFINITION_MAX_LINES = 5 +#: Neither number is reduced for comfort: every factor whose description fits its +#: own branch keeps rendering in full, and only the ones that genuinely overflow +#: (7 of the 11 minute factors at 5 lines daily, 9 of 11 at 4 lines intraday) +#: get an elision marker. +#: +#: The slack at capacity is thin (+3.8 / +1.5 px, about a quarter of a line). +#: That is a fact about the layout, not a hidden risk: +#: ``tests/test_definition_band_layout.py`` re-measures every shipped spec in +#: BOTH forms against the real geometry, so a font or matplotlib change that +#: eats the slack turns the guard RED instead of silently overprinting. The +#: numbers are checked, never trusted. +DEFINITION_MAX_LINES_DAILY = 5 +DEFINITION_MAX_LINES_INTRADAY = 4 + + +def definition_max_lines(spec) -> int: + """The description-line budget for THIS spec's form — daily 5, intraday 4. + + Keyed on ``spec.is_intraday`` because that is what moves the metadata row + (see :data:`DEFINITION_MAX_LINES_DAILY`). A spec carries its own form, so + the budget cannot be measured on one branch and spent on the other. + """ + if spec.is_intraday: + return DEFINITION_MAX_LINES_INTRADAY + return DEFINITION_MAX_LINES_DAILY def definition_description_lines(spec) -> list[str]: """The description lines the band may draw, bounded so it cannot overlap. - Six of the eleven shipped minute factors have descriptions long enough to - collide with the metadata row (measured with the real renderer: - ``valley_price_quantile_20`` overlaps by 291 px, and it is 25 wrapped lines - against a 4-line slot). The renderer has ONE commit in its history, so those - dashboards have always been drawn that way — this is a standing defect, not a - regression, and "both texts unreadable" is the worst of the options. + The bound is per spec form (:func:`definition_max_lines`): the slot holds + 5 lines in the daily branch and 4 in the intraday one, so the same + description can fit one dashboard and need elision on the other. Before + any bound existed, nine of the eleven shipped minute factors had + descriptions long enough to collide with the metadata row (measured with + the real renderer: ``valley_price_quantile_20`` is 25 wrapped lines and, + unbounded, overruns the metadata row by 296 px in the daily branch / 314 + px in the intraday one). The renderer had exactly one commit in its + history when the bound was added, so every dashboard produced before it + was drawn that way — this is a standing defect, not a regression, and + "both texts unreadable" is the worst of the options. Bounding it visibly is strictly better than overdrawing: an elided tail is marked and points at the copy that carries the description in full (the Markdown provenance box — the JSON's is capped at - ``analytics.eval.render.MAX_VALUE_CHARS``). Nothing is dropped without saying - so, which is the whole difference between this and what it replaces. + ``analytics.eval.render.MAX_VALUE_CHARS``). Nothing is dropped without + saying so, which is the whole difference between this and what it + replaces. """ + cap = definition_max_lines(spec) lines = textwrap.wrap(spec.description, width=DEFINITION_WRAP_WIDTH) - if len(lines) <= DEFINITION_MAX_LINES: + if len(lines) <= cap: return lines - elided = len(lines) - (DEFINITION_MAX_LINES - 1) - return lines[: DEFINITION_MAX_LINES - 1] + [ + elided = len(lines) - (cap - 1) + return lines[: cap - 1] + [ f"... [{elided} more lines elided to fit — FULL definition in the " f"Markdown report's provenance box]" ] diff --git a/tests/test_definition_band_layout.py b/tests/test_definition_band_layout.py index dcf40ae..d8b81e4 100644 --- a/tests/test_definition_band_layout.py +++ b/tests/test_definition_band_layout.py @@ -3,21 +3,34 @@ WHAT THIS CATCHES, AND WHY A HUMAN LOOKING AT THE PNG DOES NOT COUNT -------------------------------------------------------------------- The band anchors its pieces at FIXED axes fractions: the description at y=0.62 -(va="top") and the metadata row at y=0.20/0.10 (va="bottom"). Nothing bounds the -description, so a long one simply keeps wrapping downward and is DRAWN OVER the -metadata — both texts become unreadable, and the artifact silently loses content -it appears to show. Measured with the real renderer before the fix, six of the -eleven shipped minute factors overlapped (``valley_price_quantile_20`` by 291 px +(va="top") and the metadata row at y=0.20/0.10 (va="bottom") — 0.20 when the +spec is intraday (the minute block then also draws at 0.04), 0.10 otherwise. +Nothing bounds the description, so a long one simply keeps wrapping downward +and is DRAWN OVER the metadata — both texts become unreadable, and the +artifact silently loses content it appears to show. Measured with the real +renderer before the bound existed, nine of the eleven shipped minute factors +overlapped (``valley_price_quantile_20`` by 296 px daily / 314 px intraday, at 25 wrapped lines against the slot). -That is a standing defect rather than a regression: ``analytics/eval/figures.py`` -has exactly one commit in its history, so every dashboard ever produced was drawn -by this code. +WHY BOTH FORMS ARE MEASURED, NOT ONE +------------------------------------ +The two spec forms have DIFFERENT capacities (5 lines daily, 4 intraday — +the metadata row sits 0.10 higher in the intraday branch). The dashboard a +minute factor's exec-basis report actually ships renders the +``intraday_spec_variant`` (``is_intraday=True``), while ``build(factor_id).spec`` +is the daily form — so a guard that only measures the close form protects the +half that was never broken and misses the 9-of-11 exec-form overlap. Every +assertion in this file therefore runs against BOTH the close spec and the +exec spec derived by the same ``intraday_spec_variant`` the exec path uses. + +That the underlying defect is standing rather than a regression: +``analytics/eval/figures.py`` had exactly one commit in its history when the +bound was added, so every dashboard ever produced was drawn by that code. The check is geometric, not visual: render the band on the REAL dashboard -geometry, ask matplotlib for each Text's window extent, and assert the boxes do -not intersect. A person squinting at a PNG is not a guard, and neither is a -substring match on anything — the failure mode is pixels landing on pixels. +geometry, ask matplotlib for each Text's window extent, and assert the boxes +do not intersect. A person squinting at a PNG is not a guard, and neither is +a substring match on anything — the failure mode is pixels landing on pixels. """ from __future__ import annotations @@ -31,13 +44,20 @@ from matplotlib.gridspec import GridSpec # noqa: E402 from analytics.eval.figures import ( # noqa: E402 - DEFINITION_MAX_LINES, + DEFINITION_MAX_LINES_DAILY, + DEFINITION_MAX_LINES_INTRADAY, DEFINITION_WRAP_WIDTH, _definition_band, definition_description_lines, + definition_max_lines, ) from factors.compute.minute import binding as binding_module # noqa: E402 from factors.registry import build # noqa: E402 +from qt.config import IntradayCfg # noqa: E402 +from qt.exec_forward_returns import ( # noqa: E402 + ExecBasisParams, + intraday_spec_variant, +) #: Every minute factor that reaches a dashboard. Derived from the binding tables #: (plus the deferred one) rather than listed, so a new factor is covered without @@ -49,6 +69,38 @@ ) ) +#: The two dashboard forms a minute factor ships in: the close-basis one +#: (daily spec) and the exec-basis one (``intraday_spec_variant``). +FORMS = ("close", "exec") + + +def _exec_params() -> ExecBasisParams: + """The exec block, sourced from the SAME defaults the exec path falls back to. + + Built field-by-field from ``IntradayCfg()`` exactly as + ``ExecBasisParams.from_config`` does for a config with no intraday block, + so a changed project default moves this test instead of silently making it + measure a geometry the shipped dashboards no longer use. + """ + ic = IntradayCfg() + return ExecBasisParams( + decision_cutoff=ic.decision_time, + data_lag=ic.data_lag, + session_open=ic.session_open, + execution_model=ic.execution_model, + execution_window=(ic.execution_window[0], ic.execution_window[1]), + execution_price_basis=ic.execution_price_basis, + source="test: qt.config.IntradayCfg defaults", + ) + + +def _spec_for(factor_id: str, form: str): + """The spec the dashboard of the given form actually renders.""" + spec = build(factor_id).spec + if form == "close": + return spec + return intraday_spec_variant(spec, _exec_params()) + class _Data: """Minimal stand-in for DashboardData: the band reads only ``spec``.""" @@ -99,61 +151,111 @@ def _band_boxes(spec): @pytest.mark.parametrize("factor_id", MINUTE_FACTOR_IDS) -def test_the_description_never_overlaps_the_metadata_row(factor_id): +@pytest.mark.parametrize("form", FORMS) +def test_the_description_never_overlaps_the_metadata_row(factor_id, form): """The load-bearing assertion: the two boxes must not intersect. - ``DEFINITION_MAX_LINES`` is set to the MEASURED capacity (5), which leaves - only ~4 px of slack — so this test is what makes that number safe to use. A - font or matplotlib change that eats the slack fails HERE, loudly, instead of - silently overprinting two texts in every dashboard. + The capacities are set to the MEASURED per-branch maxima (5 daily / 4 + intraday), which leave only +3.8 / +1.5 px of slack — so this test is what + makes those numbers safe to use. A font or matplotlib change that eats the + slack fails HERE, loudly, instead of silently overprinting two texts in + every dashboard. """ - spec = build(factor_id).spec + spec = _spec_for(factor_id, form) description, metadata = _band_boxes(spec) gap = description.y0 - metadata.y1 assert gap >= 0.0, ( - f"{factor_id}: the description block overruns the metadata row by " - f"{-gap:.1f} px — both texts are drawn on top of each other and the " - f"dashboard silently loses content it appears to show. If the layout " - f"itself changed, re-measure DEFINITION_MAX_LINES rather than relaxing " - f"this assertion." + f"{factor_id} ({form} form): the description block overruns the " + f"metadata row by {-gap:.1f} px — both texts are drawn on top of each " + f"other and the dashboard silently loses content it appears to show. " + f"If the layout itself changed, re-measure DEFINITION_MAX_LINES_DAILY " + f"/ DEFINITION_MAX_LINES_INTRADAY rather than relaxing this assertion." ) @pytest.mark.parametrize("factor_id", MINUTE_FACTOR_IDS) -def test_the_drawn_description_stays_within_the_line_budget(factor_id): +@pytest.mark.parametrize("form", FORMS) +def test_the_drawn_description_stays_within_the_line_budget(factor_id, form): """The mechanism behind the geometric assertion, pinned separately. Two assertions rather than one because they fail for different reasons: this one goes red if the bounding stops being applied, the geometric one goes red if the budget itself becomes wrong (a font or layout change). """ - lines = definition_description_lines(build(factor_id).spec) - assert len(lines) <= DEFINITION_MAX_LINES + spec = _spec_for(factor_id, form) + lines = definition_description_lines(spec) + assert len(lines) <= definition_max_lines(spec) + + +def test_the_budget_is_keyed_on_the_spec_form(): + """5 daily / 4 intraday — the split this file exists to keep honest. + + Pinned as numbers, not just as "different": the measured capacities are + what the geometric test re-checks, and a silent re-unification of the two + constants would re-open the exec-form overlap. + """ + assert DEFINITION_MAX_LINES_DAILY == 5 + assert DEFINITION_MAX_LINES_INTRADAY == 4 + spec = build("volume_peak_count_20").spec + assert definition_max_lines(spec) == DEFINITION_MAX_LINES_DAILY + assert definition_max_lines(_spec_for("volume_peak_count_20", "exec")) == ( + DEFINITION_MAX_LINES_INTRADAY + ) + + +def test_a_five_line_description_fits_close_but_is_elided_in_the_exec_form(): + """The teeth of the per-form split, in BOTH directions. + + ``volume_peak_count_20`` wraps to exactly 5 lines: the close form (budget + 5) must render it VERBATIM, the exec form (budget 4) must elide it to + 4 lines with a marked tail. If the budget were measured on the daily + branch only — the defect being guarded — the first assertion would pass + and the second would be the 9-of-11 overlap all over again. + """ + import textwrap + + close_spec = _spec_for("volume_peak_count_20", "close") + full = textwrap.wrap(close_spec.description, width=DEFINITION_WRAP_WIDTH) + assert len(full) == DEFINITION_MAX_LINES_DAILY, ( + "the fixture factor no longer wraps to exactly 5 lines — pick another " + "one for this boundary test rather than weakening it" + ) + + close_lines = definition_description_lines(close_spec) + assert close_lines == full # verbatim, no marker + + exec_lines = definition_description_lines(_spec_for("volume_peak_count_20", "exec")) + assert len(exec_lines) == DEFINITION_MAX_LINES_INTRADAY + assert "elided" in exec_lines[-1] + assert str(len(full) - (DEFINITION_MAX_LINES_INTRADAY - 1)) in exec_lines[-1] -def test_an_over_long_description_is_marked_not_silently_dropped(): +@pytest.mark.parametrize("form", FORMS) +def test_an_over_long_description_is_marked_not_silently_dropped(form): """Elision must announce itself and say where the full text is. Silently showing the first few lines would be the same class of defect as the JSON's silent truncation: a document not saying what it left out. """ - spec = build("valley_price_quantile_20").spec + spec = _spec_for("valley_price_quantile_20", form) + cap = definition_max_lines(spec) lines = definition_description_lines(spec) - assert len(lines) == DEFINITION_MAX_LINES + assert len(lines) == cap assert "elided" in lines[-1] assert "Markdown" in lines[-1] # the elided count is real, not decorative import textwrap full = textwrap.wrap(spec.description, width=DEFINITION_WRAP_WIDTH) - assert str(len(full) - (DEFINITION_MAX_LINES - 1)) in lines[-1] + assert str(len(full) - (cap - 1)) in lines[-1] -def test_a_short_description_is_untouched(): +@pytest.mark.parametrize("form", FORMS) +def test_a_short_description_is_untouched(form): """A factor that fits must be rendered verbatim — no marker, no elision.""" import textwrap - spec = build("jump_amount_corr_20").spec + spec = _spec_for("jump_amount_corr_20", form) lines = definition_description_lines(spec) assert lines == textwrap.wrap(spec.description, width=DEFINITION_WRAP_WIDTH) assert "elided" not in " ".join(lines) From 497e50da1d1b71fc12ee24cb6653d2d9e9e9447a Mon Sep 17 00:00:00 2001 From: shaofl <2899218482@qq.com> Date: Mon, 27 Jul 2026 01:57:17 -0700 Subject: [PATCH 7/7] test(eval): make the correction-carrier census able to fail (prefix match) test_the_correction_carrier_is_never_truncated_on_disk could never fire: _leaves() emits no container path for a list, so the census keys for the corrections block are corrections.date / corrections.superseded / ..., and intersecting them with the bare {'corrections'} set was empty by construction. Mutation proof: truncating corrections[0].superseded in an on-disk artifact left this test green while its two sibling tests went red. Match by prefix instead: a census field is offending when it equals 'corrections' or starts with 'corrections.'. Mutation evidence on a tmp copy of the real 44-file corpus (real artifacts untouched): intact corpus -> PASS; corrections[0].superseded truncated -> FAIL naming ['corrections.superseded']; the OLD exact-set logic on the same doctored corpus -> still PASS, replicating the blind guard. All four census tests also re-run green against the intact real corpus. --- tests/test_report_json_truncation_census.py | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/tests/test_report_json_truncation_census.py b/tests/test_report_json_truncation_census.py index 75cce85..340e31b 100644 --- a/tests/test_report_json_truncation_census.py +++ b/tests/test_report_json_truncation_census.py @@ -69,7 +69,15 @@ #: Fields this branch is responsible for. A correction that cannot survive export #: is the exact defect this PR exists to fix, so it is called out separately from #: the inherited set rather than folded into it. -MUST_NEVER_TRUNCATE: frozenset[str] = frozenset({"corrections"}) +#: +#: PREFIXES, not leaf paths: ``_leaves`` does not emit a container path for a +#: list, so the census keys for the corrections block are ``corrections.date`` / +#: ``corrections.superseded`` / ... and a bare ``"corrections"`` set entry would +#: intersect the census EMPTY FOREVER — the guard could never fire (proved by +#: mutation: truncating ``corrections[0].superseded`` on disk left the exact-set +#: version of this test green). A census field counts as offending when it +#: equals a prefix or starts with ``prefix + "."``. +MUST_NEVER_TRUNCATE_PREFIXES: tuple[str, ...] = ("corrections",) _REPORTS = pathlib.Path("artifacts/reports") @@ -141,9 +149,16 @@ def test_the_truncated_field_set_is_exactly_the_recorded_one(): def test_the_correction_carrier_is_never_truncated_on_disk(): """The one field this branch owns. Separate test, separate reason to fail.""" census = _census(_shipped_reports()) - offending = MUST_NEVER_TRUNCATE & frozenset(census) + offending = sorted( + field + for field in census + if any( + field == prefix or field.startswith(prefix + ".") + for prefix in MUST_NEVER_TRUNCATE_PREFIXES + ) + ) assert not offending, ( - f"{sorted(offending)} was truncated in the exported record — the " + f"{offending} was truncated in the exported record — the " f"superseded-values disclosure is exactly what must not be cut." )