feat(eval): unified exec-only FactorEvalRunner via factors.service (D5 C4)

config/factor_eval_csi500.yaml

@@ -0,0 +1,110 @@
+# D5 C4 — UNIFIED exec-only factor evaluation runner (run-factor-eval).
+#
+# ONE config for all eleven minute-derived factors: the SUBJECT factor comes
+# from the CLI (--factor <registry id>, e.g. jump_amount_corr_20), not from
+# this file. The evaluation is EXEC-ONLY: the factor's decision-view (14:50)
+# values are scored on the 14:51-VWAP exec_to_exec forward return, through the
+# factor SERVICE (store read-through + the single materializer engine); the
+# reports are written as factor_eval_{factor_id}_exec_{no,with}_book.* (a stem
+# the legacy eval_{name}_* artifacts never collide with).
+#
+# CACHE-ONLY: the minute read is provably live-call-free (the minute store has
+# no fetch closure); daily / universe / covariate endpoints go through the
+# read-through cache (warm -> 0 gap fetches).
+#
+# Universe = CSI500 (000905.SH), PIT membership; window = 2021-07-01 ..
+# 2026-06-30; daily rebalance (contract default); fee 0.001; quantiles 5;
+# SW-L1 industry + size neutralization — the same cell every legacy eval
+# config declared (catalogue section 4: the 11 configs differed in exactly two
+# keys, project.name and data.output_name).
+
+project:
+  name: quantitative_trading_factor_eval
+  timezone: Asia/Shanghai
+data:
+  source: tushare
+  freq: D
+  start: '2021-07-01'
+  end: '2026-06-30'
+  external_secret_file: /home/shaofl/Projects/financial_projects/.config.json
+  tushare_token_key: tushare.token
+  output_name: factor_eval_csi500
+  cache:
+    enabled: true
+    root_dir: artifacts/cache/tushare/v1
+    refresh_recent_days: 14
+    refresh_dimension_days: 30
+    force_refresh: []
+universe:
+  type: index
+  index_code: 000905.SH
+  symbols: []
+  min_listing_days: 60
+  filters:
+    missing_close: true
+    suspended: false
+    st: false
+    limit_up_down: false
+# factors = EMPTY ON PURPOSE (catalogue BUG 5): the subject factor comes from
+# --factor and the evaluated BOOK is fixed inside the runner (value_ep /
+# value_bp / volatility_20, lead ruling Q1). A non-empty block must match that
+# effective book exactly or the runner raises — this file must never silently
+# describe a factor set the run does not use.
+factors: []
+processing:
+  drop_missing: true
+  standardize:
+    enabled: true
+    method: zscore
+  # winsorize is a P0 no-op in this codebase; the EvalConfig declares
+  # winsorize=None accordingly (nothing is clipped), so the report never
+  # overstates preprocessing.
+  winsorize:
+    enabled: false
+    method: mad
+    n: 3.0
+  # Industry + market-cap neutralization (SW-L1), matching the report's
+  # neutral column and the EvalConfig neutralization declaration.
+  neutralize:
+    enabled: true
+    industry_col: industry
+    size_col: market_cap
+    industry_level: L1
+alpha:
+  model: equal_weight
+  params: {}
+portfolio:
+  # Required by the schema but unused: this runner evaluates a factor, it does
+  # not build/execute a portfolio.
+  constructor: topn_equal_weight
+  top_n: 50
+  long_only: true
+  max_weight: null
+  turnover_cap: null
+backtest:
+  # Required by the schema but unused (no backtest is run). The EvalConfig
+  # uses a DAILY rebalance (contract default), independent of this field.
+  initial_nav: 1.0
+  rebalance: monthly
+  event_order: close_to_next_period
+  cash_return: 0.0
+cost:
+  fee_rate: 0.001
+  slippage_rate: 0.0
+  turnover_formula: l1
+analytics:
+  forward_return_periods:
+  - 1
+  quantiles: 5
+  benchmark: null
+# OOS split (window midpoint) so the OOS section runs and the Predictive axis
+# can be assessed (sign consistency across both holdout subperiods).
+oos:
+  split_date: '2024-01-01'
+output:
+  root_dir: artifacts
+  data_dir: artifacts/data
+  factor_dir: artifacts/factors
+  report_dir: artifacts/reports
+  log_dir: artifacts/logs
+  overwrite: true

docs/factors/d5_runner_difference_catalogue.md

@@ -317,3 +317,64 @@ unknown-never-convicts / N_eff CI / exploratory 封顶 Watch 的全部规则。
 ⚠️ 因此 C5 对账在**逐字节**层面对 22 份 md/json 一定不成立，能成立的是 **IC / ICIR /
 分位价差 / verdict 的值级对账**加上「JSON 差异恰为上表三项 + §七 的 spec 四键」这个
 **结构性**断言。把「逐字节相同」当作 C5 的通过条件会让人在这里误判为回归。
+
+### 七之三、C4 首轮对账（3 因子 × 3 模式）暴露的判据缺口 —— 补登记（2026-07-27）
+
+C4 harness（`qt/factor_eval_reconcile.py`）首轮对账 minute_ideal_amp_10 / jump_amount_corr_20 /
+volume_peak_count_20 三模式全红。独立调查（证据脚本 `/tmp/q1_dump.py` / `/tmp/q1_twogeom.py` /
+`/tmp/q1b_tail.py`，首轮日志 `/tmp/c4_reconcile_run.log`）把**全部**差异归到两个根因，均为
+**加载几何/浮点层面、非取值 bug**。本节把判据补全为具名类，每条都有边界与反例约束
+（类外差异仍非零退出——未编目的差异算失败）。
+
+**认知修正（本节最重要的一条）**：C4 handoff §3 ① 曾预测「bounded 因子 vs D1 baseline 预期
+**零差异**，因为旧 runner 也逐票读全窗口」。**这个预测错了**：旧 runner 的「全窗口」左端同样
+锚在 `data.start`（2021-07-01）——即 **anchor 截断**；新 materializer 的饱和加载左延到分钟缓存
+真实起点（2015-01-05）。首轮「预期全绿却没绿」的源头是这个预测，**不是引擎 bug**。
+
+**根因一：warmup 左延漂移（设计内）**，具名类 `warmup_left_extension`，bounded 与 pooled 通用，
+覆盖**全部三个方向**（frozen-NaN→new-finite、finite→finite 即 partial pool→full pool、
+new-only finite 行——旧 runner 网格根本没发射的那些行）：
+
+| 形态 | 边界（实测精确） | 首轮实测 |
+|---|---|---|
+| bounded（minute_ideal_amp_10 / jump_amount_corr_20） | 冻结网格**前 w−1 个交易日**（w=该因子 `lookback_depth`；第 w 日已完全 warmup）。边界取**网格左端**而非 per-symbol 首日：旧 runner 对每个 symbol 的加载都锚在 `data.start`，只有评估窗左端欠 warmup；晚上市/停牌票的首日两种几何看到同一批 bar，不可能有差异 | NaN→finite 3644 + 180；finite→finite 4554 + 17109 |
+| pooled（valid-day 池化，volume_peak_count_20） | 早区窗 [2021-07-01, 2021-10-31]，且**按月计数非递增**（违反即失败） | NaN→finite 8189；finite→finite 17193（07-01..08-24 按月衰减）；new-only finite 9059（07-01..07-14，旧网格未发射） |
+
+anchors 腿同判据：hand 侧按旧几何算、service 侧按新几何算，失败行全在 warmup 区。原 harness
+只给 pooled 备了早区类、bounded 没有 → **判据不对称**（minute_ideal_amp 与 jump 的首轮 anchors
+失败即此）。bounded 的 warmup 边界与 panels 腿相同（冻结网格前 w−1 个交易日）；jump 的
+**非** warmup 行仍必须 reconcile（它证明 service 携带截断定义，首轮 3 个 random 行 rel ~1e-15
+已绿），warmup 区外的 jump mismatch 仍 FAIL。
+
+**根因二：两个浮点/阈值尾部（非 bug）**，各带**双边界**（不调大全局容差，保住对真回归的牙）：
+
+| 类 | 机制 | 边界 | 实测 |
+|---|---|---|---|
+| `float_reordering_tail` | rolling 相关求和顺序差异（JC1 已裁 1e-12 为可归因浮点重排地板，这是地板之上的实测尾部） | rel ≤ **5e-12** 且 cell 数 ≤ **101**（超任一即失败） | jump 101 cell，5 票散在，rel 1.0e-12–2.9e-12 |
+| `threshold_flip_tail` | rolling σ 浮点噪声（~4e-10）× 整数成交量恰压阈值 → 计数翻转 | 幅度**恰 ±1 count**、rel ≤ 1e-2、cell 数 ≤ **25**（超任一即失败） | volume_peak 20 cell，600623.SH 2023-06-15..07-14 每天恰 −1（整数成交量 13300.0） |
+
+**reports 腿两处口径修正 + 三个 JSON 登记项**：
+
+1. **MD 按 key 配对成行级 change 再分类**。裸行集合差把每个值变化报成一对（一条 removal +
+   一条同 `- key:` 头的 addition）——首轮 83–96 条「removal」全是这种幻影。配对后跟随与 JSON
+   相同的数值归因梯。配对变化的分类：**数字承载**（任一侧含数字，散文行的数字可能在冒号
+   **之前**）才进 `warmup_aggregate_effect`；纯标签翻转（verdict PASS→FAIL 不含数字）仍
+   unregistered 失败。**配对是两遍的**：先按精确 `- key:` 头配对，再对剩余行按**数字归一化**
+   的头配对（实测必要：incremental 轴理由行的变化数字在首个冒号之前，精确头永远配不上；
+   且该行头长 ~103 字符，配对 key 上限须 ≥200——改了散文**词**而非数字的行归一化后仍配不上，
+   照失败，有反向测试锁定）。
+2. `warmup_aggregate_effect`（JSON 与 MD 共用）：聚合指标叶子（`sections[*]` /
+   `verdict.reasons[*]` / `verdict.axes.*`）的数值/散文变化是 panels 腿已登记 warmup 差异的
+   **下游**（聚合一个 warmup cell 变了的面板；panels 腿才是值级闸门）。**牙仍在**：聚合路径
+   之外（verdict 标签、spec、eval_config、criteria）任何变化仍失败；无数字的标签翻转仍失败。
+3. `spec.requires[0..n]` 由精确匹配改**前缀**匹配（§七 已登记 spec 16→20 键，`requires` 是
+   repr 字符串**列表**，展平后是索引叶子）。
+4. `registered_sanity_stem_rename`：`sections[5].payload.sanity_report`（及 MD 同名行）的值从
+   `eval_<name>_exec_basis_sanity.md` 改为 `factor_eval_<factor_id>_exec_basis_sanity.md`——
+   runner 有意的防碰撞改名，双侧都以 `_exec_basis_sanity.md` 收尾才认。
+5. `registered_run_order_artifact`：`sections[5].payload.exec_price_artifact_reused`
+   False→True——同会话 run-order 产物（首个 run 建 exec 价格 artifact，后续 run 复用）。
+   **反向（True→False）不登记，仍失败**。
+6. jump 的 `spec.description` 与 `sections[7].payload.factor_version`（连同原已特判的
+   `spec.version`）纳入 `registered_correction_effect`——仅当新 JSON 带 `corrections`
+   结构化更正承载（契约 v1.1）时成立。

factors/materialize.py

@@ -829,6 +829,15 @@ def make_recompute_fn(
     (``emit_start is None`` -> from ``data_start``). The same trailing-trim floor
     as ``materialize_range`` keeps the incremental overlap bit-identical to a full
     column (the D3 batch=incremental invariant).
+
+    CROSS-SECTIONAL FACTORS (``stores_intermediate``) are NOT WIRED here: this
+    fn returns the VALUE Series, while the store persists the per-symbol
+    intermediate for them, so ``tail_recompute`` refuses their payload up front
+    with ``factors.store.incremental.IntermediatePayloadNotWiredError`` — a
+    deliberate D5 C4 deferral, not an oversight. Wiring it (an
+    intermediate-returning recompute fn, footprint semantics in the overlap
+    validation, and a post-combine value-level check) is its own engine work,
+    tracked for the 21:00-update story.
     """
 
     def _recompute(emit_start, end, warmup) -> pd.Series:

factors/store/__init__.py

@@ -35,6 +35,7 @@
 from factors.store.incremental import (
     CacheHorizonConfig,
     IncrementalResult,
+    IntermediatePayloadNotWiredError,
     endpoint_horizons,
     factor_lookback_depth,
     overlap_window,
@@ -56,6 +57,7 @@
     "CacheHorizonConfig",
     "FactorValueStore",
     "IncrementalResult",
+    "IntermediatePayloadNotWiredError",
     "RunRecord",
     "RunRegistry",
     "StoreKey",

factors/store/incremental.py

@@ -29,7 +29,14 @@
 re-computed (not locally patched), classified by the factor's ``adjustment``:
 ``price_level`` mismatches are EXPECTED (an ex-date re-based the level — the
 key/fingerprint invalidation path); ``returns_invariant`` / ``none`` mismatches
-mean upstream data actually changed and are recorded LOUDLY.
+mean upstream data actually changed and are recorded LOUDLY. A stored NaN cell
+recomputing to finite is NOT a mismatch: it is a footprint fill (the D4c fill
+recorded the cell as covered-but-empty and the cache later warmed), counted in
+the result notes and absorbed, never ``revision_detected``.
+
+A CROSS-SECTIONAL factor's stored payload (its per-symbol intermediate, D4c) is
+refused up front with :class:`IntermediatePayloadNotWiredError`: the incremental
+path is deliberately not wired for that shape (D5 C4 deferral).
 
 Layering: ``factors.store`` never imports ``qt`` (red line #10); this uses stdlib
 + numpy/pandas + the availability-policy leaf + the sibling store modules.
@@ -48,14 +55,30 @@
 from data.clean.schema import DATE_LEVEL, SYMBOL_LEVEL
 from factors.base import Factor
 from factors.store.keys import StoreKey
-from factors.store.values import FactorValueStore
+from factors.store.values import FactorValueStore, payload_columns
 
 #: recompute(emit_start, end, warmup) -> Series over [emit_start, end], having
 #: loaded ``warmup`` trading days of input history before emit_start. emit_start
 #: is None to compute from the very beginning (a cold store / full column).
 RecomputeFn = Callable[[pd.Timestamp | None, pd.Timestamp, int], pd.Series]
 
 
+class IntermediatePayloadNotWiredError(ValueError):
+    """``tail_recompute`` reached a CROSS-SECTIONAL factor's stored payload.
+
+    Such a factor stores its universe-independent per-symbol INTERMEDIATE
+    (several columns), not its value (D4c). The incremental path is NOT WIRED
+    for that shape: the recompute callable returns a value Series, the overlap
+    validation is value-level, and the footprint semantics of an intermediate
+    payload are unhandled. That is a DELIBERATE deferral (D5 C4 — the eval
+    runner reaches these factors through the service's read-through, which
+    never calls tail_recompute), tracked for the 21:00-update story; wiring it
+    (an intermediate-returning recompute fn, footprint-aware overlap
+    validation, post-combine value validation) is its own engine work. See
+    ``factors.materialize.make_recompute_fn``.
+    """
+
+
 @dataclass(frozen=True)
 class CacheHorizonConfig:
     """The LIVE cache-config values the revision horizon is derived from (R5).
@@ -132,8 +155,28 @@ def _symbols(series: pd.Series) -> pd.Index:
     return series.index.get_level_values(SYMBOL_LEVEL)
 
 
-def _mismatched_symbols(stored_overlap: pd.Series, new_overlap: pd.Series) -> list[str]:
-    """Symbols whose overlap values differ bit-for-bit (NaN-aware) between the two."""
+def _classify_overlap_differences(
+    stored_overlap: pd.Series, new_overlap: pd.Series
+) -> tuple[list[str], list[str]]:
+    """Split NaN-aware overlap differences into REVISIONS vs footprint FILLS.
+
+    Returns ``(revised_symbols, filled_symbols)``:
+
+    * REVISED — a stored FINITE value recomputed to a different value or to
+      NaN. The store asserted a value and the recompute contradicts it: that is
+      an upstream data revision (loud for ``returns_invariant``/``none``,
+      expected for ``price_level``).
+    * FILLED — a stored NaN recomputed to a FINITE value. A stored NaN asserts
+      NO value, so it cannot be contradicted: D4c writes explicit NaN
+      "footprint" rows for the cells a fill covered but the factor produced
+      nothing for (``factors.service._record_fill_footprint``), and those cells
+      legitimately turn finite once the upstream cache warms. Reading that as a
+      revision misdiagnoses the store's own bookkeeping as upstream data
+      churn, so it is classified separately (counted in the result notes,
+      never ``revision_detected``) and the recomputed value simply replaces
+      the NaN. An honestly under-warmed stored NaN (near the data start)
+      filling later is the same shape and gets the same treatment.
+    """
     old = stored_overlap.sort_index(kind="mergesort")
     new = new_overlap.reindex(old.index)
     ov = old.to_numpy(dtype=float)
@@ -142,9 +185,12 @@ def _mismatched_symbols(stored_overlap: pd.Series, new_overlap: pd.Series) -> li
     both_nan = np.isnan(ov) & np.isnan(nv)
     differs = (~(ov == nv)) & ~both_nan
     if not differs.any():
-        return []
-    syms = old.index.get_level_values(SYMBOL_LEVEL)[differs]
-    return sorted(set(map(str, syms)))
+        return [], []
+    old_nan = np.isnan(ov)
+    syms = old.index.get_level_values(SYMBOL_LEVEL)
+    revised = sorted(set(map(str, syms[differs & ~old_nan])))
+    filled = sorted(set(map(str, syms[differs & old_nan])))
+    return revised, filled
 
 
 def tail_recompute(
@@ -165,7 +211,19 @@ def tail_recompute(
     (classified by ``adjustment``), and append the new tail.
     """
     w_dep = factor_lookback_depth(factor)
-    stored = store.read(key)
+    stored_frame = store.read_frame(key)
+    if stored_frame is not None and not stored_frame.empty:
+        columns = payload_columns(stored_frame)
+        if columns != (key.factor_id,):
+            raise IntermediatePayloadNotWiredError(
+                f"{key.factor_id}: the stored payload carries {list(columns)}, not the "
+                f"single value column {key.factor_id!r} — a cross-sectional factor "
+                f"stores its per-symbol INTERMEDIATE, and the incremental path is NOT "
+                f"WIRED for it (deliberate deferral, D5 C4; tracked for the "
+                f"21:00-update story). The eval runner reaches this factor through the "
+                f"service's read-through, which never calls tail_recompute."
+            )
+    stored = None if stored_frame is None else stored_frame[key.factor_id].rename(key.factor_id)
     if stored is None or stored.empty:
         full = recompute(None, today, w_dep)
         store.write(key, full, fingerprint=fingerprint)
@@ -213,12 +271,23 @@ def tail_recompute(
     tail = recomputed[rec_dates > last]
 
     stored_overlap = stored[_dates(stored) >= emit_start]
-    mismatched = _mismatched_symbols(stored_overlap, new_overlap)
+    mismatched, filled = _classify_overlap_differences(stored_overlap, new_overlap)
 
     is_price_level = factor.spec.adjustment is Adjustment.PRICE_LEVEL
     notes: list[str] = []
     recolumned: tuple[str, ...] = ()
 
+    if filled:
+        # NOT a revision: a stored NaN asserts no value (D4c fill footprint or an
+        # honestly under-warmed row), so it cannot be contradicted. The combined
+        # assembly below already replaces those cells with the recomputed values.
+        notes.append(
+            f"filled_after_footprint: {len(filled)} symbol(s) had stored NaN "
+            f"cells (fill footprint / under-warm) recompute to finite values — "
+            f"absorbing the recomputed values; this is NOT an upstream data "
+            f"revision."
+        )
+
     # assemble: untouched history + recomputed overlap/tail for normal symbols.
     head = stored[_dates(stored) < emit_start]
     combined = pd.concat([head, recomputed])
@@ -266,6 +335,7 @@ def tail_recompute(
 __all__ = [
     "CacheHorizonConfig",
     "IncrementalResult",
+    "IntermediatePayloadNotWiredError",
     "RecomputeFn",
     "endpoint_horizons",
     "factor_lookback_depth",