Skip to content

feat(eval): unified exec-only FactorEvalRunner via factors.service (D5 C4) - #105

Merged
StackOverFlow11 merged 4 commits into
mainfrom
feat/factor-refactor-d5-c4-unified-runner
Jul 28, 2026
Merged

feat(eval): unified exec-only FactorEvalRunner via factors.service (D5 C4)#105
StackOverFlow11 merged 4 commits into
mainfrom
feat/factor-refactor-d5-c4-unified-runner

Conversation

@StackOverFlow11

Copy link
Copy Markdown
Owner

D5 C4 — unified exec-only FactorEvalRunner (factor data access now goes through factors.service)

Design tmp/design/factor_refactor_design_v3.md step D5 C4. The eleven per-factor eval runners keep living side-by-side (deletion is C6, gated on the C5 four-leg audit); this PR adds the unified runner and its reconciliation harness, and proves it on three factors against the frozen baselines.

What changed (4 commits)

  1. qt/factor_eval_providers.py — real-cache providers: CacheMinuteProvider (promoted from factor_hotpath_smoke.py, single source, all three importers repointed), DailyEvalPanelProvider (close-view, un-lagged — the materializer applies daily_decision_lag itself), EvalServiceBundle + build_eval_service reusing qt/pipeline in the old runners' call order. Plus factors/store/incremental.py: stored-NaN→fresh-finite is now classified filled_after_footprint (was misdiagnosed as "upstream data REVISION" — the D4c footprint rows), and multi-column intermediate payloads raise the named IntermediatePayloadNotWiredError (deliberate deferral, cross-referenced in make_recompute_fn).
  2. qt/factor_eval_runner.py + qt/factor_eval_disclosures.py — the unified runner: service.panel(..., view=DECISION, basis=EXEC_TO_EXEC) replaces all eleven _load_*_panel paths; heterogeneous coverage disclosures moved verbatim (8/8 VERBATIM, programmatically checked) into one home with old runners re-exporting; book_mode = decision (book via service, closing the close(d)-book live defect) or close (faithful legacy); add-Section assembly keeps mandatory sections and verdict bit-identical; new artifact stem factor_eval_* (never collides with legacy eval_*); catalogue BUG 5 (unread factors: config block → readable error) / BUG 6 (name-clash tests) / BUG 3 (PeakCoverage zero tests) closed. New CLI run-factor-eval + config/factor_eval_csi500.yaml.
  3. qt/factor_eval_reconcile.py — the C5 four-leg audit harness: panels / reports / anchors modes, entry hard-gate requiring the frozen baseline 77/77 (unreachable baseline = hard error, never a skip), jump routed to the pr_c_cutoff_fix reference.
  4. Harness predicates completed after the first real reconciliation failed (the harness doing its job): independent root-cause investigation found the differences are the warmup left-extension (old runners anchored at data.start with no extension, so frozen panels are under-warmed for the first w−1 trading days; the materializer left-extends toward the true cache start 2015-01-05) plus two float tails. New named, machine-checkable classes: warmup_left_extension (bounded: first w−1 days, all directions; pooled: early window + monthly non-increasing), float_reordering_tail (rel ≤ 5e-12, ≤101 cells), threshold_flip_tail (±1 count, rolling-σ noise × integer volume threshold), MD key-pairing, JSON registrations. Catalogue docs/factors/d5_runner_difference_catalogue.md gained §七之三, including a cognition correction: the handoff §3① prediction "bounded factors expect zero diff vs the D1 baseline" was wrong — the old runners' "full window" was itself anchor-truncated; that wrong prediction, not an engine bug, was the source of the first-round red.

Real-run reconciliation (cache-only, stk_mins_live_calls=0 everywhere)

Three factors × three modes, 9/9 green, zero unclassified diffs:

factor panels reports anchors
minute_ideal_amp_10 warmup=8,198; unclassified=0 4/4 OK ok=3 warmup=2 failed=0
jump_amount_corr_20 warmup=17,289; float_tail=101; unclassified=0 4/4 OK ok=3 warmup=2 failed=0
volume_peak_count_20 warmup=34,441 (monthly 19,977→14,464 monotone ✓); threshold_flip=20; unclassified=0 4/4 OK ok=3 warmup=2 failed=0

jump's 3 non-warmup anchor rows went green (rel ~1e-15) — the #103 cutoff fix confirmed carried by the new path.

Review

Adversarial review (separate executor): APPROVE-WITH-NITS — 0 CRIT/HIGH/MEDIUM, with 64 self-built mutation probes against the classifier classes (out-of-window diffs, over-cap tails, non-monotone monthly counts, label-only flips, reverse run-order artifact all verified to FAIL), verbatim-move rechecked 8/8, hard gate tamper-tested on a /tmp manifest copy, and an independent rerun of the minute_ideal_amp_10 panels mode matching the implementer's counts cell-for-cell. Registered non-blocking items:

  • LOW-1: bounded warmup_left_extension has no magnitude/cell-count ceiling — an in-window error of arbitrary size is absorbed (~1.6% of the grid for w=20, conservative direction; pooled side has the monthly-monotonicity constraint). C5 follow-up: add a cell-count ceiling.
  • LOW-2: the reports-leg digit rule is old∨new — a digit-bearing verdict.reasons line replaced by a digit-free different-meaning line still registers as warmup_aggregate_effect. Verdict labels/axes/spec/eval_config remain strict. Acceptable for C5's purpose; recorded.
  • NIT-1: duplicate (date,symbol) keys crash with a raw pandas error instead of ReconciliationError (decorative; service panels cannot contain dup keys). NIT-2: the exec_price_artifact_reused reverse direction (True→False) has no pinning test (behavior verified FAIL by the reviewer's own mutation).

Gates (independently rerun by the lead)

  • pytest 2492 passed (node-ID set diff: 0 pre-existing tests displaced), ruff clean
  • run-phase0 demo anchor ic 0.9600 / annual 0.8408 (unchanged)
  • validate-config 32/32 (new factor_eval_csi500.yaml covered by the config glob)
  • secret scan: 2 benign hits — the external_secret_file path and tushare_token_key name in the new config, byte-identical in form to the 11 existing eval configs on main; no secret value anywhere
  • exec_baseline_freeze --verify 77/77 @ 45c14aa; frozen panels/baselines untouched (read-only throughout)

Deliberately NOT in this PR

Old-runner deletion (C6), full 11-factor four-leg audit (C5), valley_price_quantile binding + folding binding.py into code_hash (PR-C4b, lead-adjudicated), mmp_ew (D6c), run-registry integration (D7), tail_recompute wiring for intermediate payloads (explicit named refusal, 21:00-update story).

One runner body for the eleven minute-derived factors, replacing the
eleven near-identical eval_*.py main flows per the D5 difference
catalogue (section 4 = fixed body; sections 2/3 = four extension points):

* qt/factor_eval_disclosures.py (new): the single home of the four
  heterogeneous coverage disclosures (RidgeCoverage / RidgeReturnCoverage
  / PeakCoverage / NeutralizationCoverage + summarizers), moved VERBATIM
  from the four legacy runners (verified block-for-block against HEAD),
  which now re-export them. Two documented additions: a render() on
  NeutralizationCoverage (catalogue one-site normalization; the CLI's
  inline f-string now goes through it, byte-identical format pinned) and
  to_section() packing a coverage into an add-Section (contract 3.6:
  verdict-lazy by construction, pinned by test). Catalogue BUG 3 closed:
  summarize_peak_coverage gets the counterfactual + empty-frames tests
  its siblings had.
* qt/factor_eval_runner.py (new): subject values from factors.service.panel
  (decision view x exec_to_exec, store read-through + D4c read-assembly
  combine), processing via the shared _process_factors, exec evaluation
  via run_exec_basis_evaluation with an additive default-off
  extra_sections seam. Dual book_mode: 'close' replicates the legacy book
  path; 'decision' (default) takes the book through the same service
  panel (design 1.1 close-view book defect closed on this path). The
  EvalConfig declares the exec identity explicitly (17 shared kwargs
  unchanged). Catalogue BUG 5 closed: a config factors: block that
  disagrees with the fixed book is a readable error. Artifact stem
  factor_eval_{factor_id} (+ _bookclose suffix in close mode) never
  collides with the legacy eval_{name} artifacts. RunRegistry append is
  a deliberate deferral to D7. valley_price_quantile is a readable
  deferral (PR-C4b).
* CLI: one new subcommand run-factor-eval --config X --factor F
  [--book-mode decision|close]; the eleven legacy _cmd_run_eval_* are
  untouched (C6 convergence).
* config/factor_eval_csi500.yaml: the 11-config shared block with
  factors: [] (BUG 5 honest declaration).

Gates: pytest 2431 passed / 5 skipped (node-ID diff +29, 0 existing
squeezed), ruff clean, validate-config 32/32, phase0 demo anchor
ic 0.9600 / annual 0.8408 unchanged.
…at tails as named classes (D5 C4)

First full reconciliation (3 factors x 3 modes) failed every leg. The
independent investigation attributed ALL differences to two non-bug root
causes; this change completes the judging criteria so each is a NAMED,
machine-checkable, bounded class — anything outside still fails (an
unregistered difference is a failure):

* warmup_left_extension (panels + anchors): the old runners anchored every
  symbol's load at data.start (2021-07-01), so the frozen grid's first w-1
  trading dates are under-warmed; the materializer saturates to the minute
  cache's real start (2015-01-05). Bounded: the grid's first
  lookback_depth-1 trading dates, all three directions (NaN->finite,
  finite->finite as partial pools fill, new-only finite rows). Pooled: the
  early region [2021-07-01, 2021-10-31] with non-increasing per-month
  counts. This also corrects the attribution text: the effect is the
  left-extension warmup, not the D4 union-trim repair. Anchors: bounded
  factors get the same boundary (the harness only had the pooled early
  region before — the asymmetry that failed minute_ideal_amp and jump);
  jump's NON-warmup rows must still reconcile (they prove the truncated
  definition is carried).
* float_reordering_tail: scattered finite-vs-finite cells with rel <= 5e-12
  AND <= 101 cells (the measured jump tail; rolling-correlation summation
  order). The global tolerance is NOT widened.
* threshold_flip_tail: count factors — rolling-sigma float noise x an
  integer volume on the peak threshold flips the count by EXACTLY +/-1
  (rel <= 1e-2, <= 25 cells; measured 20 cells, 600623.SH).
* reports leg: Markdown is now paired into row-level changes by the
  '- key:' head (two passes: exact head, then digit-normalized head for
  prose lines whose changed number precedes the first colon) instead of a
  bare set diff that reported every value change twice; aggregate-metric
  leaves (sections[*], verdict.reasons/axes) follow the numeric
  attribution class warmup_aggregate_effect — digit-carrying only, so a
  pure label flip still fails. New registered items: spec.requires prefix
  matching, the deliberate sanity_report stem rename,
  exec_price_artifact_reused False->True (same-session run-order artifact),
  and jump's spec.description / sections[7].payload.factor_version under
  the correction class (structured corrections block required).

Catalogue: new section documents the classes, measured bounds, the MD
pairing rule, the JSON registrations, and the cognitive fix — the C4
handoff's 'bounded factors expect zero differences vs the D1 baseline'
prediction was wrong: the old runners' 'full window' is also
anchor-truncated at data.start.

Rerun (cache-only, stk_mins_live_calls=0 everywhere): all 3 factors x 3
modes rc=0, out-of-class differences 0 in every leg.

Tests: 29 -> 56 in test_factor_eval_reconcile.py (6 node IDs renamed with
the retired class names, assertions preserved or strengthened; every new
class has positive AND negative tests — warmup-window-outside changes,
out-of-bound float tails, flip amplitude > 1, cap overflows, and prose
word changes all still FAIL).
@StackOverFlow11
StackOverFlow11 merged commit b93c3b8 into main Jul 28, 2026
StackOverFlow11 added a commit that referenced this pull request Jul 28, 2026
…5-c4

docs: record refactor step D5 C4 (PR #105)
@StackOverFlow11
StackOverFlow11 deleted the feat/factor-refactor-d5-c4-unified-runner branch July 28, 2026 01:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant