feat(eval): unified exec-only FactorEvalRunner via factors.service (D5 C4) - #105
Merged
Merged
Conversation
…sification (D5 C4)
One runner body for the eleven minute-derived factors, replacing the
eleven near-identical eval_*.py main flows per the D5 difference
catalogue (section 4 = fixed body; sections 2/3 = four extension points):
* qt/factor_eval_disclosures.py (new): the single home of the four
heterogeneous coverage disclosures (RidgeCoverage / RidgeReturnCoverage
/ PeakCoverage / NeutralizationCoverage + summarizers), moved VERBATIM
from the four legacy runners (verified block-for-block against HEAD),
which now re-export them. Two documented additions: a render() on
NeutralizationCoverage (catalogue one-site normalization; the CLI's
inline f-string now goes through it, byte-identical format pinned) and
to_section() packing a coverage into an add-Section (contract 3.6:
verdict-lazy by construction, pinned by test). Catalogue BUG 3 closed:
summarize_peak_coverage gets the counterfactual + empty-frames tests
its siblings had.
* qt/factor_eval_runner.py (new): subject values from factors.service.panel
(decision view x exec_to_exec, store read-through + D4c read-assembly
combine), processing via the shared _process_factors, exec evaluation
via run_exec_basis_evaluation with an additive default-off
extra_sections seam. Dual book_mode: 'close' replicates the legacy book
path; 'decision' (default) takes the book through the same service
panel (design 1.1 close-view book defect closed on this path). The
EvalConfig declares the exec identity explicitly (17 shared kwargs
unchanged). Catalogue BUG 5 closed: a config factors: block that
disagrees with the fixed book is a readable error. Artifact stem
factor_eval_{factor_id} (+ _bookclose suffix in close mode) never
collides with the legacy eval_{name} artifacts. RunRegistry append is
a deliberate deferral to D7. valley_price_quantile is a readable
deferral (PR-C4b).
* CLI: one new subcommand run-factor-eval --config X --factor F
[--book-mode decision|close]; the eleven legacy _cmd_run_eval_* are
untouched (C6 convergence).
* config/factor_eval_csi500.yaml: the 11-config shared block with
factors: [] (BUG 5 honest declaration).
Gates: pytest 2431 passed / 5 skipped (node-ID diff +29, 0 existing
squeezed), ruff clean, validate-config 32/32, phase0 demo anchor
ic 0.9600 / annual 0.8408 unchanged.
…at tails as named classes (D5 C4) First full reconciliation (3 factors x 3 modes) failed every leg. The independent investigation attributed ALL differences to two non-bug root causes; this change completes the judging criteria so each is a NAMED, machine-checkable, bounded class — anything outside still fails (an unregistered difference is a failure): * warmup_left_extension (panels + anchors): the old runners anchored every symbol's load at data.start (2021-07-01), so the frozen grid's first w-1 trading dates are under-warmed; the materializer saturates to the minute cache's real start (2015-01-05). Bounded: the grid's first lookback_depth-1 trading dates, all three directions (NaN->finite, finite->finite as partial pools fill, new-only finite rows). Pooled: the early region [2021-07-01, 2021-10-31] with non-increasing per-month counts. This also corrects the attribution text: the effect is the left-extension warmup, not the D4 union-trim repair. Anchors: bounded factors get the same boundary (the harness only had the pooled early region before — the asymmetry that failed minute_ideal_amp and jump); jump's NON-warmup rows must still reconcile (they prove the truncated definition is carried). * float_reordering_tail: scattered finite-vs-finite cells with rel <= 5e-12 AND <= 101 cells (the measured jump tail; rolling-correlation summation order). The global tolerance is NOT widened. * threshold_flip_tail: count factors — rolling-sigma float noise x an integer volume on the peak threshold flips the count by EXACTLY +/-1 (rel <= 1e-2, <= 25 cells; measured 20 cells, 600623.SH). * reports leg: Markdown is now paired into row-level changes by the '- key:' head (two passes: exact head, then digit-normalized head for prose lines whose changed number precedes the first colon) instead of a bare set diff that reported every value change twice; aggregate-metric leaves (sections[*], verdict.reasons/axes) follow the numeric attribution class warmup_aggregate_effect — digit-carrying only, so a pure label flip still fails. New registered items: spec.requires prefix matching, the deliberate sanity_report stem rename, exec_price_artifact_reused False->True (same-session run-order artifact), and jump's spec.description / sections[7].payload.factor_version under the correction class (structured corrections block required). Catalogue: new section documents the classes, measured bounds, the MD pairing rule, the JSON registrations, and the cognitive fix — the C4 handoff's 'bounded factors expect zero differences vs the D1 baseline' prediction was wrong: the old runners' 'full window' is also anchor-truncated at data.start. Rerun (cache-only, stk_mins_live_calls=0 everywhere): all 3 factors x 3 modes rc=0, out-of-class differences 0 in every leg. Tests: 29 -> 56 in test_factor_eval_reconcile.py (6 node IDs renamed with the retired class names, assertions preserved or strengthened; every new class has positive AND negative tests — warmup-window-outside changes, out-of-bound float tails, flip amplitude > 1, cap overflows, and prose word changes all still FAIL).
StackOverFlow11
added a commit
that referenced
this pull request
Jul 28, 2026
…5-c4 docs: record refactor step D5 C4 (PR #105)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
D5 C4 — unified exec-only FactorEvalRunner (factor data access now goes through factors.service)
Design
tmp/design/factor_refactor_design_v3.mdstep D5 C4. The eleven per-factor eval runners keep living side-by-side (deletion is C6, gated on the C5 four-leg audit); this PR adds the unified runner and its reconciliation harness, and proves it on three factors against the frozen baselines.What changed (4 commits)
qt/factor_eval_providers.py— real-cache providers:CacheMinuteProvider(promoted fromfactor_hotpath_smoke.py, single source, all three importers repointed),DailyEvalPanelProvider(close-view, un-lagged — the materializer appliesdaily_decision_lagitself),EvalServiceBundle+build_eval_servicereusingqt/pipelinein the old runners' call order. Plusfactors/store/incremental.py: stored-NaN→fresh-finite is now classifiedfilled_after_footprint(was misdiagnosed as "upstream data REVISION" — the D4c footprint rows), and multi-column intermediate payloads raise the namedIntermediatePayloadNotWiredError(deliberate deferral, cross-referenced inmake_recompute_fn).qt/factor_eval_runner.py+qt/factor_eval_disclosures.py— the unified runner:service.panel(..., view=DECISION, basis=EXEC_TO_EXEC)replaces all eleven_load_*_panelpaths; heterogeneous coverage disclosures moved verbatim (8/8 VERBATIM, programmatically checked) into one home with old runners re-exporting;book_mode=decision(book via service, closing the close(d)-book live defect) orclose(faithful legacy); add-Section assembly keeps mandatory sections and verdict bit-identical; new artifact stemfactor_eval_*(never collides with legacyeval_*); catalogue BUG 5 (unreadfactors:config block → readable error) / BUG 6 (name-clash tests) / BUG 3 (PeakCoverage zero tests) closed. New CLIrun-factor-eval+config/factor_eval_csi500.yaml.qt/factor_eval_reconcile.py— the C5 four-leg audit harness:panels/reports/anchorsmodes, entry hard-gate requiring the frozen baseline 77/77 (unreachable baseline = hard error, never a skip), jump routed to thepr_c_cutoff_fixreference.data.startwith no extension, so frozen panels are under-warmed for the first w−1 trading days; the materializer left-extends toward the true cache start 2015-01-05) plus two float tails. New named, machine-checkable classes:warmup_left_extension(bounded: first w−1 days, all directions; pooled: early window + monthly non-increasing),float_reordering_tail(rel ≤ 5e-12, ≤101 cells),threshold_flip_tail(±1 count, rolling-σ noise × integer volume threshold), MD key-pairing, JSON registrations. Cataloguedocs/factors/d5_runner_difference_catalogue.mdgained §七之三, including a cognition correction: the handoff §3① prediction "bounded factors expect zero diff vs the D1 baseline" was wrong — the old runners' "full window" was itself anchor-truncated; that wrong prediction, not an engine bug, was the source of the first-round red.Real-run reconciliation (cache-only,
stk_mins_live_calls=0everywhere)Three factors × three modes, 9/9 green, zero unclassified diffs:
minute_ideal_amp_10jump_amount_corr_20volume_peak_count_20jump's 3 non-warmup anchor rows went green (rel ~1e-15) — the #103 cutoff fix confirmed carried by the new path.
Review
Adversarial review (separate executor): APPROVE-WITH-NITS — 0 CRIT/HIGH/MEDIUM, with 64 self-built mutation probes against the classifier classes (out-of-window diffs, over-cap tails, non-monotone monthly counts, label-only flips, reverse run-order artifact all verified to FAIL), verbatim-move rechecked 8/8, hard gate tamper-tested on a /tmp manifest copy, and an independent rerun of the
minute_ideal_amp_10panels mode matching the implementer's counts cell-for-cell. Registered non-blocking items:warmup_left_extensionhas no magnitude/cell-count ceiling — an in-window error of arbitrary size is absorbed (~1.6% of the grid for w=20, conservative direction; pooled side has the monthly-monotonicity constraint). C5 follow-up: add a cell-count ceiling.verdict.reasonsline replaced by a digit-free different-meaning line still registers aswarmup_aggregate_effect. Verdict labels/axes/spec/eval_config remain strict. Acceptable for C5's purpose; recorded.ReconciliationError(decorative; service panels cannot contain dup keys). NIT-2: theexec_price_artifact_reusedreverse direction (True→False) has no pinning test (behavior verified FAIL by the reviewer's own mutation).Gates (independently rerun by the lead)
pytest2492 passed (node-ID set diff: 0 pre-existing tests displaced),ruffcleanrun-phase0demo anchor ic 0.9600 / annual 0.8408 (unchanged)validate-config32/32 (newfactor_eval_csi500.yamlcovered by the config glob)external_secret_filepath andtushare_token_keyname in the new config, byte-identical in form to the 11 existing eval configs on main; no secret value anywhereexec_baseline_freeze --verify77/77 @45c14aa; frozen panels/baselines untouched (read-only throughout)Deliberately NOT in this PR
Old-runner deletion (C6), full 11-factor four-leg audit (C5),
valley_price_quantilebinding + foldingbinding.pyintocode_hash(PR-C4b, lead-adjudicated),mmp_ew(D6c), run-registry integration (D7),tail_recomputewiring for intermediate payloads (explicit named refusal, 21:00-update story).