feat: factor refactor D3 — sparse value store, keys/fingerprints, tail-recompute (+R5 fina horizon unification) - #91
Merged
Conversation
The backtest read-through cache builder (qt.pipeline._build_cache) passed NO recent_tail_overrides, so the fina_indicator revision horizon silently collapsed to refresh_recent_days (14) while the data-update warm used 400. The factor store's incremental overlap window (D3) is sized from this horizon, so a disagreeing value is a correctness bug, not a perf nit. Move the knob to data.cache.fina_tail_days (default 400) as the single source: both the backtest read-through and the data-update warm resolve it from there, so the two paths can never drift. Remove the drift-prone data_update.fina_tail_days knob and migrate the two data_update configs to data.cache.fina_tail_days. Behaviour change (disclosed, never silent): with data.cache enabled the backtest read-through now refetches the fina 400-day tail to catch late disclosures, matching the data-update warm. Tests (network-free): the value reaches both builders, a config bump follows through (mutation), the removed knob is rejected, and the value flows into the D0 revision_horizon the D3 engine consumes.
…Key (D3) The complete store key (red line #6): factor_id + params + code_hash + view, plus a data fingerprint DERIVED FROM the adjustment declaration. - hashing.py: content-hash primitives. content_hash_of_labeled_files hashes file CONTENT + a stable module LABEL, never the path, so the digest is checkout-independent (a moved repo must not invalidate stored values). - code_hash.py: code_hash(factor) = content hash of the factor's own module + the ENUMERATED shared set {minute.primitives, ops/*, base, spec}. NO transitive import-graph walker (R3); an AST import allowlist keeps the set honest — a factor module may import only stdlib + numpy/pandas + the allowlisted internal leaves, and a new first-party import goes red, forcing a review. Dynamic / relative imports are refused. - fingerprint.py: schema-version dimension (content hash of the PIT-bearing intraday_schema + intraday_aggregate) + adjustment-derived anchor events — none/returns_invariant carry NO adj-factor table (supplying one is a contradiction); price_level MUST include the per-symbol adj_factor event hash (omitting it is a loud error, §六.17 qfq-anchor trap). Zero closing-set price_level members does not waive the mechanism (A5-F02). - keys.py: StoreKey -> {factor_id}/{params_hash}__{code_hash}__{view}.parquet; the fingerprint stays OUT of the filename (validated on read); view coerced through the availability-policy View (unknown view = readable error). Tests (network-free): params/code hash stability + module distinction; the three key-mutation teeth (shared-set member content change -> code_hash change; schema-module content change -> fingerprint change; allowlist violation -> red, with a real-module injection mutation rc=1->rc=0); price_level fixture factor.
FactorValueStore (values.py): one parquet per StoreKey holding a RAW MultiIndex(date, symbol) Series, single-file + date-sorted (not partitioned). Writes are atomic (tmp + os.replace) under a per-artifact O_CREAT|O_EXCL lock (mirrors CacheParquetStore._locked), so a concurrent gap-fill never loses rows. No coverage ledger — a miss just recomputes (values are deterministic). A corrupt file is a MISS (read returns None), never a crash. The data fingerprint is stored in the parquet FILE METADATA (not the filename): read_valid voids the WHOLE artifact on a schema-version mismatch and voids only the restated symbol column on a price_level adj-hash mismatch. RunRegistry (run_registry.py): append-only JSONL lineage (verdicts append-only, #74) — status is a plain string field (exploratory|watch|book|retired), NOT a transition machine (R25). No-secret contract (R22): only factor_id / hashes / view / status / endpoint names / adjustment + schema fingerprint / timestamp / note, every free-text leaf redacted via data.quality.sanitize_text. Startup consistency (assert_registry_consistent): the code registry must know every run-registry factor id, else a readable error. Tests (network-free): roundtrip / corrupt-miss / atomic no-residue / upsert keep-last / fingerprint read-validation (schema void + price_level per-symbol void) / two-process concurrent gap-fill loses zero rows (lock mutation rc=1 when disabled). Registry: append-only, status validation, secret redacted in a note, no-secret in a real record, startup consistency pass/flag.
…th (D3) The 21:00 factor-update mechanism (design §3.3): read the store tail, recompute a bounded overlap window, validate it bit-for-bit against the stored values, then truncate the overlap and append the new tail. The overlap validation is the only detector of a silent upstream data revision, so it is an entry ticket (§六.14). Overlap window = max(W_dep, max endpoint revision horizon) (R4/R5): - W_dep is the factor's TRANSITIVE lookback depth, declared on a NEW additive FactorSpec field ``lookback_depth`` (contract v1.1, stated outright). It is optional at construction (so v1.0 specs stay valid) but HARD-REQUIRED by the engine — a missing declaration is a readable error, NEVER a silent 0 or headline fallback (§六.18: the headline window deceives a nested-lookback factor such as ridge = lookback 20 over baseline 20, depth ~40). - the per-endpoint revision horizon is resolved from the LIVE cache config via data.availability_policy.revision_horizon, so bumping refresh_recent_days widens the overlap (R5). Mismatch action (§3.3 修订 2): the whole column of each mismatched symbol is re-computed (not locally patched), classified by adjustment — price_level mismatches are EXPECTED (an ex-date re-based the level); returns_invariant/none mismatches mean upstream data actually changed and are recorded LOUDLY. Tests (network-free): a faithful NESTED-LOOKBACK fixture (trailing-L sum of a trailing-B mean, depth L+B-2, computed per-date so it is bitwise-stable across load windows) proves batch==incremental with the correct transitive depth AND the §六.18 teeth (headline depth spuriously mismatches); overlap window follows the config (R5); missing lookback_depth is a loud error; cold path; and the returns_invariant (loud) vs price_level (expected) mismatch classification.
…T-NEW) Pins the adjustment declaration testability (D0 §3.4 P9) to D3, since the store fingerprint is derived from it (Commit 2): - returns_invariant, runtime: perturbing the price-adjustment anchor leaves the factor value bit-for-bit unchanged — a DAILY factor (volatility_20, per-symbol qfq-anchor rescale) and a MINUTE factor (valley_relative_vwap, the within-day VWAP ratio under a per-day price rescale). A power-of-two anchor makes the mathematical cancellation exact in float, so the invariance is bitwise. - returns_invariant, batch: all 9 returns_invariant closing factors' fingerprints omit the adj-factor event table; all 5 none factors likewise (and their input-list static check held at construction). The census matches D0 (9/5/0). - price_level fixture (zero closing-set members NOT waived, A5-F02): the fingerprint includes the per-symbol adj-factor event hash and changes when the anchor is restated; mislabelling it returns_invariant makes the fingerprint blind to the anchor (the §六.17 stale-reuse trap). Mutation evidence: each invariance test has a positive-control teeth test (the rescale moves a level-dependent stat / a non-uniform within-day change moves the ratio), and the returns_invariant teeth was flipped to expect invariance on a level stat -> rc=1, reverted -> rc=0.
…lowlist (D3) MEDIUM — close the momentum->reversal stale-reuse hole. code_hash now folds a factor module's DIRECT project-internal imports that are NOT in the shared set (ONE HOP, module-granular, derived from the AST — no manual list, red line #6; no transitive-closure walk, design §3.4). candidates.py composes MomentumFactor, so momentum.py's value surface folds into every candidates factor's key: a change to momentum.py invalidates reversal/value/volatility but NOT factors in other modules (financial.py), so no global over-invalidation. Granularity = the module file (over-invalidation costs a recompute; under-invalidation serves a stale value). Mutation evidence: mutate momentum.py -> reversal_20/value_ep code_hash CHANGED, roe/ridge unchanged. LOW-1 — tail_recompute validated only bar content, so a schema-version change was misclassified as a returns_invariant upstream revision (loud). Now it validates the stored fingerprint's schema dimension FIRST: a mismatch is a MISS -> schema_void_full recompute, never a revision warning. The price_level per-symbol anchor mismatch stays on the existing adjustment-classified overlap path. LOW-3 — remove dead allowlist entries data.clean.schema / factors.requires (no shipped factor imports them). A future import goes red and forces an explicit decision (the intended behaviour). Every allowlisted internal import also folds into the factor's code hash via the one-hop rule, so allowlist and code hash cannot silently drift. LOW-2 / NIT (docstrings): CacheHorizonConfig — D4 must pass the live data.cache.fina_tail_days, not the 400 default; short_hash — the 64-bit truncation collision risk is known/ignorable (not a fingerprint, git convention).
StackOverFlow11
added a commit
that referenced
this pull request
Jul 24, 2026
docs: record refactor step D3 (PR #91)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Factor-layer refactor step D3 (design v3.2 §3.3/§3.4/§9 D3 row): the sparse factor-value store — key derivation, fingerprints, atomic storage + append-only run registry, and the tail-recompute incremental engine. The store is a purely additive layer in this step: nothing consumes it yet (D4 wires the read-through service), so rollback = don't use it. Plus the R5 prerequisite fix (fina revision-horizon unification).
What (6 commits)
875a3bdR5 prerequisite:fina_tail_daysmoves to its single sourcedata.cache.fina_tail_days(default 400, validated); BOTH cache builders (backtest_build_cache+ data-update_build_caches) now passrecent_tail_overrides={"fina_indicator": ...}from it. Previously only the data-update path carried the 400-day late-disclosure tail — the revision horizon was misstated for the very path that will fill the factor store.33c5c11Key derivation:params_hash(sorted-JSON) +code_hash(content hash of the factor module + the enumerated shared set {minute primitives,ops/*, base, spec}; AST import-allowlist guard, no dynamic imports, no transitive graph walker) + view + data fingerprint (schema-version dimension = content hash of the PIT-bearingdata/cleanmodules;price_leveladditionally folds a per-symbol adj_factor event-table hash incl. anchor — exercised via fixture, closing set has zero members but the mechanism is tested, A5-F02).64cfdffSparse value store + run registry:artifacts/factor_store/values/{factor_id}/{params_hash}__{code_hash}__{view}.parquet(MultiIndex(date,symbol), single file, date-sorted) +registry.jsonl. Atomic writes (tmp +os.replace), per-artifact O_CREAT|O_EXCL lock, corrupted file ⇒ miss, no coverage ledger (values are deterministically recomputable), raw-only, no-secret contract (sanitize_text on free text; tested), startup consistency assertion (registry entries must resolve in the code registry).5bd350dTail-recompute:w_overlap = max(W_dep, max endpoint revision horizon);W_dep= transitive lookback depth via a new optionalFactorSpec.lookback_depth(contract v1.1, additive) that the store engine hard-requires at consumption (missing ⇒ readable error, never a silent 0/headline). Overlap re-verified bitwise (NaN-aware); mismatch ⇒ whole-symbol recompute, classified byadjustment(returns_invariant⇒ loud upstream-revision record;price_level⇒ expected). Horizons derived viadata.availability_policy.revision_horizon()from live config — no hardcoded 14/400.f768e36#adjustment NET-NEW property tests (pinned to D3 per D0 ledger): perturb the qfq anchor ⇒returns_invariantfactors bit-identical (real daily + real minute factor);price_levelfixture key/fingerprint moves with the anchor; declaration census 9/5/0 matches D0.fd906f4Review fixes (see below).Acceptance (dual-track)
Adversarial review (fresh reviewer): APPROVE-WITH-NITS — 0 CRIT / 0 HIGH / 1 MEDIUM / 3 LOW / 1 NIT; reviewer independently re-ran the allowlist-injection and lock-neuter mutations (rc teeth confirmed) and hunted for impossible-to-fail tests (none found; batch≡incremental is an assembly test with a §6.18 headline-depth teeth case, not a self-comparison).
fd906f4):reversal_20is buildable and delegates toMomentumFactor.compute, butmomentum.py's surface was not folded into itscode_hash⇒ editing momentum could change reversal's values without changing its key (stale reuse; red line docs: progress — PR #5 (P2-3 PIT industry) merged #6). Fix:code_hashnow also folds one-hop direct project-internal imports outside the shared set (AST-derived, module granularity, no manual list, no transitive closure). Delta-verified by the reviewer with an on-disk mutation: mutatingmomentum.pychanges reversal/value/vol keys (candidates.py hosts) and does not change financial/minute keys (no global over-invalidation).tail_recomputenow validates the stored schema-version fingerprint first; mismatch ⇒schema_void_fullfull recompute, not a spurious upstream-revision warning.data.cache.fina_tail_days— the 400 default onCacheHorizonConfigis for tests/standalone only.data.clean.schema,factors.requires) removed; importing them now trips the guard, forcing an explicit decision.Lead independent gates (not rubber-stamping agent numbers): pytest 2031 passed, rc=0 (dot-count 2031 = 1955 baseline + 76 new; FAILED/ERROR grep 0) · ruff clean · phase0 anchor
ic 0.9600 / annual 0.8408unchanged · validate-config 31/31 · provisional D2 anchor green · secret scan ongit diff main...HEAD: 0 real hits (the only.config.json/token literals are a redaction test's synthetic path + fake token with assertions they are NOT persisted) · layering spot-check:factors/storehas zeroqt/analyticsimports · R5 wiring spot-check atqt/pipeline.py:579.Disclosures
data_update.fina_tail_daysis removed (strict configs reject it); both in-repo data_update YAMLs migrated. Out-of-repo YAMLs still carrying the old key will fail validation loudly — breaking, disclosed.lookback_depthoptional-at-construction / hard-required-at-store-consumption (declaring transitive depth is a pre-registration act, left to D4 per-factor wiring — the 14 real factors do not declare it yet, so feeding them to tail-recompute errors loudly by design); fingerprint lives in parquet file metadata (self-describing read validation) with the short schema hash mirrored in the registry line (lineage); shared-set/schema files hashed by module dotted-name label (checkout-independent).data/availability_policy.pyedits now over-invalidate factor stores that import it (safe direction — costs recompute, never staleness — but a real ops cost during D4/D5 churn); single-fill ≡ batch-fill (P8) is a D4 acceptance item.No CLAUDE.md/AGENTS.md/progress-doc changes in this PR (docs go in a separate PR per convention). Frozen baseline
artifacts/refactor_baseline/untouched.