feat: P4-3 factor-support caching + 21:00 data updater - #31
Merged
Conversation
Extends the daily read-through cache (shared ledger/store) to the remaining pipeline endpoints, with not-ready + field-set-independent fina coverage. - daily_basic: dense date-range (pe/pb/total_mv); one cached call feeds both market_cap and value_ratios. - fina_indicator: coverage by report-period end_date, keeps ann_date raw, long trailing tail (recent_tail_overrides) catches LATE disclosures. ALWAYS stores the canonical FINA_FIELDS superset (roe/netprofit_yoy/grossprofit_margin), so a warm for one config's fields never blocks another's (a fields_hash-only fix would corrupt the shared per-symbol parquet on upsert). The feed selects its requested subset on read; a drift test guards FINA_FIELDS >= financial.SUPPORTED_FIELDS. - index_member_all: per-symbol SW in/out dimension (staleness refresh). - not_ready pending window: today's empty return is recorded not_ready (NOT coverage), retried next run; not_ready_days=0 (default) keeps old behaviour byte-identical. Per-endpoint trailing tail + written/not-ready summary counters. - Feeds read through the cache (cache=None -> direct byte-identical); shared cache threaded into the financials/value/covariates enrich; DataUpdateCfg (optional on RootConfig). cached==direct tested for market_cap/value_ratios/pit_sw/fina-as-of. Daily backtest math / factor / alpha / portfolio / runtime untouched; Phase 0/2/3 unchanged.
Standalone entry point that ONLY warms/updates the read-through caches on a daily 21:00 Asia/Shanghai schedule (scheduling is external — systemd timer / cron). It never computes factors / alpha / portfolio, never runs a backtest, never writes a PanelStore; the backtest still does its own read-through to fill gaps. - qt/data_updater.py: run_data_update + testable update_endpoints + format_summary. Builds the daily + intraday caches with the not-ready window and the fina late-disclosure tail; warms each configured endpoint; emits a per-endpoint summary (requests / rows_written / not_ready). Daily + intraday use separate ledgers/stores. - qt/cli.py: data-update subcommand. - config/data_update.yaml: the 21:00 job config (timezone / scheduled_start / endpoints / lookback / tails / not_ready / rate_limit 450). fina_fields lists grossprofit_margin (the cache stores the superset regardless; explicit for a self-documenting summary). - Incremental semantics: covered historical run ~0 API calls; new trading day fetches only new dates / tail; failed fetch records no coverage; today's empty is not_ready (retried), never frozen as covered. validate-config OK for example.yaml + data_update.yaml; run-phase0 unchanged.
StackOverFlow11
added a commit
that referenced
this pull request
Jun 13, 2026
docs: record P4-3 factor-support caching + data updater (PR #31)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Caches the remaining pipeline endpoints (
daily_basic,fina_indicator,index_member_all) on the existing daily read-through cache, and adds a standalonedata-updateCLI that warms ALL caches incrementally on a daily 21:00 Asia/Shanghai schedule (scheduling is external — systemd timer / cron). The updater never runs factors/alpha/portfolio/backtest and never writes a PanelStore; the backtest still does its own read-through to fill gaps.Daily backtest math / factor / alpha / portfolio / runtime are untouched; Phase 0/2/3 numbers unchanged (
run-phase0stillic_mean=0.9600, annual_return=0.8408).Commit 1 —
feat(cache): cache the 3 factor-support endpointsmarket_capandvalue_ratios.end_date, keepsann_dateraw, long trailing tail catches LATE disclosures. ALWAYS stores the canonicalFINA_FIELDSsuperset (roe/netprofit_yoy/grossprofit_margin) so a warm for one config's fields never blocks another's — afields_hash-only fix would corrupt the shared per-symbol parquet on upsert (codex acceptance blocker, now fixed). The feed selects its requested subset on read; a drift test guardsFINA_FIELDS ⊇ factors.compute.financial.SUPPORTED_FIELDS.not_ready(NOT coverage), retried next run;not_ready_days=0(default) keeps old behaviour byte-identical. Per-endpoint trailing tail + summary counters.cache=None→ direct byte-identical);DataUpdateCfgoptional on RootConfig.Commit 2 —
feat(cli): data-update 21:00 incremental warmqt/data_updater.py(run_data_update+ testableupdate_endpoints+format_summary),qt/cli.pydata-updatesubcommand,config/data_update.yaml.Invariants / safety
cache=None→ every feed's direct path byte-identical (old feed tests pass).Test plan
pytest tests -q→ 502 passed (+17 P4-3: 10 cache + 7 feed/updater, incl. the fina mixed-field regression + drift guard; 0 regressions).ruff check .clean.validate-configOK forconfig/example.yaml+config/data_update.yaml.run-phase0 --config config/example.yaml→ic_mean=0.9600, annual_return=0.8408(unchanged).Codex acceptance: originally FAIL (fina mixed-field blocker) → cc fixed via canonical-superset storage → revalidated PASS.