P4-1: persistent Tushare market-data cache (daily + adj_factor) + run-log stats - #23
Merged
Merged
Conversation
Endpoint-level raw cache below the feeds: real runs no longer refetch
full daily bars + adj_factor every time. Disabled by default
(backward compatible); an opted-in config reads market bars
read-through so only uncovered date ranges hit the API.
- config: DataCfg.cache (CacheCfg) — enabled=False, root_dir
(default artifacts/cache/tushare/v1), refresh_recent_days=14,
force_refresh=[]. Validated; every existing config still loads.
- data/cache/ package:
- intervals.py: closed-day interval algebra (merge/subtract) for
gap planning — coverage is by calendar range, not row presence.
- parquet_store.py: per-(endpoint, symbol) parquet under a
symbol_prefix shard; atomic upsert-by-(date,symbol) dedup
(latest wins); best-effort per-symbol lock.
- coverage.py: append-only ledger (endpoint, key_type, key,
start/end_date, fields_hash, fetched_at, row_count, status
ok/empty/failed, schema_version, source_version). Only ok/empty
count as coverage; a failed fetch stays uncovered. No token or
secret-file content is ever written.
- tushare_cache.py: read-through daily_bars/adj_factor — fetch only
gaps, upsert raw rows, record coverage incl. empty returns, then
return the requested range from cache. refresh_recent_days
refetches a recent tail; force_refresh re-pulls an endpoint in
full. Per-run fetch-count stats logged.
- TushareFeed.get_bars: cache=None -> unchanged direct path;
cache set -> read-through, then the SAME join + select as the direct
path so the panel is byte-identical before front_adjust(). The
per-symbol retry/throttle stay in the feed via _call closures.
- pipeline._build_market_cache wires the cache only when
data.cache.enabled. front_adjust, factor/alpha/portfolio/execution/
OOS/report math all unchanged. PanelStore stays a per-run artifact.
- config/phase2_real_baseline_cached.yaml: cache-enabled smoke variant.
- tests: +24 (cache config validation + all-configs parametrized;
market cache: full miss/hit, partial gap, empty coverage, duplicate
upsert one-key, qfq equivalence vs direct fetch, no-secret in cache
files) -> 406 passed; ruff clean; demo run-phase0 unchanged
(ic 0.96 / annual 0.8408).
- CLAUDE.md / AGENTS.md (verbatim copy): P4-1 progress entry — design (raw-only endpoint cache below feeds, coverage ledger as SoT, gap planning by calendar range, front_adjust unchanged downstream), the cache-disabled backward-compat invariant, and the real phase2 smoke: non-cached reference vs cached cold vs cached warm metrics ALL IDENTICAL (IC 0.0083 / annual -10.19% / maxDD -16.52% / vol 16.59% / sharpe -0.5703 / turnover 1.0818 / cost 1.19%); warm run made ZERO market_daily/adj_factor calls (coverage ledger 68+68 unchanged between cold and warm); wall ref 998s / cold 1025s / warm 734s; cache files + ledger carry no token/secret. Gates updated to 407 passed; roadmap now points at P4-2 (universe/tradability) then P4-3 (factor-support endpoints). - TEST_REPORT.md: P4-1 per-file breakdown (cache config 17 + market cache 7 = 24) + real-smoke entry; total 407. - RUNBOOK.md: Phase 4-1 section (opt-in config, read-through behaviour, cache layout, real smoke commands, market-bars-only scope note).
Review follow-up (non-blocking): the run log did not directly show the
cache hit rate — warm-hit evidence relied on coverage-ledger row counts.
Now the run-scoped logger emits the per-endpoint gap-fetch tally so a
warm historical rerun is self-evident.
- TushareFeed.cache_stats() -> dict | None: the read-through cache's
per-endpoint API-call tally (None when caching is disabled). Endpoint
counts only — no token, secret path, or per-symbol detail.
- pipeline._log_cache_stats(feed, logger): called in _load_panel right
after get_bars; logs one line through the existing run-scoped logger
when the feed exposes cache_stats() (no-op for DemoFeed / cache off):
data cache: market_daily_gap_fetches=<N> adj_factor_gap_fetches=<M>
A cold run shows nonzero counts; a warm historical rerun shows 0/0.
- No change to cache behaviour, schema, or any factor/alpha/portfolio/
execution/report math; no new endpoints cached.
- tests: +4 (feed exposes cold {2,2} then warm {0,0} stats over a shared
cache root; cache_stats None without a cache; pipeline logs the exact
line; no line when cache absent/disabled) -> 411 passed; ruff clean.
Records the P4-1 follow-up: TushareFeed.cache_stats() + the run-log line `data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=M` (cold nonzero, warm historical rerun 0/0 — confirmed in run_phase2_baseline.log, secret scan 0). Gate count 407 -> 411 across CLAUDE.md / AGENTS.md / TEST_REPORT.md; RUNBOOK P4-1 read-through bullets mention the visible-hit-rate line.
StackOverFlow11
added a commit
that referenced
this pull request
Jun 13, 2026
docs: mark PR #23 (P4-1) merged in progress docs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
P4-1: a persistent endpoint-level RAW cache below the feeds so real runs stop refetching full daily bars + adj_factor every time. Disabled by default (backward compatible — an existing config runs exactly as before). The cache stores RAW rows only (unadjusted OHLCV/amount, raw adj_factor);
front_adjuststill runs in memory downstream, unchanged.PanelStorestays a per-run artifact, NOT the cache source of truth. Market bars only —index_weight/daily_basic/fina_indicator/... are still fetched live (cached in P4-2/P4-3).data.cache(CacheCfg):enabled=False,root_dir(defaultartifacts/cache/tushare/v1, gitignored),refresh_recent_days=14,force_refresh=[]. Validated; every existing config still loads.data/cache/:intervals(closed-day interval algebra — coverage is by calendar range, not row presence, so the planner distinguishes "not fetched" from "source has no row");parquet_store(per-(endpoint,symbol)parquet under a symbol_prefix shard; atomic upsert-by-(date,symbol), latest wins; best-effort per-symbol lock);coverage(append-only ledger — endpoint/key_type/key/start_date/end_date/fields_hash/fetched_at/row_count/status ok|empty|failed/schema_version/source_version; only ok/empty count as coverage; no token or secret ever written);tushare_cache(read-through: fetch only gaps, upsert, record coverage incl. empty returns;refresh_recent_daysrefetches a recent tail;force_refreshre-pulls an endpoint; per-run fetch-count stats).TushareFeed.get_bars:cache=None→ unchanged direct path; cache set → read-through then the SAME join + select so the panel is byte-identical beforefront_adjust(). Per-symbol retry/throttle stay in the feed via_callclosures.pipeline._build_market_cachewires it only whendata.cache.enabled.TushareFeed.cache_stats()+pipeline._log_cache_statsemit one run-log line —data cache: market_daily_gap_fetches=<N> adj_factor_gap_fetches=<M>(cold nonzero; warm historical rerun 0/0).Real smoke (phase2 baseline)
run_phase2_baseline.logcontainsdata cache: market_daily_gap_fetches=0 adj_factor_gap_fetches=0.Test plan
pytest -p no:cacheprovider— 411 passed (+28: cache config validation incl. all-configs parametrized; market cache full miss/hit, partial gap, empty coverage, duplicate upsert one-key, qfq equivalence vs direct fetch, no-secret; feedcache_stats()cold {2,2}→warm {0,0}; pipeline logs the exact line)ruff check .— cleanvalidate-configon all 12 configs — OKrun-phase0demo regression — ic 0.9600 / annual 0.8408 unchangedgit diff --check+ conflict marker scan — cleanInvariants preserved:
front_adjustformula/timing unchanged (qfq-equivalence test + 3-run real evidence); factor/alpha/portfolio/execution/OOS/report math untouched;artifacts/data/{output_name}.parquetnot treated as the cache SoT.