Skip to content

P4-1: persistent Tushare market-data cache (daily + adj_factor) + run-log stats - #23

Merged
StackOverFlow11 merged 4 commits into
mainfrom
data-persistent-market-cache
Jun 13, 2026
Merged

P4-1: persistent Tushare market-data cache (daily + adj_factor) + run-log stats#23
StackOverFlow11 merged 4 commits into
mainfrom
data-persistent-market-cache

Conversation

@StackOverFlow11

Copy link
Copy Markdown
Owner

Summary

P4-1: a persistent endpoint-level RAW cache below the feeds so real runs stop refetching full daily bars + adj_factor every time. Disabled by default (backward compatible — an existing config runs exactly as before). The cache stores RAW rows only (unadjusted OHLCV/amount, raw adj_factor); front_adjust still runs in memory downstream, unchanged. PanelStore stays a per-run artifact, NOT the cache source of truth. Market bars only — index_weight/daily_basic/fina_indicator/... are still fetched live (cached in P4-2/P4-3).

  • config data.cache (CacheCfg): enabled=False, root_dir (default artifacts/cache/tushare/v1, gitignored), refresh_recent_days=14, force_refresh=[]. Validated; every existing config still loads.
  • data/cache/: intervals (closed-day interval algebra — coverage is by calendar range, not row presence, so the planner distinguishes "not fetched" from "source has no row"); parquet_store (per-(endpoint,symbol) parquet under a symbol_prefix shard; atomic upsert-by-(date,symbol), latest wins; best-effort per-symbol lock); coverage (append-only ledger — endpoint/key_type/key/start_date/end_date/fields_hash/fetched_at/row_count/status ok|empty|failed/schema_version/source_version; only ok/empty count as coverage; no token or secret ever written); tushare_cache (read-through: fetch only gaps, upsert, record coverage incl. empty returns; refresh_recent_days refetches a recent tail; force_refresh re-pulls an endpoint; per-run fetch-count stats).
  • TushareFeed.get_bars: cache=None → unchanged direct path; cache set → read-through then the SAME join + select so the panel is byte-identical before front_adjust(). Per-symbol retry/throttle stay in the feed via _call closures. pipeline._build_market_cache wires it only when data.cache.enabled.
  • Review follow-up (visible hit rate): TushareFeed.cache_stats() + pipeline._log_cache_stats emit one run-log line — data cache: market_daily_gap_fetches=<N> adj_factor_gap_fetches=<M> (cold nonzero; warm historical rerun 0/0).

Real smoke (phase2 baseline)

  • Non-cached reference vs cached cold vs cached warm: report metrics IDENTICAL across all three (IC 0.0083 / annual -10.19% / maxDD -16.52% / vol 16.59% / sharpe -0.5703 / turnover 1.0818 / cost 1.19%) — the qfq-equivalence unit test proves cached==direct at the data layer.
  • Warm run makes ZERO market_daily / adj_factor calls: coverage ledger 68+68 unchanged between cold and warm; a dedicated warm rerun's run_phase2_baseline.log contains data cache: market_daily_gap_fetches=0 adj_factor_gap_fetches=0.
  • wall: ref 998s / cold 1025s / warm 734s (warm saves the market-bar fetches; other endpoints still fetch live). Cache files + ledger carry no token / secret-file content.

Test plan

  • pytest -p no:cacheprovider411 passed (+28: cache config validation incl. all-configs parametrized; market cache full miss/hit, partial gap, empty coverage, duplicate upsert one-key, qfq equivalence vs direct fetch, no-secret; feed cache_stats() cold {2,2}→warm {0,0}; pipeline logs the exact line)
  • ruff check . — clean
  • validate-config on all 12 configs — OK
  • run-phase0 demo regression — ic 0.9600 / annual 0.8408 unchanged
  • git diff --check + conflict marker scan — clean
  • AGENTS.md == CLAUDE.md (verbatim copy convention)
  • secret scan on the merge diff — 0 token, no artifacts/parquet committed
  • real smoke run (3-run metrics identical + warm 0/0 in the run log)

Invariants preserved: front_adjust formula/timing unchanged (qfq-equivalence test + 3-run real evidence); factor/alpha/portfolio/execution/OOS/report math untouched; artifacts/data/{output_name}.parquet not treated as the cache SoT.

Endpoint-level raw cache below the feeds: real runs no longer refetch
full daily bars + adj_factor every time. Disabled by default
(backward compatible); an opted-in config reads market bars
read-through so only uncovered date ranges hit the API.

- config: DataCfg.cache (CacheCfg) — enabled=False, root_dir
  (default artifacts/cache/tushare/v1), refresh_recent_days=14,
  force_refresh=[]. Validated; every existing config still loads.
- data/cache/ package:
  - intervals.py: closed-day interval algebra (merge/subtract) for
    gap planning — coverage is by calendar range, not row presence.
  - parquet_store.py: per-(endpoint, symbol) parquet under a
    symbol_prefix shard; atomic upsert-by-(date,symbol) dedup
    (latest wins); best-effort per-symbol lock.
  - coverage.py: append-only ledger (endpoint, key_type, key,
    start/end_date, fields_hash, fetched_at, row_count, status
    ok/empty/failed, schema_version, source_version). Only ok/empty
    count as coverage; a failed fetch stays uncovered. No token or
    secret-file content is ever written.
  - tushare_cache.py: read-through daily_bars/adj_factor — fetch only
    gaps, upsert raw rows, record coverage incl. empty returns, then
    return the requested range from cache. refresh_recent_days
    refetches a recent tail; force_refresh re-pulls an endpoint in
    full. Per-run fetch-count stats logged.
- TushareFeed.get_bars: cache=None -> unchanged direct path;
  cache set -> read-through, then the SAME join + select as the direct
  path so the panel is byte-identical before front_adjust(). The
  per-symbol retry/throttle stay in the feed via _call closures.
- pipeline._build_market_cache wires the cache only when
  data.cache.enabled. front_adjust, factor/alpha/portfolio/execution/
  OOS/report math all unchanged. PanelStore stays a per-run artifact.
- config/phase2_real_baseline_cached.yaml: cache-enabled smoke variant.
- tests: +24 (cache config validation + all-configs parametrized;
  market cache: full miss/hit, partial gap, empty coverage, duplicate
  upsert one-key, qfq equivalence vs direct fetch, no-secret in cache
  files) -> 406 passed; ruff clean; demo run-phase0 unchanged
  (ic 0.96 / annual 0.8408).
- CLAUDE.md / AGENTS.md (verbatim copy): P4-1 progress entry — design
  (raw-only endpoint cache below feeds, coverage ledger as SoT, gap
  planning by calendar range, front_adjust unchanged downstream), the
  cache-disabled backward-compat invariant, and the real phase2 smoke:
  non-cached reference vs cached cold vs cached warm metrics ALL
  IDENTICAL (IC 0.0083 / annual -10.19% / maxDD -16.52% / vol 16.59% /
  sharpe -0.5703 / turnover 1.0818 / cost 1.19%); warm run made ZERO
  market_daily/adj_factor calls (coverage ledger 68+68 unchanged between
  cold and warm); wall ref 998s / cold 1025s / warm 734s; cache files +
  ledger carry no token/secret. Gates updated to 407 passed; roadmap now
  points at P4-2 (universe/tradability) then P4-3 (factor-support
  endpoints).
- TEST_REPORT.md: P4-1 per-file breakdown (cache config 17 + market
  cache 7 = 24) + real-smoke entry; total 407.
- RUNBOOK.md: Phase 4-1 section (opt-in config, read-through behaviour,
  cache layout, real smoke commands, market-bars-only scope note).
Review follow-up (non-blocking): the run log did not directly show the
cache hit rate — warm-hit evidence relied on coverage-ledger row counts.
Now the run-scoped logger emits the per-endpoint gap-fetch tally so a
warm historical rerun is self-evident.

- TushareFeed.cache_stats() -> dict | None: the read-through cache's
  per-endpoint API-call tally (None when caching is disabled). Endpoint
  counts only — no token, secret path, or per-symbol detail.
- pipeline._log_cache_stats(feed, logger): called in _load_panel right
  after get_bars; logs one line through the existing run-scoped logger
  when the feed exposes cache_stats() (no-op for DemoFeed / cache off):
    data cache: market_daily_gap_fetches=<N> adj_factor_gap_fetches=<M>
  A cold run shows nonzero counts; a warm historical rerun shows 0/0.
- No change to cache behaviour, schema, or any factor/alpha/portfolio/
  execution/report math; no new endpoints cached.
- tests: +4 (feed exposes cold {2,2} then warm {0,0} stats over a shared
  cache root; cache_stats None without a cache; pipeline logs the exact
  line; no line when cache absent/disabled) -> 411 passed; ruff clean.
Records the P4-1 follow-up: TushareFeed.cache_stats() + the run-log line
`data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=M`
(cold nonzero, warm historical rerun 0/0 — confirmed in
run_phase2_baseline.log, secret scan 0). Gate count 407 -> 411 across
CLAUDE.md / AGENTS.md / TEST_REPORT.md; RUNBOOK P4-1 read-through bullets
mention the visible-hit-rate line.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant