Skip to content

docs/refactor: document data-layer contracts and dedupe token parsing (D1) - #41

Merged
StackOverFlow11 merged 2 commits into
mainfrom
feat/data-layer-d1-contracts
Jun 19, 2026
Merged

docs/refactor: document data-layer contracts and dedupe token parsing (D1)#41
StackOverFlow11 merged 2 commits into
mainfrom
feat/data-layer-d1-contracts

Conversation

@StackOverFlow11

Copy link
Copy Markdown
Owner

Summary

Data Layer D1 — the first low-risk data-layer cleanup. Two things only:

  1. a committed doc making the cache vs store boundary explicit;
  2. removing the stale duplicated token parsing in TushareFeed and
    IndexConstituentsFeed by routing both onto the existing
    data/feed/secret.py::read_token (the four sibling feeds already use it).

Behavior-preserving: no data-fetch, cache, factor, alpha, portfolio, runtime,
or report-metric behavior changes.

Changed files (4)

  • new docs/data/data_layer_contracts.md — states: data/cache is the
    reusable raw endpoint cache / source of truth; data/store/PanelStore is a
    per-run canonical panel artifact (not the raw lake); durable cache rows stay
    raw only; front_adjust / PIT index membership / PIT SW industry / ann_date
    financial alignment / raw price-limit checks / factors/alpha/portfolio/runtime
    math all remain downstream; data-update warms caches only; data-quality and
    concurrency are later phases (not claimed implemented).
  • mod data/feed/tushare_feed.py — drop the duplicated module-level
    _lookup_dotted and the inline json/Path parsing; _read_token is now a
    one-line wrapper over read_token(self._secret_file, self._token_key).
  • mod data/feed/index_feed.py — stop importing the private _lookup_dotted
    from tushare_feed.py; _client reads the token via read_token directly.
  • mod tests/test_index_feed.py — +2 narrow tests exercising the index
    feed's real _client() token path with a fake tushare.pro_api (token sourced
    from a fake external config, never leaked; missing-key error stays readable).
    No test reads the real .config.json; no test hits the network.

_lookup_dotted is now fully removed repo-wide; all six tushare-backed feeds
(feed/index/flags/fina/intraday/covariates) source the token through
secret.read_token.

Verification (rebased onto current main)

  • pytest -p no:cacheprovider579 passed (current main baseline 577 + 2 new
    index-feed tests; rose from Codex's earlier 540 only because the branch now sits
    on newer main, not because D1 changed).
  • ruff check .All checks passed!
  • validate-config over all 17 config/*.yaml → all OK.
  • run-phase0 --config config/example.yaml
    ic_mean=0.9600, annual_return=0.8408 (anchor unchanged — no behavior drift).
  • git diff --check main...HEAD → clean; merge-marker scan → 0.
  • Secret scan over changed files: real .config.json token value 0 hits;
    FAKE_TUSHARE_TOKEN… only in the test fixture; .config.json/tushare.token
    only as config-key/scan-target descriptions.

Explicit non-goals (deferred to D2+)

  • no TushareCache internal split;
  • no endpoint schema registry;
  • no data-quality validator;
  • no concurrency / thread pool / async fetching;
  • no CoverageLedger storage change;
  • no PanelStore append/partition feature;
  • no factor/alpha/portfolio/runtime/report metric changes.

Add docs/data/data_layer_contracts.md stating the boundary invariants:
- data/cache is the reusable raw endpoint cache / source of truth;
- data/store/PanelStore is a per-run canonical panel artifact, NOT the
  raw data lake;
- durable cache rows stay raw only; front_adjust, PIT index membership,
  PIT SW industry, ann_date financial alignment, raw price-limit checks,
  factors/alpha/portfolio/runtime math all remain downstream;
- data-update warms caches only and never runs factors/backtests;
- data-quality checks and concurrency are later phases (NOT claimed
  implemented in D1).
TushareFeed and IndexConstituentsFeed each carried their own copy of the
external-config JSON parsing (and IndexConstituentsFeed imported a private
_lookup_dotted from tushare_feed). Route both onto data/feed/secret.read_token,
matching the four sibling feeds (flags/fina/intraday/covariates).

- tushare_feed: drop the duplicated module-level _lookup_dotted and the
  inline json/Path parsing; _read_token is now a one-line wrapper over
  read_token(self._secret_file, self._token_key).
- index_feed: stop importing _lookup_dotted from tushare_feed; _client reads
  the token via read_token directly.

Public constructor signatures, repr() token safety, and fetch/cache behavior
are unchanged. Add two narrow tests exercising IndexConstituentsFeed's real
_client() token path with a fake tushare.pro_api (token sourced from a fake
external config, never leaked; missing-key error stays readable).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant