P4-2: persistent cache for universe + tradability endpoints

AGENTS.md

@@ -166,6 +166,19 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri
     - secret scan：缓存 parquet + ledger 0 处 token / `.config.json`；ledger 列只有端点元数据。
   - **缓存命中直接可见（review follow-up）**：`TushareFeed.cache_stats()` 暴露各端点 gap-fetch 计数，`_load_panel` 经 run-scoped logger 打 `data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=M`——冷跑非零、暖跑 0/0（warm rerun 实证 run_phase2_baseline.log 含 `0/0`，secret scan 0）。
 - **不变量守住**：factor/alpha/portfolio/execution/OOS 切片/report 全不动；`artifacts/data/{output_name}.parquet` 不当 SoT。范围克制：P4-1 只 market_daily+adj_factor，其余端点（index_weight/daily_basic/fina_indicator/...）P4-2/P4-3 再缓存。
-- ✅ 质量门：`pytest` **411 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16 / P3-4=15 / P3-5=22 / P3-6=27 / P3-7=25+1 throttle / P3-8=8 / P4-1=28）；`ruff` clean；`validate-config`（全部 12 配置）+ `run-phase0`（demo）均 OK。
+- ✅ **Phase 4-2 持久化 Tushare universe + tradability 缓存**（**分支 `data-cache-universe-tradability`**，index_weight + suspend_d + namechange + stk_limit + stock_basic）：把 P4-1 端点级 raw 缓存扩到 universe/可交易性端点——真实 run 不再每次重抓成分股/停牌/ST/涨跌停/上市日。**默认仍 disabled（向后兼容，旧配置行为一字不变）**；opt-in 后这五端点走 read-through。
+  - **三种规划形态共用一引擎**：① dense per-symbol 日期区间（`suspend_d`/`stk_limit`，复用 P4-1 gap + recent-tail）；② index_code 维日期区间（`index_weight`，coverage key=index_code，gap 内仍 **90 天分页**，raw 快照入库）；③ snapshot 维度（`namechange` per-symbol、`stock_basic` 全局 sentinel），用 `refresh_dimension_days`（默认 30）staleness + force_refresh（`CoverageLedger.snapshot_fetched_at` 给新鲜度判定）。
+  - **语义全保**：`index_weight` 仍 PIT/as-of（370 天 pre-start lookback 进缓存请求区间，latest-snapshot-on-or-before 仍在 feed/universe）；`stk_limit` 仍 **raw price**（限价检查在 front-adjust 前，不碰 qfq）；`stock_basic` 只取 list_date 供 `min_listing_days`（缺失仍 kept+披露），current-tag `industry` 绝不入缓存/中性化；`suspend_d`/`namechange`/ST 区间形状不变；**缓存只存 raw 端点事实，不存派生 flag 作 SoT**。
+  - **三 feed 接线**：`IndexConstituentsFeed`/`TushareFlagsFeed`/`TushareCovariatesFeed` 各加 `cache=None` 注入；cache present → read-through + **共享 finalizer**（cached==direct，限价 frame `assert_frame_equal`、ST 区间集合相等、suspend set / listing dict 相等单测锁定）；`cache=None` → 直抓路径**逐字不变**（旧 feed 测试原样过）。per-symbol 限流/重试仍在各 feed 的 `_call` 闭包里（缓存 transport-agnostic）。
+  - **coverage ledger 复用 + 扩展**：11 列不变；**empty 算覆盖、failed 不算**（fetch 抛错则不记 coverage、留待重试，测试锁定）；ledger 列只有端点元数据，无 token/secret。
+  - **单一共享 cache 贯穿 4 runner**（run_phase0/phase2/oos/subset）：`_build_cache(cfg)` 建一个实例线穿 `_build_universe`/`_load_panel`/`_enrich_tradability`/`_maybe_enrich_listing`，跑完所有缓存端点后 `_log_run_cache_stats` 打**一行** 7 端点统计；P4-1 的 market 行前缀保留（旧统计测试过）。
+  - **配置**：`data.cache.refresh_dimension_days`（默认 30，>=0 校验）；旧配置全 validate（含未知键拒绝）。
+  - **真实 smoke（phase2 baseline，fresh temp root，cold→warm；fresh root 不动已 merge 的 v1 缓存）**：
+    - **cold 行**：`market_daily=68 adj_factor=68 index_weight=9 suspend_d=68 namechange=68 stk_limit=68 stock_basic=1`（全非零）；coverage = market/adj/namechange/stk_limit 各 68 ok + index_weight **1 ok**（整 gap 一行，内部 9 窗分页）+ stock_basic 1 ok + **suspend_d 68 empty**（SSE50 大盘股该窗口无停牌 → 空返回算覆盖，warm 不重抓）。
+    - **warm 行：7 端点全 0**；coverage ledger **零新行**；report 指标与 cold/P4-1 cached baseline **逐数一致**（IC 0.0083 / annual −10.19% / maxDD −16.52% / vol 16.59% / sharpe −0.5703 / turnover 1.0818 / cost 1.19%）——cached==direct 二度端到端实证。
+    - wall：cold **960s** / warm **366s**（暖跑省 ~594s = universe+tradability+market 抓取；`daily_basic`(market_cap)/`index_member_all`(pit_sw) 仍 live——P4-2 不缓存这俩，P4-3 再说，诚实标注）。
+    - secret scan：缓存 parquet + ledger + 日志 + 报告 0 处 token 值 / `.config.json`；ledger 无 token 列。
+  - **不变量守住**：factor/alpha/portfolio/execution/OOS 切片/report/`front_adjust` 全不动；`artifacts/data/*.parquet` 不当 SoT。范围克制：`daily_basic`/`fina_indicator`/`index_member_all` 仍留 P4-3。
+- ✅ 质量门：`pytest` **428 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16 / P3-4=15 / P3-5=22 / P3-6=27 / P3-7=25+1 throttle / P3-8=8 / P4-1=28 / P4-2=17）；`ruff` clean；`validate-config`（全部 12 配置）+ `run-phase0`（demo）均 OK。
 - ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、旧三因子无信号（P3-3/P3-4 实证;但其组合在 2024-2026 holdout 上 SSE50/CSI300/CSI500 全正、CSI500 高达 +17.8%——小样本/regime 翻转的持续例证）;value/低波信号获得**独立样本符号级确认**（P3-7 SSE50/CSI300 量级衰减;P3-8 CSI500 泛化成立且更强），组合级盈利能力仍未确立（排名跨 cell 翻转）;subset 报告文件名已可配置（P3-8 起不再互覆盖）。
-- 路线图下一步：**P4-2 universe/tradability 缓存**（index_weight/suspend_d/namechange/stk_limit/stock_basic）→ **P4-3 因子支撑端点缓存**（fina_indicator/daily_basic/index_member_all）；研究侧：更长 holdout 滚动复检 / 中证1000 / 成本模型细化,或分钟级（architecture.html §11）。
+- 路线图下一步：**P4-3 因子支撑端点缓存**（fina_indicator/daily_basic/index_member_all）——暖跑剩余 live 抓取即这三端点；研究侧：更长 holdout 滚动复检 / 中证1000 / 成本模型细化,或分钟级（architecture.html §11）。

CLAUDE.md

@@ -166,6 +166,19 @@ data → universe → factors(特征) → alpha(合成/预测) → portfolio(+ri
     - secret scan：缓存 parquet + ledger 0 处 token / `.config.json`；ledger 列只有端点元数据。
   - **缓存命中直接可见（review follow-up）**：`TushareFeed.cache_stats()` 暴露各端点 gap-fetch 计数，`_load_panel` 经 run-scoped logger 打 `data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=M`——冷跑非零、暖跑 0/0（warm rerun 实证 run_phase2_baseline.log 含 `0/0`，secret scan 0）。
 - **不变量守住**：factor/alpha/portfolio/execution/OOS 切片/report 全不动；`artifacts/data/{output_name}.parquet` 不当 SoT。范围克制：P4-1 只 market_daily+adj_factor，其余端点（index_weight/daily_basic/fina_indicator/...）P4-2/P4-3 再缓存。
-- ✅ 质量门：`pytest` **411 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16 / P3-4=15 / P3-5=22 / P3-6=27 / P3-7=25+1 throttle / P3-8=8 / P4-1=28）；`ruff` clean；`validate-config`（全部 12 配置）+ `run-phase0`（demo）均 OK。
+- ✅ **Phase 4-2 持久化 Tushare universe + tradability 缓存**（**分支 `data-cache-universe-tradability`**，index_weight + suspend_d + namechange + stk_limit + stock_basic）：把 P4-1 端点级 raw 缓存扩到 universe/可交易性端点——真实 run 不再每次重抓成分股/停牌/ST/涨跌停/上市日。**默认仍 disabled（向后兼容，旧配置行为一字不变）**；opt-in 后这五端点走 read-through。
+  - **三种规划形态共用一引擎**：① dense per-symbol 日期区间（`suspend_d`/`stk_limit`，复用 P4-1 gap + recent-tail）；② index_code 维日期区间（`index_weight`，coverage key=index_code，gap 内仍 **90 天分页**，raw 快照入库）；③ snapshot 维度（`namechange` per-symbol、`stock_basic` 全局 sentinel），用 `refresh_dimension_days`（默认 30）staleness + force_refresh（`CoverageLedger.snapshot_fetched_at` 给新鲜度判定）。
+  - **语义全保**：`index_weight` 仍 PIT/as-of（370 天 pre-start lookback 进缓存请求区间，latest-snapshot-on-or-before 仍在 feed/universe）；`stk_limit` 仍 **raw price**（限价检查在 front-adjust 前，不碰 qfq）；`stock_basic` 只取 list_date 供 `min_listing_days`（缺失仍 kept+披露），current-tag `industry` 绝不入缓存/中性化；`suspend_d`/`namechange`/ST 区间形状不变；**缓存只存 raw 端点事实，不存派生 flag 作 SoT**。
+  - **三 feed 接线**：`IndexConstituentsFeed`/`TushareFlagsFeed`/`TushareCovariatesFeed` 各加 `cache=None` 注入；cache present → read-through + **共享 finalizer**（cached==direct，限价 frame `assert_frame_equal`、ST 区间集合相等、suspend set / listing dict 相等单测锁定）；`cache=None` → 直抓路径**逐字不变**（旧 feed 测试原样过）。per-symbol 限流/重试仍在各 feed 的 `_call` 闭包里（缓存 transport-agnostic）。
+  - **coverage ledger 复用 + 扩展**：11 列不变；**empty 算覆盖、failed 不算**（fetch 抛错则不记 coverage、留待重试，测试锁定）；ledger 列只有端点元数据，无 token/secret。
+  - **单一共享 cache 贯穿 4 runner**（run_phase0/phase2/oos/subset）：`_build_cache(cfg)` 建一个实例线穿 `_build_universe`/`_load_panel`/`_enrich_tradability`/`_maybe_enrich_listing`，跑完所有缓存端点后 `_log_run_cache_stats` 打**一行** 7 端点统计；P4-1 的 market 行前缀保留（旧统计测试过）。
+  - **配置**：`data.cache.refresh_dimension_days`（默认 30，>=0 校验）；旧配置全 validate（含未知键拒绝）。
+  - **真实 smoke（phase2 baseline，fresh temp root，cold→warm；fresh root 不动已 merge 的 v1 缓存）**：
+    - **cold 行**：`market_daily=68 adj_factor=68 index_weight=9 suspend_d=68 namechange=68 stk_limit=68 stock_basic=1`（全非零）；coverage = market/adj/namechange/stk_limit 各 68 ok + index_weight **1 ok**（整 gap 一行，内部 9 窗分页）+ stock_basic 1 ok + **suspend_d 68 empty**（SSE50 大盘股该窗口无停牌 → 空返回算覆盖，warm 不重抓）。
+    - **warm 行：7 端点全 0**；coverage ledger **零新行**；report 指标与 cold/P4-1 cached baseline **逐数一致**（IC 0.0083 / annual −10.19% / maxDD −16.52% / vol 16.59% / sharpe −0.5703 / turnover 1.0818 / cost 1.19%）——cached==direct 二度端到端实证。
+    - wall：cold **960s** / warm **366s**（暖跑省 ~594s = universe+tradability+market 抓取；`daily_basic`(market_cap)/`index_member_all`(pit_sw) 仍 live——P4-2 不缓存这俩，P4-3 再说，诚实标注）。
+    - secret scan：缓存 parquet + ledger + 日志 + 报告 0 处 token 值 / `.config.json`；ledger 无 token 列。
+  - **不变量守住**：factor/alpha/portfolio/execution/OOS 切片/report/`front_adjust` 全不动；`artifacts/data/*.parquet` 不当 SoT。范围克制：`daily_basic`/`fina_indicator`/`index_member_all` 仍留 P4-3。
+- ✅ 质量门：`pytest` **428 passed**（P0=97 / P1=78 / P2-1=22 / P2-2=22 / P2-3=14 / P2-4=8 / P3-1=10 / P3-2=18 / P3-3=16 / P3-4=15 / P3-5=22 / P3-6=27 / P3-7=25+1 throttle / P3-8=8 / P4-1=28 / P4-2=17）；`ruff` clean；`validate-config`（全部 12 配置）+ `run-phase0`（demo）均 OK。
 - ⚠️ 剩余（已显式披露）：日线 only、demo 路径非真数据、旧三因子无信号（P3-3/P3-4 实证;但其组合在 2024-2026 holdout 上 SSE50/CSI300/CSI500 全正、CSI500 高达 +17.8%——小样本/regime 翻转的持续例证）;value/低波信号获得**独立样本符号级确认**（P3-7 SSE50/CSI300 量级衰减;P3-8 CSI500 泛化成立且更强），组合级盈利能力仍未确立（排名跨 cell 翻转）;subset 报告文件名已可配置（P3-8 起不再互覆盖）。
-- 路线图下一步：**P4-2 universe/tradability 缓存**（index_weight/suspend_d/namechange/stk_limit/stock_basic）→ **P4-3 因子支撑端点缓存**（fina_indicator/daily_basic/index_member_all）；研究侧：更长 holdout 滚动复检 / 中证1000 / 成本模型细化,或分钟级（architecture.html §11）。
+- 路线图下一步：**P4-3 因子支撑端点缓存**（fina_indicator/daily_basic/index_member_all）——暖跑剩余 live 抓取即这三端点；研究侧：更长 holdout 滚动复检 / 中证1000 / 成本模型细化,或分钟级（architecture.html §11）。

TEST_REPORT.md

@@ -26,12 +26,12 @@ Run from the repo root with the project python (env `quant_mf`):
 
 | Gate | Command | Result |
 |---|---|---|
-| Unit + integration | `pytest -q` | **411 passed, 0 failed** |
+| Unit + integration | `pytest -q` | **428 passed, 0 failed** |
 | Lint | `ruff check .` | **All checks passed** |
 | Config validation | `validate-config` (demo + `example_tushare.yaml` + `phase2_real_baseline.yaml` + `phase3_real_multifactor.yaml` + `phase3_real_ic_weighted.yaml` + `phase3_real_oos_stability.yaml` + `phase3_real_robustness_matrix.yaml` + `phase3_real_factor_candidates.yaml` + `phase3_real_subset_costs.yaml` + `phase3_real_independent_validation.yaml` + `phase3_real_csi500_generalization.yaml` + `phase2_real_baseline_cached.yaml`) | exit `0`, prints `OK` |
 | End-to-end run | `run-phase0` (demo) | exit `0`, writes `artifacts/reports/phase0_summary.md` |
 
-Counts below are the actual per-file `pytest` numbers (sum = 411).
+Counts below are the actual per-file `pytest` numbers (sum = 428).
 
 ## Per-file breakdown — Phase 0 core (97)
 
@@ -158,13 +158,19 @@ Counts below are the actual per-file `pytest` numbers (sum = 411).
 | `test_csi500_generalization.py` | 8 | CSI500 config validates with the expected cell roles (screened anchor SSE50|2022-2024; independent SSE50|2024-2026 + 000905.SH|2024-2026; CSI500|2022-2024 skipped+disclosed; same groups/scenarios/hypotheses as P3-7 — no tuning); sample classes labeled correctly; **`output.subset_report_name`** lets each subset-validation study own its report file (default None keeps the historical `phase3_subset_validation.md` bitwise — the P3-6/P3-7 configs are locked unchanged), so a P3-8 run never clobbers the accepted P3-7 artifact; **`output.subset_report_title`** config-drives the report H1 so the CSI500 study names itself ("Phase 3-8 — CSI500 Independent Generalization Check", asserted NOT to start with Phase 3-7) while the P3-6/P3-7 configs leave it unset and keep the renderer's sample-aware default title (regression-locked) |
 | **Total through P3-8** | **383** | |
 
-## Per-file breakdown — Phase 4-1 persistent market cache (24)
+## Per-file breakdown — Phase 4-1 persistent market cache (28)
 
 | Test file | Tests | Red-line / feature |
 |---|---|---|
 | `test_cache_config.py` | 17 | `data.cache` defaults disabled (backward compatible); fields + defaults; rejects negative refresh window / empty root / unknown key; **every existing config still validates** (parametrized over all `config/*.yaml`) |
 | `test_tushare_cache_market.py` | 11 | interval subtraction (full/partial/none); **full miss populates daily+adj cache, full hit = ZERO endpoint calls**; partial gap fetches only the missing tail; empty endpoint return still records coverage (no refetch); duplicate upsert keeps one row per (symbol,date); **cached panel == direct-fetch panel byte-for-byte after `front_adjust()`**; no token / secret-path in any cache file or ledger column; **`TushareFeed.cache_stats()` exposes cold {2,2}→warm {0,0} gap-fetch counts** and `pipeline._log_cache_stats` logs `data cache: market_daily_gap_fetches=N adj_factor_gap_fetches=M` through the run-scoped logger (review follow-up: warm-hit evidence is now directly visible in the run log) |
-| **Total (P0 + P1 + P2-1..P2-4 + P3-1..P3-8 + P4-1)** | **411** | |
+
+## Per-file breakdown — Phase 4-2 universe + tradability cache (17)
+
+| Test file | Tests | Red-line / feature |
+|---|---|---|
+| `test_tushare_cache_universe.py` | 17 | for each new endpoint (`index_weight`/`suspend_d`/`namechange`/`stk_limit`/`stock_basic`): **cold miss populates, warm rerun on a fresh cache over the same root = ZERO endpoint calls**; **cached feed output == direct feed output** (index_weight `assert_frame_equal` + PIT cross-section, suspend set, ST-interval sets incl. open `None` end, stk_limit frame in RAW price terms, listing-date dict); empty endpoint/snapshot return records coverage (no refetch); **FAILED fetch records NO coverage** (a later good fetch retries) — dense + snapshot; duplicate upsert keeps one row per natural key (index_weight `(date,symbol)`, suspend_d `(date,symbol,suspend_type)`); `force_refresh` re-pulls a fresh dimension snapshot; **cache-disabled path never touches the cache tree**; `pipeline._format_cache_stats` / `_log_run_cache_stats` name all five new endpoints in one line (None → no line); no token / secret-path in any cache file or ledger column across all five endpoints |
+| **Total (P0 + P1 + P2-1..P2-4 + P3-1..P3-8 + P4-1 + P4-2)** | **428** | |
 
 ## Real-data validation (manual, not in CI — TEST-002 keeps the suite network-free)
 
@@ -195,6 +201,21 @@ Counts below are the actual per-file `pytest` numbers (sum = 411).
   ledger unchanged between cold and warm); cache files + ledger carry no token /
   secret. Market bars only — index/financial endpoints still fetch live (P4-2/3). Two earlier run attempts died
   on transient ConnectionError → default retry budget hardened 3→6 attempts.
+- **P4-2** universe + tradability cache (real smoke = phase2 baseline on a fresh
+  temp cache root, cold → warm; the merged P4-1 `v1` cache left untouched):
+  **cold** run-log line `data cache: market_daily=68 adj_factor=68
+  index_weight=9 suspend_d=68 namechange=68 stk_limit=68 stock_basic=1` (all
+  non-zero); coverage = 68 ok each for market/adj/namechange/stk_limit, 1 ok for
+  index_weight (one gap, paged in 9 windows) and stock_basic, and **68 empty for
+  suspend_d** (SSE50 large-caps had no suspensions in the window — empty-as-
+  coverage). **warm** line shows **all seven endpoints = 0**, coverage ledger
+  unchanged (zero new rows), and report metrics byte-identical to cold and to the
+  P4-1 cached baseline (IC 0.0083 / annual −10.19% / maxDD −16.52% / vol 16.59% /
+  sharpe −0.5703 / turnover 1.0818 / cost 1.19%). Wall: cold 960s → warm 366s
+  (the ~594 s saved is the universe + tradability + market fetches now served from
+  cache; `daily_basic` / `index_member_all` still fetch live — P4-3). Secret scan:
+  0 token-value / `.config.json` occurrences across cache parquet, ledger, run
+  log, and report.
 
 ## Notes
 

data/cache/coverage.py

@@ -92,6 +92,27 @@ def covered_intervals(self, endpoint: str, key: str) -> list[Interval]:
             for s, e in zip(sub["start_date"], sub["end_date"])
         ]
 
+    def snapshot_fetched_at(self, endpoint: str, key: str) -> pd.Timestamp | None:
+        """Latest successful fetch time for a snapshot/dimension ``(endpoint, key)``.
+
+        For the dimension endpoints (``stock_basic``, ``namechange``) coverage is
+        not a date range but a freshness timestamp: the most recent ``ok``/
+        ``empty`` ``fetched_at``. ``None`` means never fetched (must fetch). The
+        caller compares against ``refresh_dimension_days`` to decide staleness.
+        """
+        ledger = self.read()
+        if ledger.empty:
+            return None
+        mask = (
+            (ledger["endpoint"] == endpoint)
+            & (ledger["key"] == str(key))
+            & (ledger["status"].isin(_COVERING_STATUSES))
+        )
+        sub = ledger[mask]
+        if sub.empty:
+            return None
+        return pd.Timestamp(sub["fetched_at"].max())
+
     # -- write -------------------------------------------------------------- #
     def record(
         self,