feat(data): all-A historical backfill (chunked, resumable, failure-tolerant)

config/data_update_all_a.yaml

@@ -116,6 +116,29 @@ data_update:
   # per thread; store + ledger writes stay deterministic.
   concurrency:
     max_workers: 4
+  # PR-2 historical backfill knobs — consumed ONLY by the SEPARATE, manual
+  #   python -m qt.cli data-backfill --config config/data_update_all_a.yaml
+  # command. The nightly `data-update` job IGNORES this block entirely.
+  #
+  # Backfill warms the caches over the WIDE window [backfill.start, today] (the
+  # full history), NOT the incremental `today - lookback_days` tail. It processes
+  # the universe in per-symbol batches of `chunk_size`, tolerates a per-batch
+  # failure (logged + skipped + retryable on re-run), and is resumable via the
+  # coverage ledgers (a re-run fetches only still-uncovered gaps — safe to Ctrl-C
+  # and re-run). Run it BEFORE enabling the nightly timer: backfill history first,
+  # then keep it current nightly.
+  #
+  # ⚠️ HONEST COST — a COLD all-A MINUTE backfill is a LONG, MANY-HOUR (potentially
+  # multi-DAY) operation. At all-A scale (~5500 symbols × ~5 years of 1min bars,
+  # paged ≤23 trading days / ≤8000 rows per call) `include_minute: true` is by far
+  # the most expensive path here — tens of thousands of stk_mins calls, run against
+  # the rate limit. It is resumable, so run it in stages and re-run to finish; if
+  # you only need daily/universe/financial history first, set include_minute: false
+  # for a much shorter run and backfill minutes separately.
+  backfill:
+    start: "2020-01-01"     # WIDE backfill window start (window = [start, today])
+    chunk_size: 300         # per-batch symbol count (durable + failure-isolated)
+    include_minute: true    # also backfill 1min history (see HONEST COST above)
 
 # --- unused by data-update (schema-only stubs) ------------------------------- #
 factors:

docs/ops/data_update_schedule.md

@@ -136,6 +136,53 @@ symbols this becomes statistically likely on any given night. Important properti
 - **Per-symbol failure isolation is a planned PR-2 hardening** (changing the fail-fast
   behavior is a warm-path semantic change, deliberately out of scope for PR-1).
 
+## Historical backfill (PR-2) — run FIRST, then keep current
+
+The nightly `data-update` job only tops up the moving front edge (`lookback_days`
+relative to *today*) and, for minutes, a 7-day tail. It does **not** load deep
+history. To populate the caches with the full historical window, use the separate,
+manual **`data-backfill`** command:
+
+```bash
+cd /home/shaofl/Projects/financial_projects/stocks_market/Quantitative_Trading
+/home/shaofl/Development/env_tools/envs/quant_mf/bin/python -m qt.cli \
+  data-backfill --config config/data_update_all_a.yaml
+```
+
+It reuses the SAME caches, feeds, universe resolution, and rate limiter as
+`data-update`, but:
+
+- **Wide window.** It warms `[backfill.start, today]` (the full history), NOT the
+  incremental `today - lookback_days` tail.
+- **Chunked.** Symbols are processed in per-symbol batches of
+  `data_update.backfill.chunk_size` (default 300), so progress is durable
+  batch-by-batch and memory stays bounded.
+- **Full minute history (optional).** With `data_update.backfill.include_minute:
+  true` it warms 1min bars over the whole `[backfill.start, today]` window — NOT
+  the nightly 7-day tail. Set it to `false` to skip minute history for a much
+  shorter run.
+- **Per-symbol / per-batch failure-tolerant.** Unlike the nightly job (which stays
+  fail-fast on purpose), a batch whose fetch fails persistently (after the feeds'
+  retries) is **logged (secret-free: batch index + exception type only) and
+  skipped**; the run CONTINUES to the next batch. The failed batch's gaps stay
+  uncovered (records no coverage), so they are retried on the next run. The
+  `OK data-backfill: …` summary tallies `failed_batches` and lists failed symbols
+  (bounded).
+- **Resumable.** Resumability is inherent to the coverage ledgers — a re-run over
+  the same window fetches only still-uncovered gaps. It is **safe to Ctrl-C and
+  re-run**; durable successes before an interruption are already persisted.
+
+⚠️ **This is a LONG, MANY-HOUR (potentially multi-day), manual run** — especially
+with `include_minute: true` at all-A scale (~5500 symbols × years of 1min bars).
+Run it in stages and re-run to finish. It never runs factors / alpha / portfolio /
+backtest / PanelStore, and (like `data-update`) stores RAW endpoint facts only —
+no token, no qfq, no derived flag.
+
+**Order of operations:** backfill the history FIRST (this command), confirm the
+summary looks sane, and only THEN enable the nightly incremental timer above so it
+keeps the caches current. The backfill window `[start, today]` is fixed, so a
+re-run after enabling the timer stays warm except for still-uncovered gaps.
+
 ## Notes
 
 - Scope is set by `data_update.universe_scope: all_a` in the config; the default

qt/cli.py

@@ -229,6 +229,24 @@ def _cmd_data_update(args: argparse.Namespace) -> int:
     return 0
 
 
+def _cmd_data_backfill(args: argparse.Namespace) -> int:
+    """Backfill the tushare caches over the WIDE historical window (PR-2); no backtest."""
+    from qt.data_backfill import format_summary, run_data_backfill
+
+    try:
+        result = run_data_backfill(args.config)
+    except (ConfigError, ValueError, FileNotFoundError) as exc:
+        print(f"ERROR: {exc}", file=sys.stderr)
+        return 1
+    print(
+        f"OK data-backfill: {result.universe_size} symbols in "
+        f"{result.n_batches} batch(es), failed_batches={result.failed_batches} "
+        f"({result.elapsed_seconds:.1f}s)\n"
+        f"{format_summary(result)}"
+    )
+    return 0
+
+
 def _cmd_fetch_data(args: argparse.Namespace) -> int:
     """Stage helper: run the spine and report the data-fetch stage."""
     return _run_pipeline_cmd(args.config, "fetch-data")
@@ -295,6 +313,13 @@ def build_parser() -> argparse.ArgumentParser:
     p_du.add_argument("--config", required=True, help="Path to the YAML config.")
     p_du.set_defaults(func=_cmd_data_update)
 
+    p_bf = sub.add_parser(
+        "data-backfill",
+        help="Backfill the tushare raw caches over the WIDE history (PR-2); no backtest.",
+    )
+    p_bf.add_argument("--config", required=True, help="Path to the YAML config.")
+    p_bf.set_defaults(func=_cmd_data_backfill)
+
     p_i5a = sub.add_parser(
         "run-phase-i5a-intraday",
         help="Run the I5a intraday tail-rebalance architecture smoke (minute cache).",

qt/config.py

@@ -718,6 +718,55 @@ def _check_report_name(cls, v: str) -> str:
         return name
 
 
+class BackfillCfg(_Strict):
+    """PR-2 historical backfill knobs for ``data-backfill`` (all defaults => a
+    no-op addition: every existing config still validates unchanged).
+
+    ``data-backfill`` is a SEPARATE, manual, long-running command from the nightly
+    incremental ``data-update``; it warms the SAME caches over a WIDE window.
+
+    * ``start`` — the WIDE backfill window start. The window is ``[start, today]``,
+      NOT the incremental ``today - lookback_days`` tail (validated as a date).
+    * ``chunk_size`` — symbols are processed in batches of this size so progress is
+      durable batch-by-batch, memory stays bounded, and a per-batch failure is
+      isolated (must be >= 1).
+    * ``include_minute`` — also backfill 1min bars over the FULL window (a LONG,
+      resumable operation at all-A scale). ``false`` skips minute history.
+    """
+
+    start: str = "2020-01-01"
+    chunk_size: int = 300
+    include_minute: bool = True
+
+    @field_validator("start", mode="before")
+    @classmethod
+    def _coerce_date_to_str(cls, v: Any) -> Any:
+        if isinstance(v, (_date, datetime)):
+            return v.strftime("%Y-%m-%d")
+        return v
+
+    @field_validator("start")
+    @classmethod
+    def _check_start(cls, v: str) -> str:
+        try:
+            datetime.strptime(v, "%Y-%m-%d")
+        except ValueError as exc:
+            raise ValueError(
+                f"data_update.backfill.start must be a 'YYYY-MM-DD' date; "
+                f"got {v!r} ({exc})."
+            ) from exc
+        return v
+
+    @field_validator("chunk_size")
+    @classmethod
+    def _check_chunk_size(cls, v: int) -> int:
+        if v < 1:
+            raise ValueError(
+                f"data_update.backfill.chunk_size must be >= 1; got {v}."
+            )
+        return v
+
+
 class DataUpdateConcurrencyCfg(_Strict):
     """D5 opt-in bounded concurrency for ``data-update`` cache warms (default serial).
 
@@ -775,6 +824,9 @@ class DataUpdateCfg(_Strict):
     concurrency: DataUpdateConcurrencyCfg = Field(
         default_factory=DataUpdateConcurrencyCfg
     )
+    # PR-2 historical backfill knobs (consumed ONLY by the separate data-backfill
+    # command; the nightly data-update ignores them). All-defaults => unchanged.
+    backfill: BackfillCfg = Field(default_factory=BackfillCfg)
 
     @field_validator("endpoints", "force_refresh")
     @classmethod