Skip to content

feat(data): D3b default-off data-update quality-report hook - #47

Merged
StackOverFlow11 merged 1 commit into
mainfrom
feat/data-layer-d3b-data-update-quality-report
Jun 19, 2026
Merged

feat(data): D3b default-off data-update quality-report hook#47
StackOverFlow11 merged 1 commit into
mainfrom
feat/data-layer-d3b-data-update-quality-report

Conversation

@StackOverFlow11

Copy link
Copy Markdown
Owner

Summary

Data Layer D3b — a small, default-off quality-report hook that surfaces the already-accepted D3 data/quality/ checks in the data-update job. Report-only; not D4.

With data_update.quality.enabled=false (the default) every existing config behaves exactly as before. When enabled, the updater runs the D3 structural checks on the frames it already warmed (no extra API call) and writes a deterministic Markdown report.

What changed

  • config (qt/config.py, config/data_update.yaml): nested DataUpdateQualityCfg under data_update.qualityenabled=false; endpoints restricted to the structural surfaces the updater materializes as frames (market_daily / adj_factor / stk_mins_1min, unknown rejected readably); report_name validated as a bare filename under output.report_dir (no separators / .. / absolute). The YAML gains the disabled block with comments.
  • hook (qt/data_update_quality.py, new): collect_findings runs the D3 checks for the selected-AND-warmed endpoints on the already-loaded frames; write_quality_report renders the deterministic D3 report (bounded, redacted examples) plus a non-secret run-context block (window / symbol count / endpoint names), written even when clean. No exchange calendar synthesized → structural checks only.
  • updater (qt/data_updater.py): update_endpoints gains an optional capture sink (default None ⇒ byte-identical warm path — same calls, args, order, summary); run_data_update runs the hook only when enabled; UpdateResult gains quality_report_path / quality_findings_count / quality_hard_count.
  • cli (qt/cli.py): data-update prints the report path + counts only when enabled; disabled output materially unchanged.

Invariants held

Report-only: never filters / repairs / mutates data, never fails the job, never changes cache coverage or the per-endpoint request summary, no new Tushare endpoint, no extra live call for quality, no PanelStore write, no ledger schema change. Daily close-to-close behavior and the phase0 anchor (ic_mean=0.9600, annual_return=0.8408) unchanged.

Out of scope (not started): D4 / CoverageLedger storage scaling / batch writes / concurrency / schema registry; the optional fail_on_hard knob.

Tests / gates

  • tests/test_data_update_quality.py (network-free, fake feeds): default disabled; clean report; bad daily+intraday frames → hard findings without raising or altering the summary; endpoint selection honored (not-warmed surface never checked); report_name path validation; report carries no secret path / key / token value (D3 redaction inherited); capture never changes the summary.
  • Full suite 643 passed (was 625; +18). ruff clean. All 17 configs validate-config. run-phase0 anchor 0.9600 / 0.8408. git diff --check clean; no merge markers. Secret scan over changed files: real token value 0 hits (the only tushare.token / .config.json literals are the pre-existing config key/path defaults + deliberate redaction-test scan targets).

Live data-update against Tushare was not run (acceptance is network-free per the goal).

Surface the accepted D3 data/quality checks in operations via an opt-in,
report-only hook on the data-update job. Default OFF: every existing config
behaves exactly as before.

- config: nested DataUpdateQualityCfg under data_update.quality
  (enabled=false; endpoints restricted to the structural surfaces the updater
  loads as frames — market_daily/adj_factor/stk_mins_1min; report_name must be
  a bare filename under output.report_dir). config/data_update.yaml gains the
  disabled block.
- hook: new qt/data_update_quality.py — collect_findings runs the D3 structural
  checks on the frames the updater ALREADY warmed (no extra API call) for the
  selected-and-warmed endpoints; write_quality_report renders the deterministic
  D3 report (bounded, redacted examples) plus a non-secret run-context block,
  written even when clean. No exchange calendar synthesized (structural only).
- updater: update_endpoints gains an optional capture sink (default None =
  byte-identical warm path); run_data_update runs the hook only when enabled;
  UpdateResult gains quality_report_path / quality_findings_count /
  quality_hard_count.
- cli: data-update prints the report path + counts only when enabled; disabled
  output is materially unchanged.

Report-only: never filters/repairs/mutates data, never fails the job, never
changes cache coverage or the per-endpoint request summary. phase0 anchor
(0.9600/0.8408) and all daily behavior unchanged.

tests/test_data_update_quality.py (network-free, fake feeds): default disabled;
clean report; bad daily+intraday frames -> hard findings without raising or
altering the summary; endpoint selection honored; report_name path validation;
report carries no secret path/key/token value.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant