feat(data): D3b default-off data-update quality-report hook - #47
Merged
StackOverFlow11 merged 1 commit intoJun 19, 2026
Merged
Conversation
Surface the accepted D3 data/quality checks in operations via an opt-in, report-only hook on the data-update job. Default OFF: every existing config behaves exactly as before. - config: nested DataUpdateQualityCfg under data_update.quality (enabled=false; endpoints restricted to the structural surfaces the updater loads as frames — market_daily/adj_factor/stk_mins_1min; report_name must be a bare filename under output.report_dir). config/data_update.yaml gains the disabled block. - hook: new qt/data_update_quality.py — collect_findings runs the D3 structural checks on the frames the updater ALREADY warmed (no extra API call) for the selected-and-warmed endpoints; write_quality_report renders the deterministic D3 report (bounded, redacted examples) plus a non-secret run-context block, written even when clean. No exchange calendar synthesized (structural only). - updater: update_endpoints gains an optional capture sink (default None = byte-identical warm path); run_data_update runs the hook only when enabled; UpdateResult gains quality_report_path / quality_findings_count / quality_hard_count. - cli: data-update prints the report path + counts only when enabled; disabled output is materially unchanged. Report-only: never filters/repairs/mutates data, never fails the job, never changes cache coverage or the per-endpoint request summary. phase0 anchor (0.9600/0.8408) and all daily behavior unchanged. tests/test_data_update_quality.py (network-free, fake feeds): default disabled; clean report; bad daily+intraday frames -> hard findings without raising or altering the summary; endpoint selection honored; report_name path validation; report carries no secret path/key/token value.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Data Layer D3b — a small, default-off quality-report hook that surfaces the already-accepted D3
data/quality/checks in thedata-updatejob. Report-only; not D4.With
data_update.quality.enabled=false(the default) every existing config behaves exactly as before. When enabled, the updater runs the D3 structural checks on the frames it already warmed (no extra API call) and writes a deterministic Markdown report.What changed
qt/config.py,config/data_update.yaml): nestedDataUpdateQualityCfgunderdata_update.quality—enabled=false;endpointsrestricted to the structural surfaces the updater materializes as frames (market_daily/adj_factor/stk_mins_1min, unknown rejected readably);report_namevalidated as a bare filename underoutput.report_dir(no separators /../ absolute). The YAML gains the disabled block with comments.qt/data_update_quality.py, new):collect_findingsruns the D3 checks for the selected-AND-warmed endpoints on the already-loaded frames;write_quality_reportrenders the deterministic D3 report (bounded, redacted examples) plus a non-secret run-context block (window / symbol count / endpoint names), written even when clean. No exchange calendar synthesized → structural checks only.qt/data_updater.py):update_endpointsgains an optionalcapturesink (defaultNone⇒ byte-identical warm path — same calls, args, order, summary);run_data_updateruns the hook only when enabled;UpdateResultgainsquality_report_path/quality_findings_count/quality_hard_count.qt/cli.py):data-updateprints the report path + counts only when enabled; disabled output materially unchanged.Invariants held
Report-only: never filters / repairs / mutates data, never fails the job, never changes cache coverage or the per-endpoint request summary, no new Tushare endpoint, no extra live call for quality, no
PanelStorewrite, no ledger schema change. Daily close-to-close behavior and the phase0 anchor (ic_mean=0.9600, annual_return=0.8408) unchanged.Out of scope (not started): D4 /
CoverageLedgerstorage scaling / batch writes / concurrency / schema registry; the optionalfail_on_hardknob.Tests / gates
tests/test_data_update_quality.py(network-free, fake feeds): default disabled; clean report; bad daily+intraday frames → hard findings without raising or altering the summary; endpoint selection honored (not-warmed surface never checked);report_namepath validation; report carries no secret path / key / token value (D3 redaction inherited); capture never changes the summary.ruffclean. All 17 configsvalidate-config.run-phase0anchor0.9600 / 0.8408.git diff --checkclean; no merge markers. Secret scan over changed files: real token value 0 hits (the onlytushare.token/.config.jsonliterals are the pre-existing config key/path defaults + deliberate redaction-test scan targets).Live
data-updateagainst Tushare was not run (acceptance is network-free per the goal).