feat: intraday 1min PIT pipeline (I1–I4: feed/schema, cache, aggregation, tail execution) - #29
Merged
Merged
Conversation
Adds a SEPARATE intraday data path that never touches the daily panel contract. - data/clean/intraday_schema.py: minute-bar schema (MultiIndex(time, symbol), time kept at minute precision, never normalized to midnight). Derives PIT time fields bar_end=trade_time, bar_start=bar_end-freq, available_time=bar_end+data_lag. Raw persisted freq is 1min only (RAW_INTRADAY_FREQ + ensure_raw_intraday_freq); coarser freqs are derived views, not raw products. - data/feed/tushare_intraday.py: TushareIntradayFeed.get_minutes over stk_mins; raw field mapping only (ts_code->symbol, vol->volume, trade_time->time/bar_end); non-1min raw freq rejected before any SDK call; token read live, never logged. Includes an opt-in cache=None hook (dormant until I2). - Tests: minute-precision, PIT derivation, reverse-input sort, dedup last-wins, readable errors, empty frame, no-token-leak. No daily feed/cache/pipeline change.
Endpoint-level read-through cache for raw 1min bars, separate from the daily cache (zero change to the 540-line daily TushareCache). - data/cache/intraday_coverage.py: timestamp-interval ledger (raw_freq/start_time/ end_time), NOT the daily date-range ledger; ok/empty cover, failed does not. - data/cache/intraday_parquet_store.py: month-partitioned store (stk_mins_1min/freq=1min/symbol_prefix/symbol/year/month.parquet); atomic idempotent upsert by (symbol, freq, bar_end); stores RAW only. - data/cache/intraday_cache.py: TushareIntradayCache.stk_mins_1min read-through — trading-day gap planning (reuses intervals.py), <=23-day window paging under the 8000-row cap, 1min-only guard, empty recorded / failed retried. - Wires an optional cache into TushareIntradayFeed (direct path byte-identical). - Tests: cold writes / warm zero-call / partial gap / empty / failed / dup-upsert / non-1min reject / no-secret / cached==direct after normalize. Daily cache/feed/pipeline unchanged.
PIT-safe aggregation from normalized 1min bars to a daily (date, symbol) feature table for the factor layer. Filter available_time <= (trade_date + decision_time) BEFORE grouping by day — minute timestamps are never date-normalized first. - data/clean/intraday_aggregate.py: asof_daily_features (default 14:50:00 cutoff; cutoff-encoded columns intraday_ret_0930_1450 / realized_vol / vwap / last30m_ret_1420_1450) + resample_intraday_bars (derived coarse bars carry available_time = max(source_1min.available_time), never bar_end+lag). - Tests: cutoff leakage (perturb post-14:50 -> features unchanged), availability exclusion, daily index, multi-symbol/day isolation, empty, custom cutoff, derived-bar max-availability. No factor/alpha/portfolio/runtime wiring; daily layer untouched.
Second backtest event model intraday_tail_rebalance as a standalone runtime/execution skeleton — NOT wired into the daily pipeline or config. - runtime/intraday_execution.py: IntradayExecutionConfig / ExecutionFill / TailRebalanceResult + resolve_fill / build_execution_prices / simulate_tail_rebalance. Three boundaries kept distinct: signal cutoff (T 14:50) / execution timestamp (T 14:51, next_minute_close) / holding period (exec(T)->exec(T_next), never close-to-close). Missing bar / NaN price / no bar in window -> explainable blocked record, never a silent daily-close fallback. Only intraday bars are read (no T-day EOD leak). - Tests: cutoff vs exec separation, next_minute_close + window fallback, blocked reasons, exec-to-exec != close-to-close, blocked exclusion. Daily close_to_next_period + Phase 0/2/3 unchanged (run-phase0 ic 0.96/annual 0.84).
StackOverFlow11
added a commit
that referenced
this pull request
Jun 13, 2026
docs: record intraday I1–I4 pipeline (PR #29) in CLAUDE.md progress
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a complete, SEPARATE intraday (1min) research track — raw feed/schema → read-through cache → minute→daily PIT aggregation → tail-rebalance execution semantics. Nothing touches the daily pipeline:
factors/alpha/portfolio, the dailyruntime/backtest, dailyTushareFeed/TushareCache, and allconfig/models are unchanged. Phase 0/2/3 behavior is identical (run-phase0stillic_mean=0.9600, annual_return=0.8408).Four logically-distinct stages, one commit each:
I1 —
feat(data): 1min raw feed + PIT schemadata/clean/intraday_schema.py: minute-bar schemaMultiIndex(time, symbol)—timekept at minute precision, NEVER normalized to midnight (separate from the daily panel contract). PIT fieldsbar_end=trade_time,bar_start=bar_end-freq,available_time=bar_end+data_lag. Raw persisted freq is 1min only; coarser freqs are derived views (ensure_raw_intraday_freq).data/feed/tushare_intraday.py:TushareIntradayFeed.get_minutesoverstk_mins; raw field mapping only; non-1min raw freq rejected before any SDK call; token read live, never logged.I2 —
feat(cache): stk_mins 1min read-through cachestk_mins_1min/freq=1min/symbol_prefix/symbol/year/month.parquet), trading-day gap planning (reusesintervals.py), ≤23-day window paging under the 8000-row cap. Cold writes / warm zero-call / partial-gap / empty-recorded / failed-retried; cached output is byte-identical to the direct feed after normalize. DailyTushareCacheuntouched (separate classes).I3 —
feat(data): minute→daily PIT aggregation (14:50)asof_daily_features(defaultdecision_time=14:50:00): filteravailable_time <= cutoffbefore grouping by day (minute timestamps never date-normalized first). Cutoff-encoded columns:intraday_ret_0930_1450,intraday_realized_vol_0930_1450,intraday_vwap_0930_1450,intraday_last30m_ret_1420_1450.resample_intraday_bars: derived coarse bars carryavailable_time = max(source_1min.available_time).I4 —
feat(runtime): tail-rebalance execution skeletonsimulate_tail_rebalancekeeps three boundaries distinct: signal cutoff (T 14:50) / execution timestamp (T 14:51,next_minute_close) / holding period (exec(T)→exec(T_next), never close-to-close). Missing bar / NaN price / no bar in window → explainable blocked record, never a silent daily-close fallback. Standalone — NOT wired into config/pipeline.Invariants held
close_to_next_periodunchanged.Dstays a separate retained source..config.jsoncontent in any committed file (scanned: 0); cache parquet + ledger are gitignored.Test plan
pytest tests -q→ 484 passed (+56 intraday: I1 schema 14 + feed 9, I2 cache 10, I3 agg 13, I4 exec 10; 0 regressions).ruff check .→ clean.qt validate-config --config config/example.yaml→ OK.qt run-phase0 --config config/example.yaml→ic_mean=0.9600, annual_return=0.8408(unchanged).Stage-by-stage acceptance evidence (PIT semantics, cold/warm/partial, leakage tests, boundary notes) lives in
tmp/context/intraday_pit_checkpoints/stage_i{1,2,3,4}_acceptance.md(gitignored, local).