feat(analytics): score the eleven minute factors on the 14:51 VWAP exec-to-exec return - #79
Conversation
…xec return
A minute-derived factor is fixed at the 14:50 signal cutoff, and the only fill
this framework can MODEL is the 1min bar opening at 14:51. Scoring it on
close(t+h)/close(t) credits it with a closing auction the project cannot
simulate (closing_call_proxy needs stk_auction_*, which the data plan lacks).
Adds the execution-anchored basis alongside the existing one — the eleven
factors are now evaluated FOUR times (close_to_close and exec_to_exec, each
no-book and with-book). The accepted close_to_close artifacts keep their names
and are untouched; the new pair is written as {stem}_exec_{no,with}_book.
R_exec[d,s] = (vwap(d+h)*af(d+h)) / (vwap(d)*af(d)) - 1
The bar is selected and priced by the existing execution layer (resolve_fill /
bar_execution_price via build_execution_prices) rather than a second copy of
those rules; the adjustment identity is the one PR #75 established for
IntradayTailEventModel.holding_returns, whose per-symbol anchor cancels in the
ratio; and d+h steps the FACTOR'S evaluation grid, matching what
FactorSpec.forward_return_horizon declares a horizon to be.
Undefined is missing, never a fallback: no bar in the window, an undefined VWAP,
or an unusable adj_factor each yield NaN under their own cause. The bar close is
never substituted for the VWAP, the daily close never for the bar, and
adj_factor never assumed 1.0.
One adjusted-execution-price artifact, keyed on universe/window/grid/execution
parameters, is shared by every factor, so the family is scored against ONE
return series and cross-factor comparison stays meaningful.
Disclosure travels inside each exec report via EvalContext.execution_capacity
(the contract's documented pass-through): execution parameters, coverage loss by
cause, the measured stk_mins live-call count, and the sanity headline. The two
keys that axis reads for a verdict are deliberately absent, so the Tradable axis
stays NOT_ASSESSED exactly as on the close_to_close runs. The evaluator's guard
against deriving a non-close basis from a close panel is unchanged and
regression-tested; the exec context withholds price_panel so an absent return
raises rather than quietly measuring close-to-close under an exec_to_exec label.
qt/exec_basis_sanity.py adds four implementation-independent checks (agreement
with close_to_close, magnitude, hand recomputation straight from the raw cached
bars, an ex-date holding period) and RAISES rather than let a disagreeing series
be scored against.
Measured on CSI500 2021-07-01..2026-06-30 (996 symbols, 1210 dates, cache-only,
zero live calls): agreement median 0.9890, exec std 2.77% vs close 2.80%, five
hand-recomputed rows exact to 0.0, coverage loss 3274 pairs = 0.283% of
close_to_close (no_bar 1733 / bad_vwap 1541 / bad_adj_factor 0, 180 symbols).
0.133% of pairs is small, but the excluded minutes are precisely the ones with no traded volume or amount — so it is a liquidity-correlated, NON-RANDOM sample restriction, not a rounding loss. Left as a bare percentage it would read as noise. Both the exec reports and the sanity report now say what it is, and a test pins the wording so it cannot quietly drop out.
… rebasis PR number (#79) Both follow-ups now have real numbers, read from gh pr list rather than assumed: #78 is fix/limit-basis-single-source and #79 is feat/exec-to-exec-rebasis, both OPEN. The rebasis entry no longer says "PR not yet open". - New entry for #78: #76's regex guard claimed no further copy of the wrong gate description could ship, and review disproved it by running seven plausible rewordings through the actual regex — all seven escape. The fix is structural: limit_basis_phrase() is the one authored sentence and three sites compose it; rendered text is unchanged; the regex stays with its scope corrected to catching a literal revert. - Sharpened the second defect-class rule accordingly: the rule is not "assert no (N+1)th copy exists" — that is precisely what the regex failed to do — it is author the claim once and have every other site compose it. A regex cannot assert that no other sentence makes a claim; having no other sentence can. - #79 entry gains its number, the reason for the rebasis (a minute factor is fixed at 14:50 and the only modellable fill is the 14:51 bar, so close-to-close credits a closing auction this project cannot simulate), the two coverage denominators kept apart as the PR keeps them, and the correctness acceptance. Sources: gh pr view 78, gh pr view 79, RESULTS_post_pr75_2026-07-21.md.
What this merged on — and what it did notA code review was dispatched for this PR and did not return within ~1.7h. Merging anyway, on the verification below rather than on an implied review pass. Recording it so nobody later assumes a review signed this off. Independently verified before merge (by me, not by the branch's own tests):
Not independently verified, and therefore still open to a later look: the five mutation claims (I read them but did not re-run them), the eleven runner edits being mechanically identical, and the Why merging is low-risk despite that: nothing depends on this PR yet, the close-to-close artifacts are untouched and remain the validated reference, and a revert is clean. If a later review finds something in the unverified set, the exec-basis numbers are restated — not the close-to-close ones. |
Review of #82 verified the sweep independently -- 12 pre-fix hits, 0 post-fix, line numbers matching -- and noted that #79's qt/exec_basis_eval.py, qt/exec_basis_sanity.py and qt/exec_forward_returns.py describe the same execution anchor from the factor side and were not in the scan's target list. They are accurate today; it read them. Adding them anyway. The entire reason this guard had to be widened twice was that its first version covered only the files already fixed, which meant it could confirm nothing except what was already known. Waiting for these three to go stale before listing them would repeat that exactly. No behaviour change; the scan passes with them included.
1 participant
Switches the eleven minute-derived factors' evaluation return basis from daily
close-to-close to the 14:51 VWAP exec-to-exec adjusted return, and keeps
close-to-close beside it as the control. Every factor is now evaluated four
times:
close_to_closeandexec_to_exec, each with and without the book.Why. A minute factor is fixed at the 14:50 cutoff, and the only fill this
framework can model is the 1min bar opening at 14:51. Scoring it on
close(t+h)/close(t)credits it with a closing auction the project cannotsimulate —
closing_call_proxyneedsstk_auction_*, which the data plan doesnot carry.
1. Regression: the rebasis did not move one existing number
This is what licenses everything below.
Leaf-by-leaf over every scalar in all 22
close_to_closereport JSONs (nested,not just top-level keys), branch vs the validated main-tree copies. A basis was
added; none was perturbed.
2. The eleven factors on both bases
ic_irandeffective_samplesare identical forno_book/with_book(thebook only enters the Incremental axis), so one row per factor. CSI500
000905.SH, 2021-07-01 → 2026-06-30, daily rebalance,h=1.20 of 22 cells unchanged. Two moved, both toward less confidence.
The mechanism is single and clean: |ICIR| rises in 10 of 11 factors while
N_eff falls in 9 of 11. A stronger point estimate on a smaller effective
sample — a wider interval around a larger number — which is exactly the trade the
v0.9 gate exists to adjudicate. It adjudicates conservatively.
3. The two changed cells
valley_ridge_vwap_ratio— no_book: Watch → INSUFFICIENT-DATAAligned per-date monotonicity 95% CI (N_eff-based):
[+0.0010, +0.0711][−0.0054, +0.0680]Its Watch rested on a lower bound of +0.0010 — the thinnest margin in the
table. Everything else on the axis still clears (ICIR lower bound +0.355 > 0.3,
|NW-t| 14.11, win rate 0.668, OOS sign consistent).
with_bookstays Watch, but now carried by Incremental PASS alone —Predictive is INSUFFICIENT_DATA on that cell too.
ridge_minute_return— with_book: Reject → INSUFFICIENT-DATAThe point estimate crossed the bar; the interval did not.
INSUFFICIENT-DATA is not a rescue
Both moves are downgrades in certainty, not upgrades in standing. Nothing was
demonstrated and nothing was refuted. INSUFFICIENT-DATA is unreachable from a
positive claim: it cannot be cited as evidence for a factor, it does not
supersede the Reject it replaced with anything better, and neither cell may be
read as "improved under the more realistic basis."
4. What the new basis costs — unsoftened
Two different denominators, kept apart on purpose because
1,541appears underboth meanings.
Per-anchor, over the 1,158,912 (date, symbol) pairs of the panel grid:
okno_bar— no 1min bar in the execution windowbad_vwap— bar exists, VWAP undefinedbad_adj_factorPer forward-return pair, per factor (each factor's own evaluation grid):
no_barbad_vwapbad_adj_factorstk_mins_live_callsAnd the VWAP basis specifically blocks 1,541 more pairs than a bar-close basis
would (0.133% of the 1,156,183 a close basis can measure). That exclusion is
LIQUIDITY-CORRELATED and NON-RANDOM: the excluded minutes are exactly the
illiquid ones — the minutes in which the stock did not trade at all. It is a
biased sample restriction with a direction, not a rounding loss, and a small
percentage is not the same as a random one. Every exec report carries this
statement in those words (
coverage_bias_bad_vwap), and the sanity report statesit in prose above the coverage table so the percentage is never read first. A
test pins the wording.
5. The N_eff cost, stated as a cost
The more realistic basis is paid for in statistical power: N_eff falls in 9
of 11 factors (1187.7 → 1116.0, 1009.5 → 933.3, 1122.9 → 1055.8). That is the
price of measuring the return you could actually have earned, and it is why two
verdicts became less certain rather than more. It is a real debit, not an
accounting artifact, and it should be weighed against the realism gained.
6. Correctness acceptance (§3), CSI500 at full scale
996 symbols, 1,210 dates, cache-only,
stk_mins_live_calls=0, all tendaily/universe gap-fetch counters 0.
1 — Agreement with close_to_close. Per-date cross-sectional correlation over
1,209 dates: median 0.9890 (p10 0.9821, p90 0.9933); Spearman median 0.9814.
Floor 0.90 → PASS. (The builder raises below the floor rather than proceeding.)
2 — Magnitude. Same order, exec marginally tighter:
3 — Five rows recomputed by hand from the raw cached 1min parquet with plain
arithmetic (no execution-layer helper imported, so it cannot inherit a bug from
one). Worst absolute difference 0.0 — exact:
4 — Ex-date holding periods, shown both ways:
The first row's exit is 12-31, not 12-30: that symbol has no panel row on 12-30,
and the positional shift correctly took its next observed period — the same
rule
analytics.factor.forward_returnsuses.7. Mutation evidence — each actually applied to the source, run, and reverted
Afterwards
grep -c MUTANT= 0 in both touched files andgit diff runtime/intraday_execution.pyis empty.(a) VWAP → bar close (
ExecBasisParams.exec_config) →test_exec_basis_price_is_bar_vwap_never_bar_close(b1) missing adj_factor → 1.0 (
_assemble) →test_exec_basis_unusable_adj_factor_is_counted_never_assumed_one[nan](b2) undefined VWAP → bar close (
runtime/intraday_execution.bar_execution_price) →test_exec_basis_undefined_vwap_is_bad_vwap_never_the_bar_close[0.0-1000.0](c) adjustment removed (
_assemble) →test_exec_basis_ex_date_period_is_corporate_action_free(d) d+h on the data grid, not the evaluation grid (
exec_forward_returns) →test_exec_basis_steps_the_evaluation_grid_not_the_data_grid8. VWAP vs bar_close at scale — a NON-confirmation, not agreement
A full
bar_closevariant was built over the same grid, both adjusted:The sign is nominally opposite to "VWAP is slightly conservative", which is
flagged rather than smoothed over — but at 0.07 standard errors it is
indistinguishable from zero.
This neither confirms nor contradicts the −0.18pp measured on the I5f/I5d
windows, and it is not offered as agreement. The two quantities cannot speak to
each other: −0.18pp is a portfolio effect, where the fill price interacts with
which names get selected; this is the unconditional return distribution, which
carries no such interaction. The I5d row-3-vs-row-4 attribution remains the only
measurement that speaks to the portfolio effect. No backtest was run here.
9. What is in the diff
qt/exec_forward_returns.pyExecBasisParams(one source → exec config / the 5 spec fields / the cache key),intraday_spec_variant, the shared adjusted-exec-price artifact builder,exec_forward_returns,coverage_lossqt/exec_basis_sanity.pyqt/exec_basis_eval.py{stem}_exec_no_book/{stem}_exec_with_book(+ dashboards, +{stem}_exec_basis_sanity.md)qt/eval_*.py,qt/cli.pytests/test_exec_forward_returns.py(29),tests/test_exec_basis_eval.py(5)Design points worth a reviewer's attention:
resolve_fill/bar_execution_priceviabuild_execution_prices), not a second copy. Theadjustment is PR fix(runtime): fill intraday tail rebalances at the execution bar's VWAP, and divide corporate actions out of holding returns #75's identity verbatim:
(raw*af)_exit / (raw*af)_entry − 1,whose per-symbol anchor cancels in the ratio.
d+hsteps the FACTOR's evaluation grid, matching whatFactorSpec.forward_return_horizondeclares a horizon to be — not the nextcalendar day and not the next day the minute cache happens to hold.
unusable
adj_factoreach yield NaN under their own cause. The bar close isnever substituted for the VWAP, the daily close never for the bar, and
adj_factoris never assumed 1.0.identical source for all eleven, while each factor's
hstill steps its owngrid. Keyed on universe / window / grid / execution parameters, so a changed
parameter cannot hit a stale artifact.
EvalContext.execution_capacity— thecontract's documented pass-through — with
tradable/capacity_sufficientdeliberately omitted so the Tradable axis stays NOT_ASSESSED exactly as on
the close_to_close runs. A coverage disclosure must not become an implicit
tradability claim; a test pins it.
exec_to_execspec with no supplied returns still raises. The exec context withholds
price_panel, so an absent return series raises rather than quietly measuringclose-to-close under an
exec_to_execlabel.measurement; blocking is an execution-feasibility question and stays in the
backtest.
The eleven configs declare no
intraday:block, so the execution parametersresolve to
qt.config.IntradayCfg's defaults (14:50 / 14:51–14:56:59 /next_minute_close/bar_vwap/ 09:30). Which source applied is printed inevery exec report as
execution_parameter_source.Gates
pytest1774 passed —main(aa19968) measured at 1740, so +34,exactly the two new test files, 0 displaced (the task card's 1728 baseline was
b39c756;mainadded 12 via #76/#77) ·ruffclean · phase0ic_mean=0.9600, annual_return=0.8408· all 31 configs validate · secret scan0 hits for the token value,
token, or.config.jsonacross the diff, thenew modules, the tests and the generated reports.
mainwas merged in before these gates were run; zero file overlap with #76/#77.