expr: fix non-monotone annotations on timestamp/date/interval functions

[antiguru]

Motivation

Fixes database-issues#9656 — the persist filter pushdown correctness violation! audit panic that has been reproducing under parallel-workload --scenario regression --complexity ddl.

Description

Interval is lex-ordered by (months, days, micros) (derived Ord), but several scalar functions that consume an interval do calendar-aware arithmetic (variable-length months, day-clamping, bin alignment to the unix epoch) which does not respect that ordering. The interpreter behind persist filter pushdown was marking these as monotone, so it mapped only the endpoints of an interval range — and missed interior values whose function results fall outside the endpoint-bounded box. Filter pushdown then incorrectly concluded the part had no matching rows, tripping the audit panic in persist_source.rs:625.

Bug 1: add/sub_{timestamp,timestamp_tz,date}_interval

For t = 2024-01-31:

interval	lex order	t + i
{0 months, 31 days}	smaller	2024-03-02
{1 month, 0 days}	larger	2024-02-29

So the smaller interval produces the larger output. For timestamps, day-clamping also collapses near-boundary inputs into the same date while preserving sub-day time, breaking monotonicity in the first argument too:

2024-01-30 23:59:59 + 1 month  →  2024-02-29 23:59:59
2024-01-31 00:00:00 + 1 month  →  2024-02-29 00:00:00

For dates the first argument is monotone (no sub-day precision means clamping only collapses, never reverses), but the interval argument has the same problem.

Demotions:

• add_timestamp_interval, add_timestamp_tz_interval, sub_timestamp_interval, sub_timestamp_tz_interval → (false, false)
• add_date_interval, sub_date_interval → (true, false)

Bug 2: date_bin_{timestamp,timestamp_tz}

date_bin(stride, source) = origin + floor((source - origin) / stride) * stride. For a fixed source like 2024-01-01 12:00:00, a 1-day stride bins to 2024-01-01 00:00:00 but a 2-day stride bins to 2023-12-31 00:00:00 — the lex-larger stride produces an earlier output, because alignment to the unix epoch depends on stride magnitude, not lex order. Demoted from (true, true) to (false, true) (still monotone in source).

Context

This is the companion fix to b607993 ("Mark some interval-related functions as non-monotone"), which corrected the analogous annotations for the *_time_interval and *_interval (interval × scalar) variants but left the calendar-aware variants untouched. Together with the cases already fixed in #9301, this completes the sweep of interval-consuming functions whose result doesn't respect interval lex order.

The second commit removes the persist_stats_filter_enabled: "false" override in parallel_workload/settings.py that was added to suppress this audit panic, so CI re-exercises the audit and surfaces any remaining latent causes.

On the "numeric trim" suspicion in the issue thread

I traced every is_monotone annotation on numeric functions (arithmetic, casts, rounding, etc.) and they all look correct. I think the audit failures previously attributed to numeric trimming were actually the timestamp/date+interval bug class showing up in mixed expressions — parallel-workload going green with only this fix supports that read.

Verification

• New regression tests in interpret.rs:
  • test_add_timestamp_interval_non_monotone — constructs (2024-01-31 + interval_col) >= 2024-03-15 with interval_col ranging over [{0m,31d}, {1m,0d}]. Under the buggy annotation the interpreter ruled out Datum::True (and so pushdown would have skipped the part); with the fix it correctly admits True.
  • test_date_bin_timestamp_non_monotone — constructs date_bin(stride_col, 2024-01-01 12:00:00) >= 2024-01-01 with stride_col ranging over [1 day, 2 days]. Asserts the interpreter admits both True (1-day stride) and False (2-day stride).
• cargo test -p mz-expr --lib — all 50 tests pass locally, including the proptest scalar::func::test::test_is_monotone.
• Parallel-workload override removed; the user has confirmed the parallel-workload regression scenario runs green with this fix.

Tradeoff and follow-up

The non-monotone marking on sub_timestamp_interval and friends is conservative — even a day-only interval shares the annotation with month-bearing intervals because the static interpreter can't distinguish them. As a result, EXPLAIN ... WITH (filter pushdown) no longer reports a pushdown= line for predicates like timestamp_col - INTERVAL '1' day < literal (the slt test in this PR reflects that). That class of predicate was a motivating use case for filter pushdown, so the regression is addressed in #36706, which stacks on this PR and recovers the pushdown via a SpecialBinary dynamic-monotonicity check on the interval value.

[def-]

Amazing! I'll rebase my workload-replay change on top of this and verify: #36686

Also triggered a nightly run: https://buildkite.com/materialize/nightly/builds/16533

[def-]

Does it actually fail without the PR? The QA LLM review claims not, but I didn't verify:

MEDIUM — test_date_bin_timestamp_non_monotone is a tautology and does not actually regression-test the fix

File: src/expr/src/interpret.rs:1844-1903

The test sets up date_bin(stride_col, 2024-01-01 12:00:00) >= 2024-01-01 00:00:00 with stride_col ∈ [{0,1,0}, {0,2,0}] and asserts that the interpreter admits both True and False. The endpoint evaluations are:

• date_bin({0,1,0}, 2024-01-01 12:00) = 2024-01-01 00:00 → >= is True.
• date_bin({0,2,0}, 2024-01-01 12:00) = 2023-12-31 00:00 → >= is False.

Because the endpoints already produce opposite boolean answers under the (buggy) (true, true) annotation, the interpreter's union of {False} and {True} already admits both outcomes. The test therefore passes regardless of whether date_bin_timestamp is marked (true, true) or (false, true).

Verified empirically: I reverted src/expr/src/scalar/func.rs:2057 to is_monotone = "(true, true)" while leaving the rest of the PR in place, and ran cargo test --lib -p mz-expr test_date_bin_timestamp_non_monotone — the test still passes.

In contrast, the companion test test_add_timestamp_interval_non_monotone is genuinely diagnostic — reverting the add_timestamp_interval annotation makes it fail with interpreter incorrectly ruled out matching rows.

The in-test comment is also self-contradictory: it asserts "both endpoint evaluations give the same boolean answer" while the lines just above explicitly list one endpoint satisfying >= and the other not.

Impact: The fix to date_bin_timestamp / date_bin_timestamp_tz is correct, but the test that's supposed to lock it in provides false assurance. A later refactor that re-promotes the annotation would not be caught by cargo test -p mz-expr --lib (the PR description points to this command as the verification gate). Given that filter-pushdown correctness bugs are P1/test-blocker (see #9656), losing the regression coverage is a real durability concern.

Suggested fix: Compare against a timestamp strictly between the two endpoint outputs so that the buggy lex-mapping rules out the matching-rows case. For example:

// Endpoint outputs are 2023-12-31 00:00 and 2024-01-01 00:00. An interior
// stride of `{0, 1 day, 12h-worth-of-micros}` bins source to 2024-01-01 12:00,
// which is outside the lex-endpoint box. Comparing against 2024-01-01 12:00:00
// forces the interpreter to (wrongly, under the buggy annotation) rule out
// True.
let expr = MirScalarExpr::column(0)
    .call_binary(ts_lit("2024-01-01T12:00:00"), DateBinTimestamp)
    .call_binary(ts_lit("2024-01-01T12:00:00"), Gte);

// ... same column setup ...

assert!(
    range_out.may_contain(Datum::True),
    "date_bin is not monotone in the stride argument; \
     interpreter must not rule out matching rows",
);

Under (true, true), the lex range [2023-12-31, 2024-01-01] is entirely < 2024-01-01 12:00:00, so the interpreter would only admit False and the may_contain(True) assertion would fail — making it a real regression test. Under the fix's (false, true), the stride flat-map yields anything(), both outcomes are reachable, and the assertion passes.

(Concretely, an interior stride such as {0, 1, 43200000000} — 1 day + 12 hours — does bin 2024-01-01 12:00:00 to itself, so the predicate is genuinely achievable. The test doesn't need to construct that value; it just needs to assert the interpreter doesn't rule out True.)

[antiguru]

Great find! Fixed in the latest commit.

[def-]

With this PR I'm still seeing the correctness panic: https://buildkite.com/materialize/release-qualification/builds/1250 I did find another non-monotone function though: #36708 Trying now: https://buildkite.com/materialize/release-qualification/builds/1251 Edit: Still the same panic.

[antiguru]

Claude found another class of problem, which is error handling: #36721

[def-]

Nice, next try with all 3 PRs included: https://buildkite.com/materialize/release-qualification/builds/1252 Edit: Failed, but only because we run a benchmark against main, and of course it crashes on main! Now trying without main: https://buildkite.com/materialize/release-qualification/builds/1256 Looking much better so far