Skip to content

fix: do not derive ordering for arithmetic that can overflow - #23910

Merged
kosiew merged 5 commits into
apache:mainfrom
buraksenn:23902-dont-remove-sort-for-overflow-and-wrap
Aug 5, 2026
Merged

fix: do not derive ordering for arithmetic that can overflow#23910
kosiew merged 5 commits into
apache:mainfrom
buraksenn:23902-dont-remove-sort-for-overflow-and-wrap

Conversation

@buraksenn

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

Please check the issue for details but the main idea is that a + b over two
sorted columns is not guaranteed to be sorted.

What changes are included in this PR?

Ordering is discarded when it overflows or wraps

Are these changes tested?

Yes, adjusted existing tests and added a regression test in order.slt.

Are there any user-facing changes?

no api changes

@github-actions github-actions Bot added physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt) labels Jul 26, 2026
@rluvaton

Copy link
Copy Markdown
Member

@buraksenn Can you please mention the devs that added those test cases in the first place for review?

@codecov-commenter

codecov-commenter commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.66298% with 35 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.88%. Comparing base (74eebbb) to head (d31efba).
⚠️ Report is 66 commits behind head on main.

Files with missing lines Patch % Lines
datafusion/physical-expr/src/expressions/binary.rs 77.98% 2 Missing and 33 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #23910      +/-   ##
==========================================
+ Coverage   80.70%   80.88%   +0.18%     
==========================================
  Files        1095     1102       +7     
  Lines      372554   376216    +3662     
  Branches   372554   376216    +3662     
==========================================
+ Hits       300661   304310    +3649     
+ Misses      53924    53763     -161     
- Partials    17969    18143     +174     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the core Core DataFusion crate label Jul 27, 2026
@buraksenn
buraksenn marked this pull request as ready for review July 27, 2026 07:18
@buraksenn

Copy link
Copy Markdown
Contributor Author

@buraksenn Can you please mention the devs that added those test cases in the first place for review?

@mustafasrepo @berkaysynnada @geoffreyclaude can you take a look at this PR if you've time?

# Conflicts:
#	datafusion/physical-expr/src/expressions/binary.rs

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@buraksenn
Nice fix. The new invariant makes sense: arithmetic orderings are only preserved when overflow cannot happen or when overflow is configured to fail. The updates across the optimizer, tests, and sqllogictests look consistent. I just have a couple of small suggestions that could make the coverage a bit stronger.

use datafusion_expr::col as logical_col;

#[test]
fn test_arithmetic_ordering_overflow() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice focused regression test. One small suggestion: could we also add a couple of Minus cases covering signed minimum and unsigned underflow boundaries? The implementation now guards both Plus and Minus, but this test only exercises addition. It would help protect the subtraction path from future regressions.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added signed minimum and underflow boundaries which resulted into finding a gap. Resolved that as well thanks for finding this

(expected | false),
"{err_msg}"
);
let may_overflow = ordering

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could narrow this exemption to only the specific arithmetic ordering that may be conservatively rejected. Using any means we skip the rejection assertion for the entire multi-key requirement whenever one key contains + or -, which reduces coverage for any later non-arithmetic keys. Not blocking, since it does not affect production behavior.

@buraksenn buraksenn Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks I've applied this as well

@kosiew kosiew left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@buraksenn, thanks for the follow-up changes.

The subtraction ordering regression coverage now includes signed-minimum overflow, unsigned underflow, safe bounded ranges, and time-minus-interval wrapping. The bounded unsigned subtraction logic also correctly compares the minimum left-hand endpoint with the maximum right-hand endpoint, which avoids deriving ordering guarantees when subtraction could underflow and wrap.

The fuzz-test exemption is now limited to the arithmetic-free ordering prefix in both the direct-ordering and projection tests, while still checking preceding non-arithmetic keys.

I did not find any new issues in the updated changes.

@kosiew
kosiew added this pull request to the merge queue Aug 5, 2026
Merged via the queue into apache:main with commit e7304ee Aug 5, 2026
40 checks passed
@buraksenn
buraksenn deleted the 23902-dont-remove-sort-for-overflow-and-wrap branch August 8, 2026 20:51
kosiew pushed a commit to kosiew/datafusion that referenced this pull request Aug 12, 2026
…23910)

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases.
You can link an issue to this PR using the GitHub syntax. For example
`Closes apache#123` indicates that this PR will close issue apache#123.
-->

- Closes apache#23902.

## Rationale for this change

Please check the issue for details but the main idea is that `a + b`
over two
sorted columns is not guaranteed to be sorted.

## What changes are included in this PR?
Ordering is discarded when it overflows or wraps

## Are these changes tested?

Yes, adjusted existing tests and added a regression test in `order.slt`.

## Are there any user-facing changes?
no api changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-expr Changes to the physical-expr crates sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sort should not be removed when order by a + b where a and b are sorted as they can overflow and wrap

4 participants