fix: consider nulls_first when propagating ordered SortProperties

[Nagato-Yuzuru]

Which issue does this PR close?

• Closes Propagation of ordered SortProperties should consider nulls_first #11596.

Rationale for this change

In the original implementation, cases where null_first is not same were not correctly handled and were still treated as ordered.

Example: a sorted ASC NULLS FIRST, b sorted ASC NULLS LAST:

a	b	a + b
NULL	1	NULL
1	2	3
2	4	6
3	NULL	NULL

Since a + b is NULL wherever either input is, the result has nulls at both ends. The previous code returned Ordered(ASC, nulls_first: true) for this case; it now returns Unordered.

What changes are included in this PR?

Unit Tests were added and issues were fixed. Additional processing was performed on and_or (see #11596 (comment) )

SortProperties::{add, sub, gt_or_gteq, and_or} now propagate an Ordered result only when both operands agree on nulls_first; otherwise they return Unordered.

This is conservative. If one input were known to be non-null the old result could be valid. But SortProperties carries no nullability information, so Unordered is the only sound answer at this layer.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

[Nagato-Yuzuru]

The old query plan can lead to this sort error

In datafusion-cli

DataFusion CLI v54.1.0

>set datafusion.execution.target_partitions = 1;

-- Materialize the data through SQL so this repro is self-contained.
COPY (SELECT * FROM (VALUES
  (1,                 CAST(NULL AS INT)),
  (2,                 10),
  (3,                 1),
  (CAST(NULL AS INT), 0)
) AS t(inc_col, desc_col))
TO '/tmp/nulls_demo.csv' STORED AS CSV;

CREATE EXTERNAL TABLE demo (
  inc_col INTEGER,
  desc_col INTEGER
)
STORED AS CSV
WITH ORDER (inc_col ASC)     -- ASC  defaults to NULLS LAST  -> nulls at the end
WITH ORDER (desc_col DESC)   -- DESC defaults to NULLS FIRST -> nulls at the start
LOCATION '/tmp/nulls_demo.csv'
OPTIONS ('format.has_header' 'true');

-- Both columns really do match their declared orderings.
SELECT * FROM demo;

SELECT CAST((inc_col > desc_col) AS integer) AS c FROM demo ORDER BY c;

Get

+---------+----------+
| inc_col | desc_col |
+---------+----------+
| 1       | NULL     |   inc_col:  1, 2, 3, NULL  -> ASC NULLS LAST  ✓
| 2       | 10       |   desc_col: NULL, 10, 1, 0 -> DESC NULLS FIRST ✓
| 3       | 1        |
| NULL    | 0        |
+---------+----------+


+------+
| c    |
+------+
| NULL |
| 0    |
| 1    |
| NULL |
+------+

Order C was silently violated

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.02%. Comparing base (eec8b94) to head (db9e889).
⚠️ Report is 34 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #24206      +/-   ##
==========================================
- Coverage   81.05%   81.02%   -0.03%     
==========================================
  Files        1106     1106              
  Lines      382287   384267    +1980     
  Branches   382287   384267    +1980     
==========================================
+ Hits       309851   311364    +1513     
- Misses      54121    54560     +439     
- Partials    18315    18343      +28     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jayzhan211]

Do we have sqltest like the example in the description covered, like null + numeric -> null, then we expect to see the null first -> unordered

[Nagato-Yuzuru]

Thanks for comment. The add case is added.

i found that the + slt would produce an identical plan with and without this patch, and wouldn't guard the change. The add/sub half of the fix is only observable in the unit tests; gt_or_gteq and and_or are the parts reachable end to end.

#23910 made BinaryExpr::arithmetic_sort_properties fall back to Unordered unless it can prove the arithmetic doesn't overflow.

 let cannot_overflow = !range.is_unbounded()
            && !unsigned_subtraction_may_underflow(self.op, l_range, r_range, range);
        if !wraps_in_domain && (self.fail_on_overflow || cannot_overflow) {
            sort_properties
        } else {
            SortProperties::Unordered
        }

Here if endpoint null, so range.is_unbounded() holds for any col + col and the result is Unordered regardless of null placement. SortProperties::add's return value is discarded before nulls_first can matter. The SortExec still survives if declaring both columns ASC NULLS FIRST.

Before current patch:

DataFusion CLI v54.1.0
> set datafusion.execution.target_partitions = 1;
0 row(s) fetched.
Elapsed 0.001 seconds.

> COPY (VALUES (NULL, 1), (1, 2), (2, 4), (3, NULL))
TO 'test_files/scratch/order/mixed_null_placement.csv'
OPTIONS ('format.has_header' 'true');
+-------+
| count |
+-------+
| 4     |
+-------+
1 row(s) fetched.
Elapsed 0.002 seconds.

> CREATE EXTERNAL TABLE mixed_null_placement (
  a BIGINT,
  b BIGINT
)
STORED AS CSV
WITH ORDER (a ASC NULLS FIRST)
WITH ORDER (b ASC NULLS LAST)
LOCATION 'test_files/scratch/order/mixed_null_placement.csv'
OPTIONS ('format.has_header' 'true');
0 row(s) fetched.
Elapsed 0.001 seconds.

> SELECT a, b, a + b AS s FROM mixed_null_placement ORDER BY s ASC NULLS FIRST;
+------+------+------+
| a    | b    | s    |
+------+------+------+
| NULL | 1    | NULL |
| 3    | NULL | NULL |
| 1    | 2    | 3    |
| 2    | 4    | 6    |
+------+------+------+
4 row(s) fetched.
Elapsed 0.003 seconds.

> EXPLAIN SELECT a + b AS s FROM mixed_null_placement ORDER BY s;
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │          SortExec         │ |
|               | │    --------------------   │ |
|               | │     s@0 ASC NULLS LAST    │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │          files: 1         │ |
|               | │        format: csv        │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.
Elapsed 0.001 seconds.