Skip to content

allow expressions to pass through RepartitionExec #19365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
adriangb wants to merge 3 commits into
base: main
Choose a base branch
from
Open

allow expressions to pass through RepartitionExec #19365

adriangb wants to merge 3 commits into from
+143 −216

Conversation

@adriangb
Copy link
Contributor

@adriangb adriangb commented Dec 16, 2025 •
edited
Loading

I don't see any reason for these limitations.
It can be incredibly beneficial to push down non-column expressions, I don't think the logic of benefits / does not benefit from re-partitioning is sound.
Maybe this made more sense when e.g. ParquetSource couldn't push down struct field access, couldn't optimize away data casts, etc?
This behavior was introduced in #14235, @buraksenn @berkaysynnada if you have any context please share.

@github-actions github-actions bot added physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt) labels Dec 16, 2025
@adriangb adriangb force-pushed the repartition-exec-projection-swap branch from 6572ff5 to 966491d Compare December 17, 2025 04:23
Dandandan
Dandandan reviewed Dec 17, 2025
View reviewed changes
datafusion/physical-plan/src/repartition/mod.rs
&self,
projection: &ProjectionExec,
) -> Result<Option<Arc<dyn ExecutionPlan>>> {
// If the projection does not narrow the schema, we should not try to push it down.
Copy link
Contributor

@Dandandan Dandandan Dec 17, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this still shows up in the plans, in some cases the number of fields is bigger before repartitition because of this (and might hurt performance as RepartitionExec copies the input columns (currently twice)). Can we instead relax the conditition to > instead of >= so it will be pushed down if it also keeps the number of fields equal?

@Dandandan
Copy link
Contributor

Dandandan commented Dec 17, 2025

run benchmarks

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 17, 2025

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing repartition-exec-projection-swap (7759d7b) to 2b67ba8 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@adriangb
Copy link
Contributor Author

adriangb commented Dec 17, 2025

We discussed in the meeting today.
The conclusion was basically that the heuristic that is currently in place is trying to answer the question "is this expression more or less expensive to evaluate than it's constituent parts".
For example, if the projection adds columns a,b,a+b that is true, if it removes columns a,b instead of a,b,c it's not.
However this is broken for things like struct field access or parquet variant field access where the expression large_struct.small_int_field might be considerably cheaper to evaluate than it's constituent parts (i.e. reading a single field from a struct is cheaper than reading the whole thing and then slicing it).
The most promising approach we discussed was adding a method to PhysicalExpr to let the expression determine if it is cheaper or more expensive than it's constituent parts.
There are other options, I'll open an issue to discuss.

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 17, 2025

🤖: Benchmark completed

Details 

Comparing HEAD and repartition-exec-projection-swap
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ repartition-exec-projection-swap ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2840.48 ms │                       2686.98 ms │ +1.06x faster │
│ QQuery 1     │  1239.76 ms │                       1180.19 ms │     no change │
│ QQuery 2     │  2337.11 ms │                       2235.83 ms │     no change │
│ QQuery 3     │  1157.14 ms │                       1141.77 ms │     no change │
│ QQuery 4     │  2321.77 ms │                       2278.91 ms │     no change │
│ QQuery 5     │ 29088.66 ms │                      28690.61 ms │     no change │
│ QQuery 6     │  4028.89 ms │                       4025.35 ms │     no change │
│ QQuery 7     │  3622.20 ms │                       3550.95 ms │     no change │
└──────────────┴─────────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                               ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                               │ 46636.01ms │
│ Total Time (repartition-exec-projection-swap)   │ 45790.59ms │
│ Average Time (HEAD)                             │  5829.50ms │
│ Average Time (repartition-exec-projection-swap) │  5723.82ms │
│ Queries Faster                                  │          1 │
│ Queries Slower                                  │          0 │
│ Queries with No Change                          │          7 │
│ Queries with Failure                            │          0 │
└─────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ repartition-exec-projection-swap ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.61 ms │                          2.56 ms │     no change │
│ QQuery 1     │    51.81 ms │                         51.42 ms │     no change │
│ QQuery 2     │   135.08 ms │                        135.85 ms │     no change │
│ QQuery 3     │   156.96 ms │                        159.16 ms │     no change │
│ QQuery 4     │  1091.22 ms │                       1227.73 ms │  1.13x slower │
│ QQuery 5     │  1491.44 ms │                       1583.53 ms │  1.06x slower │
│ QQuery 6     │     2.03 ms │                          2.26 ms │  1.11x slower │
│ QQuery 7     │    56.91 ms │                         61.22 ms │  1.08x slower │
│ QQuery 8     │  1469.91 ms │                       1544.11 ms │  1.05x slower │
│ QQuery 9     │  1870.62 ms │                       1973.73 ms │  1.06x slower │
│ QQuery 10    │   372.86 ms │                        372.76 ms │     no change │
│ QQuery 11    │   417.83 ms │                        429.44 ms │     no change │
│ QQuery 12    │  1366.19 ms │                       1511.92 ms │  1.11x slower │
│ QQuery 13    │  2039.67 ms │                       2131.77 ms │     no change │
│ QQuery 14    │  1270.21 ms │                       1337.92 ms │  1.05x slower │
│ QQuery 15    │  1216.67 ms │                       1349.11 ms │  1.11x slower │
│ QQuery 16    │  2645.72 ms │                       2785.32 ms │  1.05x slower │
│ QQuery 17    │  2626.45 ms │                       2766.10 ms │  1.05x slower │
│ QQuery 18    │  5567.30 ms │                       5054.93 ms │ +1.10x faster │
│ QQuery 19    │   129.28 ms │                        127.67 ms │     no change │
│ QQuery 20    │  1963.74 ms │                       1922.43 ms │     no change │
│ QQuery 21    │  2264.13 ms │                       2215.85 ms │     no change │
│ QQuery 22    │  4173.48 ms │                       3788.04 ms │ +1.10x faster │
│ QQuery 23    │ 19389.31 ms │                      12417.19 ms │ +1.56x faster │
│ QQuery 24    │   230.44 ms │                        224.62 ms │     no change │
│ QQuery 25    │   494.03 ms │                        457.99 ms │ +1.08x faster │
│ QQuery 26    │   227.41 ms │                        214.61 ms │ +1.06x faster │
│ QQuery 27    │  2848.47 ms │                       2791.87 ms │     no change │
│ QQuery 28    │ 23844.65 ms │                      23548.77 ms │     no change │
│ QQuery 29    │   969.27 ms │                        944.55 ms │     no change │
│ QQuery 30    │  1342.46 ms │                       1381.74 ms │     no change │
│ QQuery 31    │  1408.55 ms │                       1360.04 ms │     no change │
│ QQuery 32    │  5005.00 ms │                       4745.24 ms │ +1.05x faster │
│ QQuery 33    │  6055.20 ms │                       5930.96 ms │     no change │
│ QQuery 34    │  6123.53 ms │                       6267.73 ms │     no change │
│ QQuery 35    │  2009.27 ms │                       2008.01 ms │     no change │
│ QQuery 36    │    65.69 ms │                         69.25 ms │  1.05x slower │
│ QQuery 37    │    46.99 ms │                         45.49 ms │     no change │
│ QQuery 38    │    67.15 ms │                         65.93 ms │     no change │
│ QQuery 39    │   103.02 ms │                        108.67 ms │  1.05x slower │
│ QQuery 40    │    28.01 ms │                         28.54 ms │     no change │
│ QQuery 41    │    23.14 ms │                         25.48 ms │  1.10x slower │
│ QQuery 42    │    20.25 ms │                         20.72 ms │     no change │
└──────────────┴─────────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                               ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                               │ 102683.97ms │
│ Total Time (repartition-exec-projection-swap)   │  95192.23ms │
│ Average Time (HEAD)                             │   2388.00ms │
│ Average Time (repartition-exec-projection-swap) │   2213.77ms │
│ Queries Faster                                  │           6 │
│ Queries Slower                                  │          14 │
│ Queries with No Change                          │          23 │
│ Queries with Failure                            │           0 │
└─────────────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ repartition-exec-projection-swap ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 132.92 ms │                        131.11 ms │     no change │
│ QQuery 2     │  30.03 ms │                         30.48 ms │     no change │
│ QQuery 3     │  40.03 ms │                         36.96 ms │ +1.08x faster │
│ QQuery 4     │  29.50 ms │                         29.18 ms │     no change │
│ QQuery 5     │  89.10 ms │                         90.10 ms │     no change │
│ QQuery 6     │  20.13 ms │                         20.04 ms │     no change │
│ QQuery 7     │ 236.03 ms │                        240.67 ms │     no change │
│ QQuery 8     │  38.93 ms │                         38.79 ms │     no change │
│ QQuery 9     │ 100.04 ms │                        106.81 ms │  1.07x slower │
│ QQuery 10    │  65.91 ms │                         64.84 ms │     no change │
│ QQuery 11    │  17.47 ms │                         19.38 ms │  1.11x slower │
│ QQuery 12    │  53.68 ms │                         52.42 ms │     no change │
│ QQuery 13    │  49.84 ms │                         51.77 ms │     no change │
│ QQuery 14    │  15.28 ms │                         13.99 ms │ +1.09x faster │
│ QQuery 15    │  25.01 ms │                         23.77 ms │     no change │
│ QQuery 16    │  24.79 ms │                         24.82 ms │     no change │
│ QQuery 17    │ 155.65 ms │                        159.10 ms │     no change │
│ QQuery 18    │ 292.45 ms │                        288.87 ms │     no change │
│ QQuery 19    │  37.42 ms │                         38.37 ms │     no change │
│ QQuery 20    │  50.51 ms │                         49.97 ms │     no change │
│ QQuery 21    │ 328.39 ms │                        326.80 ms │     no change │
│ QQuery 22    │  18.17 ms │                         17.68 ms │     no change │
└──────────────┴───────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                               ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                               │ 1851.28ms │
│ Total Time (repartition-exec-projection-swap)   │ 1855.91ms │
│ Average Time (HEAD)                             │   84.15ms │
│ Average Time (repartition-exec-projection-swap) │   84.36ms │
│ Queries Faster                                  │         2 │
│ Queries Slower                                  │         2 │
│ Queries with No Change                          │        18 │
│ Queries with Failure                            │         0 │
└─────────────────────────────────────────────────┴───────────┘

@alamb
Copy link
Contributor

alamb commented Dec 17, 2025

run benchmarks

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 17, 2025

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing repartition-exec-projection-swap (7759d7b) to 2b67ba8 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 18, 2025

🤖: Benchmark completed

Details 

Comparing HEAD and repartition-exec-projection-swap
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ repartition-exec-projection-swap ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2754.37 ms │                       2763.91 ms │     no change │
│ QQuery 1     │  1224.37 ms │                       1222.34 ms │     no change │
│ QQuery 2     │  2327.60 ms │                       2364.70 ms │     no change │
│ QQuery 3     │  1199.44 ms │                       1159.69 ms │     no change │
│ QQuery 4     │  2283.38 ms │                       2314.42 ms │     no change │
│ QQuery 5     │ 29261.13 ms │                      29361.39 ms │     no change │
│ QQuery 6     │  4112.78 ms │                       4036.08 ms │     no change │
│ QQuery 7     │  3931.12 ms │                       3731.86 ms │ +1.05x faster │
└──────────────┴─────────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                               ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                               │ 47094.18ms │
│ Total Time (repartition-exec-projection-swap)   │ 46954.39ms │
│ Average Time (HEAD)                             │  5886.77ms │
│ Average Time (repartition-exec-projection-swap) │  5869.30ms │
│ Queries Faster                                  │          1 │
│ Queries Slower                                  │          0 │
│ Queries with No Change                          │          7 │
│ Queries with Failure                            │          0 │
└─────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ repartition-exec-projection-swap ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.23 ms │                          2.38 ms │  1.07x slower │
│ QQuery 1     │    50.94 ms │                         51.32 ms │     no change │
│ QQuery 2     │   138.30 ms │                        138.20 ms │     no change │
│ QQuery 3     │   155.84 ms │                        153.73 ms │     no change │
│ QQuery 4     │  1149.65 ms │                       1148.51 ms │     no change │
│ QQuery 5     │  1505.46 ms │                       1533.06 ms │     no change │
│ QQuery 6     │     2.10 ms │                          2.05 ms │     no change │
│ QQuery 7     │    57.24 ms │                         59.55 ms │     no change │
│ QQuery 8     │  1475.04 ms │                       1469.12 ms │     no change │
│ QQuery 9     │  1898.74 ms │                       1958.68 ms │     no change │
│ QQuery 10    │   372.36 ms │                        371.88 ms │     no change │
│ QQuery 11    │   428.42 ms │                        427.50 ms │     no change │
│ QQuery 12    │  1375.71 ms │                       1379.38 ms │     no change │
│ QQuery 13    │  2045.25 ms │                       2049.84 ms │     no change │
│ QQuery 14    │  1272.71 ms │                       1294.44 ms │     no change │
│ QQuery 15    │  1280.91 ms │                       1280.95 ms │     no change │
│ QQuery 16    │  2726.83 ms │                       2681.93 ms │     no change │
│ QQuery 17    │  2699.63 ms │                       2736.94 ms │     no change │
│ QQuery 18    │  5619.65 ms │                       4981.69 ms │ +1.13x faster │
│ QQuery 19    │   123.66 ms │                        122.55 ms │     no change │
│ QQuery 20    │  1913.69 ms │                       1918.62 ms │     no change │
│ QQuery 21    │  2230.53 ms │                       2210.76 ms │     no change │
│ QQuery 22    │  3803.98 ms │                       3777.10 ms │     no change │
│ QQuery 23    │ 12557.92 ms │                      12193.07 ms │     no change │
│ QQuery 24    │   213.81 ms │                        208.52 ms │     no change │
│ QQuery 25    │   474.44 ms │                        480.03 ms │     no change │
│ QQuery 26    │   241.63 ms │                        227.73 ms │ +1.06x faster │
│ QQuery 27    │  2798.44 ms │                       2759.81 ms │     no change │
│ QQuery 28    │ 23685.63 ms │                      23708.62 ms │     no change │
│ QQuery 29    │   972.25 ms │                        976.77 ms │     no change │
│ QQuery 30    │  1361.57 ms │                       1319.12 ms │     no change │
│ QQuery 31    │  1365.84 ms │                       1341.59 ms │     no change │
│ QQuery 32    │  5462.39 ms │                       5324.64 ms │     no change │
│ QQuery 33    │  6240.03 ms │                       5906.89 ms │ +1.06x faster │
│ QQuery 34    │  6318.35 ms │                       6292.16 ms │     no change │
│ QQuery 35    │  1987.13 ms │                       1978.48 ms │     no change │
│ QQuery 36    │    67.65 ms │                         68.71 ms │     no change │
│ QQuery 37    │    48.26 ms │                         46.95 ms │     no change │
│ QQuery 38    │    68.72 ms │                         67.46 ms │     no change │
│ QQuery 39    │   105.82 ms │                        109.17 ms │     no change │
│ QQuery 40    │    27.72 ms │                         26.79 ms │     no change │
│ QQuery 41    │    24.43 ms │                         24.82 ms │     no change │
│ QQuery 42    │    21.18 ms │                         21.05 ms │     no change │
└──────────────┴─────────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                               ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                               │ 96372.06ms │
│ Total Time (repartition-exec-projection-swap)   │ 94832.54ms │
│ Average Time (HEAD)                             │  2241.21ms │
│ Average Time (repartition-exec-projection-swap) │  2205.41ms │
│ Queries Faster                                  │          3 │
│ Queries Slower                                  │          1 │
│ Queries with No Change                          │         39 │
│ Queries with Failure                            │          0 │
└─────────────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ repartition-exec-projection-swap ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 134.45 ms │                        133.43 ms │     no change │
│ QQuery 2     │  29.84 ms │                         29.64 ms │     no change │
│ QQuery 3     │  39.41 ms │                         40.53 ms │     no change │
│ QQuery 4     │  29.94 ms │                         30.21 ms │     no change │
│ QQuery 5     │  89.62 ms │                         91.65 ms │     no change │
│ QQuery 6     │  19.88 ms │                         19.95 ms │     no change │
│ QQuery 7     │ 238.36 ms │                        256.37 ms │  1.08x slower │
│ QQuery 8     │  37.29 ms │                         36.29 ms │     no change │
│ QQuery 9     │ 108.42 ms │                        111.93 ms │     no change │
│ QQuery 10    │  68.09 ms │                         64.47 ms │ +1.06x faster │
│ QQuery 11    │  19.79 ms │                         17.26 ms │ +1.15x faster │
│ QQuery 12    │  52.99 ms │                         53.36 ms │     no change │
│ QQuery 13    │  49.33 ms │                         49.40 ms │     no change │
│ QQuery 14    │  14.22 ms │                         14.57 ms │     no change │
│ QQuery 15    │  24.81 ms │                         25.25 ms │     no change │
│ QQuery 16    │  25.28 ms │                         25.69 ms │     no change │
│ QQuery 17    │ 156.12 ms │                        160.40 ms │     no change │
│ QQuery 18    │ 291.56 ms │                        294.02 ms │     no change │
│ QQuery 19    │  38.85 ms │                         38.34 ms │     no change │
│ QQuery 20    │  52.13 ms │                         50.93 ms │     no change │
│ QQuery 21    │ 341.07 ms │                        334.08 ms │     no change │
│ QQuery 22    │  17.39 ms │                         17.89 ms │     no change │
└──────────────┴───────────┴──────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                               ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                               │ 1878.83ms │
│ Total Time (repartition-exec-projection-swap)   │ 1895.67ms │
│ Average Time (HEAD)                             │   85.40ms │
│ Average Time (repartition-exec-projection-swap) │   86.17ms │
│ Queries Faster                                  │         2 │
│ Queries Slower                                  │         1 │
│ Queries with No Change                          │        19 │
│ Queries with Failure                            │         0 │
└─────────────────────────────────────────────────┴───────────┘

kosiew
kosiew reviewed Dec 29, 2025
View reviewed changes
Copy link
Contributor

@kosiew kosiew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@adriangb

Thanks for working on this

I think adding unit tests to validate:

  • Projections with computed expressions (binary operators) can be swapped through repartition
  • Hash partition expressions are correctly preserved after swap

would improve test coverage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Dandandan Dandandan Dandandan left review comments

@kosiew kosiew kosiew left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

physical-plan Changes to the physical-plan crate sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@adriangb @Dandandan @alamb-ghbot @alamb @kosiew