perf: reuse projection schema in OptimizeProjections instead of recomputing it

[zhuqi-lucas]

Which issue does this close?

Closes #24264. Answers #24284 in the process.

Rationale for this change

rewrite_projection_given_requirements, the core of OptimizeProjections, prunes a projection's expressions to the subset actually required and then rebuilds it with Projection::try_new. try_new recomputes the output schema via projection_schema, calling Expr::to_field for every retained expression; column resolution is a linear scan over the input schema, so this is O(exprs * schema_width) per projection, per pass. The retained expressions are a subset of the ones the projection already has, so the answer is already sitting in proj.schema.

Reusing it turned out not to be safe as-is, which is what the first revision of this PR got wrong and what #24284 is about.

LogicalPlan::map_expressions replaces a projection's expressions while keeping its existing schema, so SimplifyExpressions can leave the two out of step. Constant folding turns arrow_cast([...], 'LargeList(...)'), a function call whose field the planner derived as nullable, into a non-null literal whose field is not, while the schema keeps the pre-folding answer. OptimizeProjections calling try_new was quietly normalising that back, so whether a stale schema survived into the final plan depended on which rules happened to fire.

So this PR fixes that first, then does the optimisation.

What changes are included in this PR?

1. SimplifyExpressions derives the projection schema after rewriting (simplify_exprs.rs), and only when the expressions actually changed.

Final plans are unchanged: the normalisation OptimizeProjections was performing simply happens earlier now. Nothing in the tree needed updating, no snapshot and no expected plan, which is the clearest evidence this is an equivalence rather than a behaviour change.

2. rewrite_projection_given_requirements derives the pruned schema by selecting the already-computed fields from the existing projection schema (project_schema_by_indices) and builds with Projection::try_new_with_schema. When nothing is pruned, the existing Arc is reused as-is. Functional dependencies are projected through the kept indices.

Cost goes from O(exprs * width) to O(k), and to O(1) when nothing is pruned. Unlike making the recompute cheaper, this removes the work rather than speeding it up: no to_field call, no field allocation, no cache and no heuristics.

Correctness

With (1) in place, proj.schema is in step with proj.expr, so slicing it at the retained indices produces exactly what projection_schema would recompute: field i corresponds to expression i, and RequiredIndices yields a sorted, deduplicated subset.

project_schema_by_indices_matches_recompute asserts that, for a mixed expression list (plain column, computed binary expr, alias, NULL literal, qualified column) and every representative index subset, the sliced schema matches projection_schema on fields, qualifiers, schema metadata and functional dependencies, and that the identity subset reuses the same Arc.

The two failures the first revision of this PR introduced are fixed by (1), not worked around:

• roundtrip_literal_list, roundtrip_literal_struct, roundtrip_literal_named_struct, roundtrip_literal_renamed_struct in datafusion-substrait, which compare plan schemas across a roundtrip
• schema_evolution_nested.slt, where the projection feeds COPY (SELECT ...) TO ... STORED AS PARQUET, so a stale nullability reached the written file and DESCRIBE reported YES instead of NO

Full local runs: datafusion-substrait 49 + 200 + 3, datafusion-optimizer 760 + 26 + 5, datafusion-expr 248 + 55, datafusion-common 547, datafusion-sql 88 + 572 + 12, and schema_evolution_nested.slt 1/1. All green, with no test or snapshot modified.

Are there any user-facing changes?

No. Optimized plans are unchanged.

[Copilot]

Pull request overview

Improves OptimizeProjections performance by avoiding repeated recomputation of projection output schemas when pruning projection expressions, instead reusing/slicing the already-computed Projection.schema.

Changes:

• Update rewrite_projection_given_requirements to build pruned projections via Projection::try_new_with_schema using a sliced schema from the existing projection schema.
• Add project_schema_by_indices helper to project fields + functional dependencies while reusing schema metadata.
• Add a unit test validating that sliced schemas match projection_schema recomputation across representative index subsets.

Suppressed comments (1)

datafusion/optimizer/src/optimize_projections/mod.rs:1285

• project_schema_by_indices also projects functional dependencies and preserves schema-level metadata, but the test currently only compares fields and qualifiers. Adding assertions for functional dependencies and schema metadata will better protect the behavior this PR relies on.

            // Output fields (name, data type, nullability, field metadata) must
            // match the from-scratch computation exactly.
            assert_eq!(
                reused.fields(),
                recomputed.fields(),
                "fields differ for indices {indices:?}"
            );

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.62500% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.13%. Comparing base (186f96f) to head (75deafe).

Files with missing lines	Patch %	Lines
...tafusion/optimizer/src/optimize_projections/mod.rs	90.00%	0 Missing and 8 partials ⚠️
...timizer/src/simplify_expressions/simplify_exprs.rs	93.75%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #24281   +/-   ##
=======================================
  Coverage   81.13%   81.13%           
=======================================
  Files        1112     1112           
  Lines      386716   386802   +86     
  Branches   386716   386802   +86     
=======================================
+ Hits       313765   313839   +74     
- Misses      54479    54481    +2     
- Partials    18472    18482   +10     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[zhuqi-lucas]

run benchmark sql_planner

[adriangbot]

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c5277634733-1581-7t54h 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing optimize-projections-reuse-schema (75deafe) to 186f96f (merge-base) diff

Run configuration

run benchmark sql_planner

Results will be posted here when complete

---

File an issue against this benchmark runner