feat: logical plan protobuf representation for range repartitioning by saadtajwar · Pull Request #23030 · apache/datafusion

saadtajwar · 2026-06-19T00:42:18Z

Which issue does this PR close?

Closes Support logical protobuf serialization for range repartitioning #22787

Rationale for this change

The range repartitioning scheme for logical plans does not currently have a protobuf representation.

What changes are included in this PR?

A protobuf representation of the RangeRepartition struct was added to datafusion.proto, and the codegened Rust types were created. Added logic for serializing and deserializing to and from the protobuf representation, and a roundtrip test as well!

Are these changes tested?

Yes! Added a test in roundtrip_logical_plan

Are there any user-facing changes?

No, adding internal protobuf serialization support for an existing logical plan variant

saadtajwar · 2026-06-19T00:44:21Z

    Error::General(message.into())
 }
+
+pub fn parse_protobuf_range_split_point(


Copied this & serialize_range_split_point from the physical plan logic by @gene-bordegaray ! 😁

Merge new MergeInto/DML imports from upstream with range partitioning additions.

saadtajwar · 2026-06-19T00:47:44Z

@gene-bordegaray tagging you in here - please let me know your thoughts! Looking forward to any feedback & working together further on this effort! 😁 🎉

gene-bordegaray

overall looking good 😄 , thank you for this work. Left some minor comments 👍

…ames

…epartitioning

saadtajwar · 2026-06-21T21:18:00Z

@gene-bordegaray pushed changes to address your comments - let me know your thoughts! Really appreciate the time taken to review!

gene-bordegaray

This looks good to me now 👍

Thank you!

For next reviewer, there is some duplication in the helpers parse_protobuf_range_split_point / serialize_range_split_point across logical and physical but think it is ok for now. If another use pops up we could make some common helper for this.

cc: @gabotechs

alamb

Thank you @saadtajwar and @gene-bordegaray -- looks good to me

alamb · 2026-06-22T12:58:02Z

+        Field::new("ts", DataType::Int64, false),
+        Field::new("region", DataType::Utf8, false),
+    ]));
+    let table = Arc::new(datafusion::datasource::empty::EmptyTable::new(Arc::clone(


as a minor nit, datafusion::datasource::empty::EmptyTable: is pretty hard on the eyes. It would be nicer if there was a single use datafusion::datasource::empty::EmptyTable; at the top and then used down here (and the other tests)

Done! Thank you! Is there anything needed on my end to get this in? Appreciate you taking the time to review this!

github-actions · 2026-06-22T13:02:26Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-proto v54.0.0 (current)
       Built [  57.761s] (current)
     Parsing datafusion-proto v54.0.0 (current)
      Parsed [   0.020s] (current)
    Building datafusion-proto v54.0.0 (baseline)
       Built [  57.413s] (baseline)
     Parsing datafusion-proto v54.0.0 (baseline)
      Parsed [   0.020s] (baseline)
    Checking datafusion-proto v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.379s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 117.803s] datafusion-proto
    Building datafusion-proto-models v54.0.0 (current)
       Built [  23.487s] (current)
     Parsing datafusion-proto-models v54.0.0 (current)
      Parsed [   0.134s] (current)
    Building datafusion-proto-models v54.0.0 (baseline)
       Built [  23.482s] (baseline)
     Parsing datafusion-proto-models v54.0.0 (baseline)
      Parsed [   0.134s] (baseline)
    Checking datafusion-proto-models v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   2.494s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_variant_added.ron

Failed in:
  variant PartitionMethod:Range in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:225
  variant PartitionMethod:Range in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:225

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  51.204s] datafusion-proto-models

…epartitioning

saadtajwar · 2026-06-22T22:54:00Z

@Dandandan & @alamb & @gene-bordegaray - thanks so much again for helping get this in team! Super excited to continue contributing to Datafusion & please let me know if there's anything else on this specific initiative I can help out with! 😁 🎉

gene-bordegaray · 2026-06-22T23:45:45Z

@Dandandan & @alamb & @gene-bordegaray - thanks so much again for helping get this in team! Super excited to continue contributing to Datafusion & please let me know if there's anything else on this specific initiative I can help out with! 😁 🎉

No problem, I should be thanking you for great work. I have other related issues up for range partitioning. Feel free to pick up any you want to take on. I am more than happy to review and answer questions

saadtajwar added 6 commits June 17, 2026 14:41

feat: supporting logical protobuf serialization for range repartitioning

39186d0

serialization to proto type and used in mod.rs

5e7e9ec

from proto

66fc007

roundtrip test added

7542d1c

quick lint fixes

d368072

lint again

12515b0

github-actions Bot added the proto Related to proto crate label Jun 19, 2026

saadtajwar commented Jun 19, 2026

View reviewed changes

Comment thread datafusion/proto-models/proto/datafusion.proto Outdated

saadtajwar commented Jun 19, 2026

View reviewed changes

Comment thread datafusion/proto/tests/cases/roundtrip_logical_plan.rs Outdated

Resolve merge conflicts with upstream/main

89afee6

Merge new MergeInto/DML imports from upstream with range partitioning additions.

saadtajwar mentioned this pull request Jun 19, 2026

Support logical protobuf serialization for range repartitioning #22787

Closed

gene-bordegaray reviewed Jun 21, 2026

View reviewed changes

saadtajwar and others added 4 commits June 21, 2026 16:54

regened proto with new field names

71cbbc7

helper func visibility change + code changed to use new proto field n…

ad24930

…ames

Merge branch 'main' into saadt/logical-protobuf-serialization-range-r…

ec12119

…epartitioning

new tests added

8b29312

gene-bordegaray approved these changes Jun 22, 2026

View reviewed changes

alamb approved these changes Jun 22, 2026

View reviewed changes

github-actions Bot added the auto detected api change Auto detected API change label Jun 22, 2026

saadtajwar and others added 2 commits June 22, 2026 10:39

import emptytable

3b27af1

Merge branch 'main' into saadt/logical-protobuf-serialization-range-r…

ef9efd0

…epartitioning

Dandandan approved these changes Jun 22, 2026

View reviewed changes

Dandandan added this pull request to the merge queue Jun 22, 2026

Merged via the queue into apache:main with commit a1f56b7 Jun 22, 2026
35 checks passed

Uh oh!

Conversation

saadtajwar commented Jun 19, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Uh oh!

saadtajwar Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

saadtajwar commented Jun 19, 2026

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

saadtajwar commented Jun 21, 2026

Uh oh!

gene-bordegaray left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

alamb Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

saadtajwar Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

saadtajwar commented Jun 22, 2026

Uh oh!

gene-bordegaray commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Jun 22, 2026 •

edited

Loading