refactor: add `try_to_proto` to `HashTableLookupExpr` by AnuragRaut08 · Pull Request #22451 · apache/datafusion

AnuragRaut08 · 2026-05-22T06:54:16Z

Which issue does this PR close?

What changes are included?

Adds try_to_proto to HashTableLookupExpr so it participates in the
expression-local serialization pattern introduced in #21929.

HashTableLookupExpr holds a runtime Arc<Map> that cannot be
serialized, so try_to_proto replaces it with lit(true). This is
safe because the filter is a performance optimisation only — lit(true)
passes all rows and the join produces correct results either way.

The centralized arm in to_proto.rs remains as a fallback for now.
Cleanup can follow in a separate PR once this lands.

Are these changes tested?

Yes — covered by the existing roundtrip_hash_table_lookup_expr_to_lit
test in datafusion/proto/tests/cases/roundtrip_physical_plan.rs.

Are there any user-facing changes?

No.

Part of apache#22435. Adds try_to_proto to HashTableLookupExpr so it participates in the expression-local serialization pattern introduced in apache#21929. HashTableLookupExpr holds a runtime Arc<Map> that cannot be serialized, so try_to_proto replaces it with lit(true). This is safe because the filter is a performance optimisation only — lit(true) passes all rows and the join produces correct results either way. The centralized arm in to_proto.rs remains as a fallback for now.

AnuragRaut08 · 2026-05-22T06:56:18Z

Thanks! CI is currently running. Please let me know if any adjustments are needed — happy to iterate quickly.

jayshrivastava · 2026-05-22T18:14:48Z

+    /// - `lit(true)` passes all rows so no valid rows are lost.
+    /// - In distributed execution the remote worker has no access to the
+    ///   build-side hash table anyway.
+    pub fn try_to_proto(


I think this must be moved to the impl PhysicalExpr for HashTableLookupExpr block to actually be used.

The way to test this code is being used is to delete the old paths in datafusion/datafusion/proto/src/physical_plan/to_proto.rs and datafusion/datafusion/proto/src/physical_plan/from_proto.rs.

Since this is a small PR, you may want to considering adding try_from_proto as well :)

jayshrivastava · 2026-05-22T18:28:36Z

+    /// - The filter is a performance optimisation, not a correctness requirement.
+    /// - `lit(true)` passes all rows so no valid rows are lost.
+    /// - In distributed execution the remote worker has no access to the
+    ///   build-side hash table anyway.


The comment comes from roundtrip_hash_table_lookup_expr_to_lit() in /datafusion/datafusion/proto/tests/cases/roundtrip_physical_plan.rs but I don't think it's very true.

IIUC, hash joins might build a HashTableLookupExpr during execution after the build side is done. These expressions get placed in the dynamic filter expr.

If you serialize before executing the plan, then there's no code path where there would be a HashTableLookupExpr in the plan today. If you deserialize and execute that plan, then the HashJoinExec may create a fresh HashTableLookupExpr for the dynamic filter. In this case, all the row pruning is preserved.

If you serialize after executing, then any potential HashTableLookupExpr would be replaced with lit(true). I don't think this has any impact. One must call reset_state to re-execute plans, in which case I would expect the HashTableLookupExpr to disappear. In this case, all the row pruning is preserved as well.

Maybe we can change the comment to explain these two cases? ^

Since HashTableLookupExpr is public it might be good to warn users that it does not get serialized. We could (a) file a ticket to track that HashTableLookupExpr are not serialized and (b) add a comment directly on HashTableLookupExpr.

kosiew

@AnuragRaut08
Thanks for working on this.
I agree with @jayshrivastava's point about moving this into the trait impl, and I also noticed a separate build issue around the proto wiring that looks blocking before merge.

kosiew · 2026-05-23T09:53:40Z

+    /// - `lit(true)` passes all rows so no valid rows are lost.
+    /// - In distributed execution the remote worker has no access to the
+    ///   build-side hash table anyway.
+    pub fn try_to_proto(


Building on Jay's comment, I think this needs to live in impl PhysicalExpr for HashTableLookupExpr rather than as an inherent HashTableLookupExpr::try_to_proto method.

The serializer calls expr.try_to_proto(&ctx) through &dyn PhysicalExpr, so this implementation is not reached. Serialization still depends on the centralized downcast fallback in datafusion/proto/src/physical_plan/to_proto.rs.

Could you move this into the PhysicalExpr impl with the same #[cfg(feature = "proto")] signature? That should make the expression-local serialization path actually take effect.

kosiew · 2026-05-23T09:53:40Z

+    /// - `lit(true)` passes all rows so no valid rows are lost.
+    /// - In distributed execution the remote worker has no access to the
+    ///   build-side hash table anyway.
+    pub fn try_to_proto(


I am also seeing a separate build issue here. This new signature references physical_expr::proto_encode, datafusion_proto_common, and datafusion_proto_models unconditionally, but datafusion-physical-plan does not expose those dependencies in a normal build.

cargo check -p datafusion-physical-plan currently fails with unresolved imports for proto_encode, datafusion_proto_common, and datafusion_proto_models.

Could you gate this consistently with the existing PhysicalExpr::try_to_proto API and wire the dependencies through the appropriate proto feature?

…pExpr

AnuragRaut08 · 2026-05-24T13:39:18Z

@jayshrivastava @kosiew — addressed the requested changes:

-moved try_to_proto into impl PhysicalExpr for HashTableLookupExprso serialization dispatches correctly through the trait object
-removed reliance on the centralized downcast serialization path
-wired the proto feature/dependencies through datafusion-physical-plan
-updated the serialization comment to explain the pre/post execution cases

Validated locally with:

-cargo check -p datafusion-physical-plan --features proto -cargo check -p datafusion-proto
-cargo test -p datafusion-proto roundtrip_hash_table_lookup_expr_to_lit`

The roundtrip test now passes through the trait-based serialization path.

kosiew

@AnuragRaut08
Thanks for the updates. No blocking findings from my follow-up review.

One small non-blocking suggestion below.

kosiew · 2026-05-25T10:01:08Z

+        // This is safe because dynamic filtering is a performance optimisation
+        // only — lit(true) passes all rows so correctness is preserved.
+        // When a serialized plan is re-executed, HashJoinExec reconstructs
+        // fresh dynamic filters at runtime anyway.


This comment is helpful and explains why lit(true) is correctness-safe. One small optional tweak would be to mirror Jay's pre-execution and post-execution wording a bit more directly, and maybe mention reset_state since that is where the runtime reconstruction behavior is tied together.

Not blocking from my side, just a readability suggestion.

AnuragRaut08 · 2026-05-25T17:31:14Z

Thanks for the suggestion @kosiew. I updated the comment to more clearly distinguish the pre-execution vs post-execution serialization behavior and added the reset_state() detail around runtime reconstruction.

github-actions Bot added the physical-plan Changes to the physical-plan crate label May 22, 2026

Anurag Tryambak Raut added 4 commits May 22, 2026 12:58

style: apply rustfmt

67bebc0

fix: remove invalid cfg(feature = "proto") in HashTableLookupExpr

d24abe8

fix: restore cfg(feature = proto) guard and apply cargo fmt

74f61fe

fix: remove invalid proto cfg gate from HashTableLookupExpr

63d14ff

jayshrivastava reviewed May 22, 2026

View reviewed changes

kosiew requested changes May 23, 2026

View reviewed changes

adriangb mentioned this pull request May 24, 2026

Port HashTableLookupExpr to use try_to_proto / try_from_proto #22435

Open

6 tasks

refactor: move try_to_proto into PhysicalExpr impl for HashTableLooku…

8ce71f5

…pExpr

github-actions Bot added the proto Related to proto crate label May 24, 2026

kosiew reviewed May 25, 2026

View reviewed changes

kosiew approved these changes May 25, 2026

View reviewed changes

docs: clarify HashTableLookupExpr serialization behavior

dd61b58

kosiew added this pull request to the merge queue May 26, 2026

github-merge-queue Bot removed this pull request from the merge queue due to no response for status checks May 26, 2026

kosiew added this pull request to the merge queue May 26, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 26, 2026

Dandandan added this pull request to the merge queue May 26, 2026

Merged via the queue into apache:main with commit eb3c564 May 26, 2026
39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: add `try_to_proto` to `HashTableLookupExpr`#22451

refactor: add `try_to_proto` to `HashTableLookupExpr`#22451
Dandandan merged 7 commits into
apache:mainfrom
AnuragRaut08:feat/port-hash-table-lookup-expr-proto

AnuragRaut08 commented May 22, 2026

Uh oh!

AnuragRaut08 commented May 22, 2026

Uh oh!

jayshrivastava May 22, 2026 •

edited

Loading

Uh oh!

jayshrivastava May 22, 2026 •

edited

Loading

Uh oh!

kosiew left a comment •

edited

Loading

Uh oh!

kosiew May 23, 2026

Uh oh!

kosiew May 23, 2026

Uh oh!

AnuragRaut08 commented May 24, 2026

Uh oh!

kosiew left a comment •

edited

Loading

Uh oh!

kosiew May 25, 2026

Uh oh!

AnuragRaut08 commented May 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

AnuragRaut08 commented May 22, 2026

Which issue does this PR close?

What changes are included?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

AnuragRaut08 commented May 22, 2026

Uh oh!

jayshrivastava May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jayshrivastava May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew May 23, 2026

Choose a reason for hiding this comment

Uh oh!

kosiew May 23, 2026

Choose a reason for hiding this comment

Uh oh!

AnuragRaut08 commented May 24, 2026

Uh oh!

kosiew left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kosiew May 25, 2026

Choose a reason for hiding this comment

Uh oh!

AnuragRaut08 commented May 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jayshrivastava May 22, 2026 •

edited

Loading

jayshrivastava May 22, 2026 •

edited

Loading

kosiew left a comment •

edited

Loading

kosiew left a comment •

edited

Loading