Skip to content

[SPARK-58209][SQL] Mark stateful expressions to prevent data races#57371

Draft
huanghanyu-vungle wants to merge 4 commits into
apache:masterfrom
huanghanyu-vungle:codex/spark-58209-stateful-expressions
Draft

[SPARK-58209][SQL] Mark stateful expressions to prevent data races#57371
huanghanyu-vungle wants to merge 4 commits into
apache:masterfrom
huanghanyu-vungle:codex/spark-58209-stateful-expressions

Conversation

@huanghanyu-vungle

Copy link
Copy Markdown

What changes were proposed in this pull request?

This PR marks XPathExtract, CallMethodViaReflection, and
UserDefinedGenerator as stateful Catalyst expressions.

It also adds focused regression tests verifying that
freshCopyIfContainsStatefulExpression() returns a distinct expression instance while preserving
the expression's children and configuration.

Why are the changes needed?

These expressions retain mutable runtime state, including an XPath evaluator, a reusable
reflection argument buffer, and generator projections/converters. They currently inherit
stateful = false, so independent query executions can reuse the same expression instance.
Concurrent executor tasks may then race on the shared state and silently produce incorrect
results.

Declaring the expressions stateful makes Catalyst create a fresh, uninitialized expression for
each execution path using its existing fresh-copy mechanism.

Does this PR introduce any user-facing change?

Yes. Concurrent query executions no longer risk incorrect results caused by shared mutable state
in these expressions. There is no SQL API or syntax change.

How was this patch tested?

Added fresh-copy regression coverage to XPathExpressionSuite,
CallMethodViaReflectionSuite, and GeneratorExpressionSuite.

./build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.xml.XPathExpressionSuite org.apache.spark.sql.catalyst.expressions.CallMethodViaReflectionSuite org.apache.spark.sql.catalyst.expressions.GeneratorExpressionSuite"
./build/sbt "catalyst/scalastyle"

The three suites passed 27 tests, and Catalyst Scalastyle processed 705 files with no findings.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex (GPT-5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant