Skip to content

[SPARK-58095][SQL] approx_top_k_combine throws scala.MatchError on empty input#57289

Open
jiwen624 wants to merge 2 commits into
apache:masterfrom
jiwen624:SPARK-58095
Open

[SPARK-58095][SQL] approx_top_k_combine throws scala.MatchError on empty input#57289
jiwen624 wants to merge 2 commits into
apache:masterfrom
jiwen624:SPARK-58095

Conversation

@jiwen624

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Fix the MatchError on empty input with changes below:

  • Fall back to the statically-known item data type carried by the state struct (uncheckedItemDataType) when no input sketch resolved it, in both eval and serialize.
  • When combine size is unspecified and no input was consumed, maxItemsTracked is still the internal VOID sentinel (-1); resolve it to the default in eval only (a partial buffer must keep VOID for the final merge), so the empty output sketch carries a valid size that downstream functions such as approx_top_k_estimate accept.

Why are the changes needed?

The query aborts with an internal error instead of returning an empty result, unlike the sibling functions (approx_top_k, approx_top_k_accumulate, approx_top_k_estimate) which handle empty input correctly:

SELECT approx_top_k_combine(sketch, 5)
FROM (SELECT approx_top_k_accumulate(expr) AS sketch
      FROM VALUES (1), (2) AS t(expr))
WHERE false;
-- before: org.apache.spark.SparkException ... scala.MatchError: null
-- after:  returns a single empty sketch (estimates to [])

Does this PR introduce any user-facing change?

Yes. Fixed an query failing issue when the input is empty.

How was this patch tested?

Added UT case

Was this patch authored or co-authored using generative AI tooling?

Yes.

@jiwen624 jiwen624 changed the title [SPARK-58095][SQL] Fix approx_top_k_combine MatchError on empty input [SPARK-58095][SQL] approx_top_k_combine throws MatchError on empty input Jul 16, 2026
@jiwen624 jiwen624 changed the title [SPARK-58095][SQL] approx_top_k_combine throws MatchError on empty input [SPARK-58095][SQL] approx_top_k_combine throws scala.MatchError on empty input Jul 16, 2026

@uros-b uros-b left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jiwen624

jiwen624 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

@uros-b @cloud-fan ah it seems I forgot to move it out of draft. I see there is a problematic scenario covered by this and currently not by the one above. I’m fine with either addressing it separately here or combining the fixes.

@jiwen624
jiwen624 marked this pull request as ready for review July 22, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants