Skip to content

[SPARK-42052][SQL] Codegen Support for HiveSimpleUDF#39865

Closed
panbingkun wants to merge 3 commits into
apache:masterfrom
panbingkun:SPARK-42052
Closed

[SPARK-42052][SQL] Codegen Support for HiveSimpleUDF#39865
panbingkun wants to merge 3 commits into
apache:masterfrom
panbingkun:SPARK-42052

Conversation

@panbingkun

@panbingkun panbingkun commented Feb 3, 2023

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

As a subtask of SPARK-42050, this PR adds Codegen Support for HiveSimpleUDF

Why are the changes needed?

Improve codegen coverage and performance

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New UT & Pass GA.

@github-actions github-actions Bot added the SQL label Feb 3, 2023
@HyukjinKwon

Copy link
Copy Markdown
Member

Hm, I think we should better go and figure out to resolve this by using Invoke so we don't have to manually implement the codegen logic. I was fine with #39555 as a one time thing but seems like there are some more to fix.

@HyukjinKwon

Copy link
Copy Markdown
Member

cc @cloud-fan @yaooqinn @dongjoon-hyun FYI

@dongjoon-hyun

Copy link
Copy Markdown
Member

I agree with @HyukjinKwon 's comment too.

@panbingkun

panbingkun commented Feb 9, 2023

Copy link
Copy Markdown
Contributor Author

Hm, I think we should better go and figure out to resolve this by using Invoke so we don't have to manually implement the codegen logic. I was fine with #39555 as a one time thing but seems like there are some more to fix.

As the first step, I have submitted a new Pr(#39949) to rewrite HiveGenericUDF with Invoke.
cc @HyukjinKwon @cloud-fan @yaooqinn @dongjoon-hyun @LuciferYang

@DonnyZone

DonnyZone commented Mar 8, 2023

Copy link
Copy Markdown
Contributor

Any progress on this topic? We find Codegen combined with the expression-reuse feature (#29975) can greatly improve performance. Many users write duplicate hiveUDFs in their queries.

@panbingkun panbingkun closed this May 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants