[SPARK-48922][SQL] Optimize nested data type insertion performance

[wForget]

What changes were proposed in this pull request?

To improve insertion performance, we do not need to add transform expressions when there is no conversion for complex types.

Why are the changes needed?

During the upgrade from Spark 3.1.1 to 3.5.0, I found a performance regression in map type inserts.

There are some extra conversion expressions in project before insert, which doesn't seem to be always necessary.

map_from_arrays(transform(map_keys(map#516), lambdafunction(lambda key#652, lambda key#652, false)), transform(map_values(map#516), lambdafunction(lambda value#654, lambda value#654, false))) AS map#656 

Does this PR introduce any user-facing change?

No

How was this patch tested?

Added benchmark case.

before this pr:

Java HotSpot(TM) 64-Bit Server VM 17.0.10+11-LTS-240 on Linux 3.10.0-957.el7.x86_64
Intel Core Processor (Skylake, IBRS)
Insert map type table benchmark, totalRows = 10240:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
----------------------------------------------------------------------------------------------------------------------------------
insert map type table with conversion                        1054           1092          54          0.0      102906.7       1.0X
insert map type table without conversion                      540            582          47          0.0       52770.3       2.0X

Java HotSpot(TM) 64-Bit Server VM 17.0.10+11-LTS-240 on Linux 3.10.0-957.el7.x86_64
Intel Core Processor (Skylake, IBRS)
Insert array type table benchmark, totalRows = 10240:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------
insert array type table with conversion                         372            399          20          0.0       36337.9       1.0X
insert array type table without conversion                      233            255          19          0.0       22775.5       1.6X

after this pr:

Java HotSpot(TM) 64-Bit Server VM 17.0.10+11-LTS-240 on Linux 3.10.0-957.el7.x86_64
Intel Core Processor (Skylake, IBRS)
Insert map type table benchmark, totalRows = 10240:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
----------------------------------------------------------------------------------------------------------------------------------
insert map type table with conversion                        1222           1226           6          0.0      119360.0       1.0X
insert map type table without conversion                      417            429          12          0.0       40714.5       2.9X

Java HotSpot(TM) 64-Bit Server VM 17.0.10+11-LTS-240 on Linux 3.10.0-957.el7.x86_64
Intel Core Processor (Skylake, IBRS)
Insert array type table benchmark, totalRows = 10240:  Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
------------------------------------------------------------------------------------------------------------------------------------
insert array type table with conversion                         389            433          31          0.0       37967.2       1.0X
insert array type table without conversion                      207            241          23          0.0       20246.6       1.9X

Was this patch authored or co-authored using generative AI tooling?

No

[wForget]

The previous behavior was introduced in #33728, @cloud-fan could you please take a look?

[pan3793]

Is the fix proposed here partially fixed by SPARK-49352? also cc @viirya

[wForget]

Is the fix proposed here partially fixed by SPARK-49352? also cc @viirya

SPARK-49352 does not seem to cover MapType

[github-actions]

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

[kazuyukitanimura]

Hi @wForget
This PR looks important, do you plan to reopen this and rebase on top of SPARK-49352 ?
#47843

[wForget]

Hi @wForget This PR looks important, do you plan to reopen this and rebase on top of SPARK-49352 ? #47843

Sure, I will rebase on master and submit a new pr.

[kazuyukitanimura]

Hi @wForget
Just checking if you had a chance

[wForget]

Hi @wForget Just checking if you had a chance

I reopened in #50245