[SPARK-48545][SQL] Create to_avro and from_avro SQL functions to match DataFrame equivalents

[dtenedor]

What changes were proposed in this pull request?

This PR creates two new SQL functions "to_avro" and "from_avro" to match existing DataFrame equivalents.

For example:

sql(
  """
    |create table t as
    |  select named_struct('u', named_struct('member0', member0, 'member1', member1)) as s
    |  from values (1, null), (null,  'a') tab(member0, member1)
    |""".stripMargin)

val jsonFormatSchema =
  """
    |{
    |  "type": "record",
    |  "name": "struct",
    |  "fields": [{
    |    "name": "u",
    |    "type": ["int","string"]
    |  }]
    |}
    |""".stripMargin

spark.sql(
  s"""
    |select from_avro(result, '$jsonFormatSchema', map()).u from (
    |  select to_avro(s, '$jsonFormatSchema') as result from t
    |)")
  .collect()

> {1, NULL}
  {NULL, "a"}

Why are the changes needed?

This brings parity between SQL and DataFrame APIs in Apache Spark.

Does this PR introduce any user-facing change?

Yes, see above.

How was this patch tested?

This PR adds extra unit tests, and I also checked that the functions work with spark-shell.

Was this patch authored or co-authored using generative AI tooling?

No GitHub copilot usage this time

[dtenedor]

Thanks @allisonwang-db for your review, followed through on your comments.

[allisonwang-db]

Looks good! cc @cloud-fan

[cloud-fan]

LGTM if CI passes

[dtenedor]

cc @cloud-fan the CI is passing now :)

[gengliangwang]

Thanks, merging to master