[SPARK-17699] Support for parsing JSON string columns

[marmbrus]

Spark SQL has great support for reading text files that contain JSON data. However, in many cases the JSON data is just one column amongst others. This is particularly true when reading from sources such as Kafka. This PR adds a new functions from_json that converts a string column into a nested StructType with a user specified schema.

Example usage:

val df = Seq("""{"a": 1}""").toDS()
val schema = new StructType().add("a", IntegerType)

df.select(from_json($"value", schema) as 'json) // => [json: <a: int>]

This PR adds support for java, scala and python. I leveraged our existing JSON parsing support by moving it into catalyst (so that we could define expressions using it). I left SQL out for now, because I'm not sure how users would specify a schema.

[SparkQA]

Test build #66016 has finished for PR 15274 at commit 62f56a7.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hvanhovell]

Should this implement ExpectsInputTypes?

[marmbrus]

Ah, yes, it definitly should. Let me update.

[rxin]

Might want to send a dev list email to solicit feedback on the API?

[marmbrus]

Emailed the list. Seems like a popular feature so far :)

[SparkQA]

Test build #66048 has finished for PR 15274 at commit 983def2.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66052 has finished for PR 15274 at commit 360b97b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

@marmbrus I just wonder if adding to_json make senses (although maybe it should be done in another PR). Just curious. I am just imaging the case to write out dataframes by some data sources not supporting nested structured types.

[yhuai]

LGTM. Merging to master.

[marmbrus]

@HyukjinKwon absolutely. I actually changed the name from json_parser to from_json in anticipation of adding to_json :)

[DanielMe]

@marmbrus: Is there any workaround I can use to achieve a similar effect in 1.6?

[yhuai]

@DanielMe The best options for 1.6 are get_json_object and json_tuple (their docs can be found at https://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.sql.functions$).

[DanielMe]

@yhuai thanks! My impression was that get_json_object does not convert json arrays to ArrayTypes, maybe I misunderstood the way it's supposed to be used though.

[yhuai]

@DanielMe oh, I see. get_json_object will not parse json array. You need to have a UDF to do that for Spark 1.6.

[gatorsmile]

Actually, to specify the schema in SQL language, maybe we can use a JSON string. A little bit odd. So far, nobody is asking for it, I guess. Let us see whether users need it in SQL

[Sazpaimon]

@gatorsmile Alternatively, one can use do what brickhouse's from_json Hive UDF does ( https://gist.github.com/jeromebanks/8855408#file-gistfile1-sql )

(For the record, I actually need this in SQL)

[gatorsmile]

Based on the comment @marmbrus in a JIRA, we prefer to using our DDL format. For example, like what we did for CREATE TABLE, we can specify the schema using a int, b string