[BUG] read_json does not output null list when the input is a list with different data type than the specified schema

[ttnghia]

When the input JSON has lists with mixed types, read_json does not output nulls correctly for the output lists column. For example:

scala> val df = Seq("""{"a": [{"b": "1", "c": "2"}]}""", """{"a": [123]}""").toDF
df: org.apache.spark.sql.DataFrame = [value: string]

scala> df.show(false)
+-----------------------------+
|value                        |
+-----------------------------+
|{"a": [{"b": "1", "c": "2"}]}|
|{"a": [123]}                 |
+-----------------------------+

scala> df.repartition(1).selectExpr("from_json(value, 'struct<a: array<struct<b: string, c: string>>>')").show()
+----------------+                                                              
|from_json(value)|
+----------------+
|      {[{1, 2}]}|
|        {[null]}|             <========== wrong, should be {null}
+----------------+

In the example above, the input of the second list is [123], while the desired schema is list<struct<string, string>>. As such, the output here should be a null list, instead of a list of null struct like above.

[ttnghia]

Another example:
GPU:

scala> val df = Seq("""{"a": [{"b": "1", "c": "2"}]}""", """{"a": [123, {"b": "1"}]}""").toDF
df: org.apache.spark.sql.DataFrame = [value: string]

scala> df.show(false)
+-----------------------------+
|value                        |
+-----------------------------+
|{"a": [{"b": "1", "c": "2"}]}|
|{"a": [123, {"b": "1"}]}     |
+-----------------------------+

scala> df.repartition(1).selectExpr("from_json(value, 'struct<a: array<struct<b: string, c: string>>>')").show()
+-------------------+
|   from_json(value)|
+-------------------+
|         {[{1, 2}]}|
|{[null, {1, null}]}|              <================ wrong
+-------------------+

The correct output from Spark CPU is also a null list:

+----------------+
|from_json(value)|
+----------------+
|      {[{1, 2}]}|
|          {null}|
+----------------+

[karthikeyann]

Input: for schemastruct<c2: array<struct<b: string, c: string>>>

      {"c2": []}
      {"c2": [{}]}
      {"c2": [[]]}
      {"c2": [[], {}]}
      {"c2": [[123], {"b": "1"}]}
      {"c2": [{"x": "y"}, {"b": "1"}]}
      {}

Spark Output:

[]
[{null, null}]
null
null
null
[{null, null}, {1, null}] 
null