[FEA][SPARK-39557][SQL] Support ARRAY, STRUCT, MAP types as DEFAULT values #6036
Labels
audit_3.4.0
Audit related tasks for 3.4.0
feature request
New feature or request
P1
Nice to have for release
Is your feature request related to a problem? Please describe.
DDL commands that attempted to use
ARRAY
,STRUCT
,MAP
as default returned error messages.This Spark feature enabled Default values since
spark-3.4.0
.Describe the solution you'd like
The new behavior is to expand DEFAULT column values to use those types. Example:
We need to update spark-rapids
com.nvidia.spark.rapids.literals.scala
so that the SQL can handle:GenericArrayData
,GenericInternalRow
,ArrayBasedMapData
Then investigate changes to be done to the Parquet and the Aggregations if any.
Describe alternatives you've considered
If we are not going to support this, then we need to update the compatibility doc.
Additional context
apache/spark@1f2c50ea0b
The text was updated successfully, but these errors were encountered: