Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FEA][SPARK-39557][SQL] Support ARRAY, STRUCT, MAP types as DEFAULT values #6036

Open
amahussein opened this issue Jul 20, 2022 · 0 comments
Labels
audit_3.4.0 Audit related tasks for 3.4.0 feature request New feature or request P1 Nice to have for release

Comments

@amahussein
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
DDL commands that attempted to use ARRAY, STRUCT, MAP as default returned error messages.
This Spark feature enabled Default values since spark-3.4.0.

Describe the solution you'd like
The new behavior is to expand DEFAULT column values to use those types. Example:

CREATE TABLE t(i BOOLEAN) USING PARQUET;
INSERT INTO t SELECT FALSE;
ALTER TABLE t ADD COLUMN s ARRAY<INT> DEFAULT ARRAY(1, 2);
SELECT * FROM t;
> false, [1, 2]

We need to update spark-rapids com.nvidia.spark.rapids.literals.scala so that the SQL can handle:

  • GenericArrayData,
  • GenericInternalRow,
  • ArrayBasedMapData

Then investigate changes to be done to the Parquet and the Aggregations if any.

Describe alternatives you've considered
If we are not going to support this, then we need to update the compatibility doc.

Additional context
apache/spark@1f2c50ea0b

@amahussein amahussein added feature request New feature or request ? - Needs Triage Need team to review and classify audit_3.4.0 Audit related tasks for 3.4.0 labels Jul 20, 2022
@mattahrens mattahrens added P1 Nice to have for release and removed ? - Needs Triage Need team to review and classify labels Aug 2, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
audit_3.4.0 Audit related tasks for 3.4.0 feature request New feature or request P1 Nice to have for release
Projects
None yet
Development

No branches or pull requests

2 participants