Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

[FEA] support ‘yyyyMMdd’ format in from_unixtime function #8807

Closed
nvliyuan opened this issue Jul 26, 2023 · 3 comments · Fixed by #10023
Closed

[FEA] support ‘yyyyMMdd’ format in from_unixtime function #8807

nvliyuan opened this issue Jul 26, 2023 · 3 comments · Fixed by #10023
Labels
feature request New feature or request

Comments

@nvliyuan
Copy link
Collaborator

It would be nice if we support 'yyyyMMdd' format for from_unixtime function. I know we could avoid the fallback by setting spark.rapids.sql.incompatibleDateFormats.enabled=true, but it would simplify the configs for customers if we support more LEGACY format.
For example:

create a hive table and insert some rows:
CREATE TABLE `xxx_timestamp`(
  `id` int,
  `name` string,
  `created_at` timestamp)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'

run below SQL:
spark.sql("select from_unixtime(unix_timestamp(created_at, 'yyyy-MM-dd HH:mm:ss'), 'yyyyMMdd') from xxx_timestamp ").show

fallback info:

!Expression <FromUnixTime> from_unixtime(unix_timestamp(created_at#27, yyyy-MM-dd HH:mm:ss, Some(UTC), false), yyyyMMdd, Some(UTC)) cannot run on GPU because CORRECTED format 'yyyyMMdd' on the GPU is not guaranteed to produce the same results as Spark on CPU. Set spark.rapids.sql.incompatibleDateFormats.enabled=true to force onto GPU.
@nvliyuan nvliyuan added feature request New feature or request ? - Needs Triage Need team to review and classify labels Jul 26, 2023
@nvliyuan nvliyuan changed the title [FEA] [FEA] support more LEGACY format in from_unixtime function Jul 26, 2023
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Aug 8, 2023
@nvliyuan
Copy link
Collaborator Author

nvliyuan commented Nov 2, 2023

Low priority for customers.

@nvliyuan
Copy link
Collaborator Author

A similar fallback:
!Expression <FromUnixTime> from_unixtime(report_create_time#174L, yyyyMMdd, Some(Etc/UTC)) cannot run on GPU because LEGACY format 'yyyyMMdd' is not supported on the GPU.

@nvliyuan
Copy link
Collaborator Author

nvliyuan commented Dec 12, 2023

CC @firestarman @winningsix

@nvliyuan nvliyuan changed the title [FEA] support more LEGACY format in from_unixtime function [FEA] support ‘yyyyMMdd’ format in from_unixtime function Dec 12, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
feature request New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants