You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It would be nice if we support 'yyyyMMdd' format for from_unixtime function. I know we could avoid the fallback by setting spark.rapids.sql.incompatibleDateFormats.enabled=true, but it would simplify the configs for customers if we support more LEGACY format.
For example:
create a hive table and insert some rows:
CREATE TABLE `xxx_timestamp`(
`id` int,
`name` string,
`created_at` timestamp)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
run below SQL:
spark.sql("select from_unixtime(unix_timestamp(created_at, 'yyyy-MM-dd HH:mm:ss'), 'yyyyMMdd') from xxx_timestamp ").show
fallback info:
!Expression <FromUnixTime> from_unixtime(unix_timestamp(created_at#27, yyyy-MM-dd HH:mm:ss, Some(UTC), false), yyyyMMdd, Some(UTC)) cannot run on GPU because CORRECTED format 'yyyyMMdd' on the GPU is not guaranteed to produce the same results as Spark on CPU. Set spark.rapids.sql.incompatibleDateFormats.enabled=true to force onto GPU.
The text was updated successfully, but these errors were encountered:
A similar fallback: !Expression <FromUnixTime> from_unixtime(report_create_time#174L, yyyyMMdd, Some(Etc/UTC)) cannot run on GPU because LEGACY format 'yyyyMMdd' is not supported on the GPU.
nvliyuan
changed the title
[FEA] support more LEGACY format in from_unixtime function
[FEA] support ‘yyyyMMdd’ format in from_unixtime function
Dec 12, 2023
It would be nice if we support 'yyyyMMdd' format for from_unixtime function. I know we could avoid the fallback by setting
spark.rapids.sql.incompatibleDateFormats.enabled=true
, but it would simplify the configs for customers if we support more LEGACY format.For example:
fallback info:
The text was updated successfully, but these errors were encountered: