-
Notifications
You must be signed in to change notification settings - Fork 245
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Throw SparkDateTimeException for InvalidInput while casting in ANSI mode [databricks] #5731
Conversation
…SI mode Signed-off-by: Niranjan Artal <nartal@nvidia.com>
build |
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
build |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what does the full error message look like now from us?
sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/330+/scala/org/apache/spark/ShimTrampolineUtil.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/330+/scala/org/apache/spark/ShimTrampolineUtil.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
With the latest commit, the error message looks like this: Input: Nan Spark's error message: Input: +Infinity/ -Infinity. |
sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/330+/scala/com/nvidia/spark/rapids/shims/AnsiUtil.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Niranjan Artal <nartal@nvidia.com>
build |
s"NaN, Infinity or out-of-range values. To return NULL instead, use 'try_cast'. " + | ||
s"If necessary set ${SQLConf.ANSI_ENABLED.key} to false to bypass this error." | ||
val msg = s"The column contains out-of-range values. To return NULL instead, use " + | ||
s"'try_cast'. If necessary set ${SQLConf.ANSI_ENABLED.key} to false to bypass this error." |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: in the comment it looks like the original Spark message uses quotes around the config key #5731 (comment)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Created follow on issue : #5750 for changing all the error messages based on above format.
Build failure is due to #5744 |
build |
This fixes #5719 .
Spark updated the error message to throw SparkDateTimeException in this PR: apache/spark#36591 .
Since SparkDateTimeException is a private class in org.apache.spark, I have included TrampolineUtil to call that Exception.
Current integration test passes now. No new test required.