Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Spark 3.4 changes to cast broke build #5807

Closed
tgravescs opened this issue Jun 13, 2022 · 3 comments · Fixed by #6922 or #8179
Closed

Spark 3.4 changes to cast broke build #5807

tgravescs opened this issue Jun 13, 2022 · 3 comments · Fixed by #6922 or #8179
Assignees
Labels
audit_3.4.0 Audit related tasks for 3.4.0 build Related to CI / CD or cleanly building P0 Must have for release

Comments

@tgravescs
Copy link
Collaborator

tgravescs commented Jun 13, 2022

Is your feature request related to a problem? Please describe.
Looks like changes in Saprk 3.4 cast broke the plugin 3.4 build:

apache/spark@46175d1
apache/spark@a4f96af

EDIT: Please aware of the followup change

[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:30: object CastBase is not a member of package org.apache.spark.sql.catalyst.expressions
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:37: not found: type CastBase
ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: not found: type CastBase
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: value timeZoneId is not a member of org.apache.spark.sql.catalyst.expressions.Expression
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: value needsTimeZone is not a member of org.apache.spark.sql.catalyst.expressions.Expression
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:59: value withTimeZone is not a member of org.apache.spark.sql.catalyst.expressions.Expression
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:55: value child is not a member of type parameter INPUT
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:56: value dataType is not a member of type parameter INPUT
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:157: value timeZoneId is not a member of type parameter INPUT
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:30: Unused import
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:927: not found: type CastBase
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:929: not found: type CastBase
[ERROR] [Error] /home/tgraves/workspace/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:2102: recursive value x$15 needs type
@tgravescs tgravescs added ? - Needs Triage Need team to review and classify P0 Must have for release labels Jun 13, 2022
@sameerz sameerz added audit_3.4.0 Audit related tasks for 3.4.0 P0 Must have for release and removed ? - Needs Triage Need team to review and classify P0 Must have for release labels Jun 14, 2022
@nartal1 nartal1 mentioned this issue Jun 29, 2022
50 tasks
@amahussein
Copy link
Collaborator

EDIT: Please aware of the followup change [SPARK-40054][SQL] Restore the error handling syntax of try_cast() apache/spark@f8d51b9940

@rwlee
Copy link
Contributor

rwlee commented Nov 1, 2022

Per @amahussein -- this bug now exists in Databricks 3.3

@amahussein
Copy link
Collaborator

Trying to build for DB 11.3 / Spark 3.3.0.
Apparently the castBase was ported to 3.3 on databricks.

[INFO] Compiling 313 Scala sources and 59 Java sources to /home/ubuntu/spark-rapids/sql-plugin/target/spark330db/classes...
[ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:32: object CastBase is not a member of package org.apache.spark.sql.catalyst.expressions
[ERROR] import org.apache.spark.sql.catalyst.expressions.{Cast, CastBase, Expression, NullIntolerant, TimeZoneAwareExpression}
[ERROR]        ^
[ERROR] /home/ubuntu/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:40: not found: type CastBase
[ERROR] final class CastExprMeta[INPUT <: CastBase](

@nartal1 nartal1 reopened this Nov 3, 2022
@nartal1 nartal1 added P1 Nice to have for release and removed P0 Must have for release labels Nov 3, 2022
@NVnavkumar NVnavkumar removed the P1 Nice to have for release label Apr 19, 2023
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
audit_3.4.0 Audit related tasks for 3.4.0 build Related to CI / CD or cleanly building P0 Must have for release
Projects
None yet
7 participants