Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Fix CastBase issues not related to PromotePrecision and CheckOverflow [databricks] #6922

Merged
merged 10 commits into from
Nov 3, 2022
Merged

Fix CastBase issues not related to PromotePrecision and CheckOverflow [databricks] #6922

merged 10 commits into from
Nov 3, 2022

Conversation

rwlee
Copy link
Contributor

@rwlee rwlee commented Oct 26, 2022

Fixes build errors in Spark 3.4.0, but does not necessarily match the functionality in 3.4.0. As a result it is related to #5807 but does not resolve it.

Current error set w/o changes:

[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:32: object CastBase is not a member of package org.apache.spark.sql.cat
alyst.expressions
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:40: not found: type CastBase
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/320+/scala/org/apache/spark/sql/execution/datasources/rapids/DataSourceStrategyUtils.scala:26: value translateRunti
meFilter is not a member of object org.apache.spark.sql.execution.datasources.DataSourceStrategy
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/330+/scala/org/apache/spark/rapids/ShimTrampolineUtil.scala:23: not enough arguments for constructor SparkDateTimeE
xception: (errorClass: String, messageParameters: Map[String,String], context: Array[org.apache.spark.QueryContext], summary: String)org.apache.spark.SparkDateTimeException.
Unspecified value parameters context, summary.
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: not found: type CastBase
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: value timeZoneId is not a member of org.apache.spark.sql.ca
talyst.expressions.Expression
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:58: value needsTimeZone is not a member of org.apache.spark.sql
.catalyst.expressions.Expression
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCanonicalize.scala:59: value withTimeZone is not a member of org.apache.spark.sql.
catalyst.expressions.Expression
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:58: value child is not a member of type parameter INPUT
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:59: value dataType is not a member of type parameter INPUT
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:158: value timeZoneId is not a member of type parameter INPUT
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCast.scala:32: Unused import
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:753: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:999: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:1842: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:1956: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:899: not found: type PromotePrecision
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:903: not found: type PromotePrecision
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:934: not found: type PromotePrecision
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:934: value child is not a member of Any
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:934: not found: type CastBase
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:935: value child is not a member of Any
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:936: value child is not a member of Any
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:936: not found: type CastBase
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:2087: recursive value x$15 needs type
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:2596: value failOnError is not a member of org.apache.spark.sql.catalyst.expressions.GetMapValue
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala:1084: value checkForNumericExpr is not a member of object org.apache.spark.sql.catalyst.util.TypeUtils
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala:1575: value checkForNumericExpr is not a member of object org.apache.spark.sql.catalyst.util.TypeUtils
[ERROR] 28 errors found

28 errors with 5 direct CastBase errors and several related downstream errors.

Current error set w/ changes:

[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/320+/scala/org/apache/spark/sql/execution/datasources/rapids/DataSourceStrategyUtils.scala:26: value translateRunti
meFilter is not a member of object org.apache.spark.sql.execution.datasources.DataSourceStrategy
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/330+/scala/org/apache/spark/rapids/ShimTrampolineUtil.scala:23: not enough arguments for constructor SparkDateTimeE
xception: (errorClass: String, messageParameters: Map[String,String], context: Array[org.apache.spark.QueryContext], summary: String)org.apache.spark.SparkDateTimeException.
Unspecified value parameters context, summary.
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:753: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:999: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:1842: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOrcScan.scala:1956: overloaded method value writeTo with alternatives:
  (x$1: org.apache.orc.protobuf.CodedOutputStream)Unit <and>
  (x$1: java.io.OutputStream)Unit
 cannot be applied to (com.google.protobuf.CodedOutputStream)
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:899: not found: type PromotePrecision
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:903: not found: type PromotePrecision
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:934: not found: type PromotePrecision
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:934: value child is not a member of Any
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:934: not found: type CastBase
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:935: value child is not a member of Any
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:936: value child is not a member of Any
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:936: not found: type CastBase
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:2087: recursive value x$15 needs type
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala:2596: value failOnError is not a member of org.apache.spark.sql.catalyst.expressions.GetMapValue
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala:1084: value checkForNumericExpr is not a member of object org.apache.spark.sql.catalyst.util.TypeUtils
[ERROR] [Error] /home/ryanlee/spark-rapids/sql-plugin/src/main/scala/org/apache/spark/sql/rapids/AggregateFunctions.scala:1575: value checkForNumericExpr is not a member of object org.apache.spark.sql.catalyst.util.TypeUtils
[ERROR] 18 errors found

18 errors with 2 CastBase related errors found in PromotePrecision and CheckOverflow -- @razajafri is working on those changes as part of #5827 which should eliminate the CastBase references.

These changes should resolve 10 build errors related to 3.4.0

@rwlee rwlee added build Related to CI / CD or cleanly building audit_3.4.0 Audit related tasks for 3.4.0 improve labels Oct 26, 2022
@rwlee
Copy link
Contributor Author

rwlee commented Oct 26, 2022

build

@rwlee
Signoff commit
b43fbae 
Signed-off-by: Ryan Lee <ryanlee@nvidia.com>
@nartal1 nartal1 changed the title Fix CastBase issues not related to PromotePrecision and CheckOverflow Fix CastBase issues not related to PromotePrecision and CheckOverflow [databricks] Oct 27, 2022
@nartal1
Copy link
Collaborator

nartal1 commented Oct 27, 2022

Running CI for databricks since the db shim is modified.

@nartal1
Copy link
Collaborator

nartal1 commented Oct 27, 2022

build

@razajafri razajafri mentioned this pull request Oct 28, 2022
Merged
@rwlee
Copy link
Contributor Author

rwlee commented Nov 1, 2022

build

@rwlee
Copy link
Contributor Author

rwlee commented Nov 1, 2022

build

1 similar comment
@rwlee
Copy link
Contributor Author

rwlee commented Nov 1, 2022

build

@sameerz sameerz linked an issue Nov 1, 2022 that may be closed by this pull request
Spark 3.4 changes to cast broke build #5807
Closed
@rwlee rwlee requested a review from revans2 November 1, 2022 21:51
@nartal1 nartal1 removed a link to an issue Nov 1, 2022
Spark 3.4 changes to cast broke build #5807
Closed
@sameerz sameerz linked an issue Nov 2, 2022 that may be closed by this pull request
Spark 3.4 changes to cast broke build #5807
Closed
@nartal1 nartal1 mentioned this pull request Nov 2, 2022
Closed
29 tasks
@rwlee
Copy link
Contributor Author

rwlee commented Nov 2, 2022

build

@rwlee rwlee requested a review from gerashegalov November 3, 2022 00:02
gerashegalov
gerashegalov approved these changes Nov 3, 2022
View reviewed changes
Copy link
Collaborator

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@revans2 revans2 merged commit 4cde843 into NVIDIA:branch-22.12 Nov 3, 2022
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@nartal1 nartal1 nartal1 left review comments

@gerashegalov gerashegalov gerashegalov approved these changes

@revans2 revans2 revans2 approved these changes

Assignees

@rwlee rwlee

Labels
audit_3.4.0 Audit related tasks for 3.4.0 build Related to CI / CD or cleanly building improve
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Spark 3.4 changes to cast broke build
4 participants
@rwlee @nartal1 @gerashegalov @revans2