Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Do not replace TableCacheQueryStageExec #10610

Merged
merged 3 commits into from
Mar 19, 2024
Merged

Do not replace TableCacheQueryStageExec #10610

merged 3 commits into from
Mar 19, 2024

Conversation

andygrove
Copy link
Contributor

@andygrove andygrove commented Mar 18, 2024 •
edited
Loading

Spark 3.5.0 adds a new TableCacheQueryStageExec. We should never try and replace a query stage exec, so this PR adds a neverReplace rule for this new exec.

I manually tested with the example in https://issues.apache.org/jira/browse/SPARK-45592 and it stops the following warning from being displayed:

! <TableCacheQueryStageExec> cannot run on GPU because GPU does not currently support the operator class org.apache.spark.sql.execution.adaptive.TableCacheQueryStageExec

@andygrove
Do not replace TableCacheQueryStageExec
edae2e2
@andygrove
signoff
601c7b1 
Signed-off-by: Andy Grove <andygrove@nvidia.com>
@andygrove andygrove self-assigned this Mar 18, 2024
gerashegalov
gerashegalov previously approved these changes Mar 18, 2024
View reviewed changes
Copy link
Collaborator

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andygrove andygrove marked this pull request as ready for review March 18, 2024 19:08
razajafri
razajafri previously approved these changes Mar 18, 2024
View reviewed changes
Copy link
Collaborator

@razajafri razajafri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@razajafri razajafri dismissed stale reviews from gerashegalov and themself via f1e3e94 March 18, 2024 19:20
@razajafri
Copy link
Collaborator

build

gerashegalov
gerashegalov approved these changes Mar 18, 2024
View reviewed changes
Copy link
Collaborator

@gerashegalov gerashegalov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@andygrove andygrove merged commit 65ec25d into NVIDIA:branch-24.04 Mar 19, 2024
42 of 43 checks passed
@andygrove andygrove deleted the never-replace-TableCacheQueryStageExec branch March 19, 2024 13:27
@sameerz sameerz added the Spark 3.5+ Spark 3.5+ issues label Mar 30, 2024
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@gerashegalov gerashegalov gerashegalov approved these changes

@razajafri razajafri razajafri left review comments

Assignees

@andygrove andygrove

Labels
Spark 3.5+ Spark 3.5+ issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@andygrove @razajafri @gerashegalov @sameerz