Fix hash_aggregate_test.py to run with ANSI enabled [databricks] #11219
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes NVIDIA#11018. This commit fixes the hash aggregate tests that fail with ANSI enabled. These tests fail most visibly on Spark 4.0, where ANSI mode is enabled by default. Signed-off-by: MithunR <mithunr@nvidia.com>
Most of the rest are borked because of exercising aggregations like SUM, COUNT, AVG, etc. in ANSI mode. NVIDIA#5114 sees to it that these aggregations fall to CPU.
|
Build |
mythrocks
changed the title
|
Thank you for reviewing, @razajafri. This change has been merged. |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11018.
This commit fixes the tests in
hash_aggregate_test.pyto run correctly when run with ANSI enabled. This is essential for running the tests with Spark 4.0, where ANSI mode is on by default.A vast majority of the tests here happen to exercise aggregations like
SUM,COUNT,AVG, etc. which fall to CPU, on account of #5114. These tests have been marked with@disable_ansi_mode, so that they run to completion correctly. These may be revisited after #5114 has been addressed.In cases where #5114 does not apply, the tests have been modified to run with ANSI on and off.