Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[FEA] Add ANSI support #5120

Open
31 of 50 tasks
nartal1 opened this issue Mar 31, 2022 · 1 comment
Open
31 of 50 tasks

[FEA] Add ANSI support #5120

nartal1 opened this issue Mar 31, 2022 · 1 comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 epic Issue that encompasses a significant feature or body of work feature request New feature or request

Comments

@nartal1
Copy link
Collaborator

nartal1 commented Mar 31, 2022 •
edited by sameerz
Loading

Is your feature request related to a problem? Please describe.
We have ANSI support for some of the SQL functions/operators and casting but it is not completely in sync with everything that Spark supports. This epic is to list out the gaps and track the remaining issues.
Auditing ANSI issues based on Spark JIRA - https://issues.apache.org/jira/browse/SPARK-30374

P0 issues

P1 issues

Completed issues
@nartal1 nartal1 added feature request New feature or request ? - Needs Triage Need team to review and classify epic Issue that encompasses a significant feature or body of work labels Mar 31, 2022
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Apr 5, 2022
@amahussein amahussein mentioned this issue Aug 15, 2022
Open
@mattahrens mattahrens added the audit_4.0.0 Audit related tasks for 4.0.0 label Apr 16, 2024
@mattahrens
Copy link
Collaborator

Note: integrate test framework currently explicitly sets ANSI mode to off for each test, so we need to change this to test with ANSI mode on.

mythrocks added a commit to mythrocks/spark-rapids that referenced this issue Jun 17, 2024
@mythrocks
Disable ANSI mode for window function tests.
3db464b 
Fixes NVIDIA#11019.

Window function tests fail on Spark 4.0 because of NVIDIA#5114 (and NVIDIA#5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>
@mythrocks mythrocks mentioned this issue Jun 17, 2024
Merged
razajafri pushed a commit that referenced this issue Jun 26, 2024
@mythrocks
Disable ANSI mode for window function tests [databricks] (#11073)
34e6bc8 
* Disable ANSI mode for window function tests.

Fixes #11019.

Window function tests fail on Spark 4.0 because of #5114 (and #5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Switch from @ansi_mode_disabled to @disable_ansi_mode.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>
wjxiz1992 added a commit to nvliyuan/yuali-spark-rapids that referenced this issue Jun 26, 2024
@wjxiz1992 @binmahone
@razajafri @jihoonson @gerashegalov @mythrocks
optimzing Expand+Aggregate in sqlw with many count distinct (#22)
4415f61 
* optimzing Expand+Aggregate in sqlw with many count distinct

Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>

* Add GpuBucketingUtils shim to Spark 4.0.0 (NVIDIA#11092)

* Add GpuBucketingUtils shim to Spark 4.0.0

* Signing off

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

---------

Signed-off-by: Raza Jafri <rjafri@nvidia.com>

* Improve the diagnostics for 'conv' fallback explain (NVIDIA#11076)

* Improve the diagnostics for 'conv' fallback explain

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

* don't use nil

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

* the bases should not be an empty string in the error message when the user input is not

Signed-off-by: Jihoon Son <ghoonson@gmail.com>

* more user-friendly message

* Update sql-plugin/src/main/scala/org/apache/spark/sql/rapids/stringFunctions.scala

Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

---------

Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>

* Disable ANSI mode for window function tests [databricks] (NVIDIA#11073)

* Disable ANSI mode for window function tests.

Fixes NVIDIA#11019.

Window function tests fail on Spark 4.0 because of NVIDIA#5114 (and NVIDIA#5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Switch from @ansi_mode_disabled to @disable_ansi_mode.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>

---------

Signed-off-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Signed-off-by: Raza Jafri <rjafri@nvidia.com>
Signed-off-by: Jihoon Son <ghoonson@gmail.com>
Signed-off-by: MithunR <mithunr@nvidia.com>
Co-authored-by: Hongbin Ma (Mahone) <mahongbin@apache.org>
Co-authored-by: Raza Jafri <razajafri@users.noreply.github.com>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
Co-authored-by: Gera Shegalov <gshegalov@nvidia.com>
Co-authored-by: MithunR <mithunr@nvidia.com>
SurajAralihalli pushed a commit to SurajAralihalli/spark-rapids that referenced this issue Jul 12, 2024
@mythrocks @SurajAralihalli
Disable ANSI mode for window function tests [databricks] (NVIDIA#11073)
11348ef 
* Disable ANSI mode for window function tests.

Fixes NVIDIA#11019.

Window function tests fail on Spark 4.0 because of NVIDIA#5114 (and NVIDIA#5120 broadly),
because spark-rapids does not support SUM, COUNT, and certain other aggregations
in ANSI mode.

This commit disables ANSI mode tests for the failing window function tests. These may be
revisited, once error/overflow checking is available for ANSI mode in spark-rapids.

Signed-off-by: MithunR <mithunr@nvidia.com>

* Switch from @ansi_mode_disabled to @disable_ansi_mode.

---------

Signed-off-by: MithunR <mithunr@nvidia.com>
@razajafri razajafri mentioned this issue Jul 26, 2024
Merged
@nartal1 nartal1 mentioned this issue Feb 11, 2025
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
audit_4.0.0 Audit related tasks for 4.0.0 epic Issue that encompasses a significant feature or body of work feature request New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mattahrens @nartal1