Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[BUG] sort does not match spark for -0.0 and 0.0 #84

Closed
revans2 opened this issue May 29, 2020 · 1 comment · Fixed by #1412
Closed

[BUG] sort does not match spark for -0.0 and 0.0 #84

revans2 opened this issue May 29, 2020 · 1 comment · Fixed by #1412
Labels
bug Something isn't working P2 Not required for release SQL part of the SQL/Dataframe plugin
Milestone
Dec 7 - Dec 18

Comments

@revans2
Copy link
Collaborator

revans2 commented May 29, 2020

Describe the bug
This is a really odd corner case, but spark will place all -0.0 entries as less than 0.0 entries. This should almost never show up in practice, but because our tests hit corner cases we end up seeing it.

cudf does not even expose a good way for us to extract the sign bit from a -0.0 to try and use that as a second order sort column. So this one might be fun to try and fix.

Steps/Code to reproduce bug
Create a dataframe that has both 0.0 and -0.0 values in it. The sort it.

Expected behavior
-0.0 values come before 0.0 values
@revans2 revans2 added bug Something isn't working ? - Needs Triage Need team to review and classify SQL part of the SQL/Dataframe plugin labels May 29, 2020
@revans2 revans2 removed the ? - Needs Triage Need team to review and classify label Jun 2, 2020
@revans2 revans2 mentioned this issue Jun 2, 2020
Closed
@revans2 revans2 mentioned this issue Jun 15, 2020
Merged
@revans2 revans2 mentioned this issue Jun 26, 2020
Open
@revans2 revans2 mentioned this issue Jul 7, 2020
Merged
5 tasks
@sameerz sameerz added the P2 Not required for release label Aug 25, 2020
@revans2 revans2 mentioned this issue Sep 17, 2020
Merged
@revans2 revans2 mentioned this issue Dec 16, 2020
Merged
@revans2 revans2 added this to the Dec 7 - Dec 18 milestone Dec 17, 2020
@revans2
Copy link
Collaborator Author

revans2 commented Dec 17, 2020

Spark actually changed behavior in 3.1.0

tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to 3f175ce (NVIDIA#84)
421c46b 
Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working P2 Not required for release SQL part of the SQL/Dataframe plugin
Projects
None yet
Milestone
Dec 7 - Dec 18
Development

Successfully merging a pull request may close this issue.

Fix a lot of tests marked with xfail for Spark 3.1.0 that no longer fail revans2/spark-rapids
2 participants
@revans2 @sameerz