Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[FEA] Support Parquet Read of Decimal FIXED_LENGTH_BYTE_ARRAY #1324

Closed
revans2 opened this issue Dec 8, 2020 · 4 comments
Closed

[FEA] Support Parquet Read of Decimal FIXED_LENGTH_BYTE_ARRAY #1324

revans2 opened this issue Dec 8, 2020 · 4 comments
Assignees
@sperlingxx
Labels
feature request New feature or request P0 Must have for release

Comments

@revans2
Copy link
Collaborator

revans2 commented Dec 8, 2020

Is your feature request related to a problem? Please describe.

rapidsai/cudf#6909 was filed to support reading FIXED_LENGTH_BYTE_ARRAY decimal values. Older versions of Spark store decimal this way so we do want to support this before we can merge decimal reading support.

#1294 is a PR that covers most of reading decimal, but not everything.
@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Dec 8, 2020
@sameerz sameerz added P0 Must have for release and removed ? - Needs Triage Need team to review and classify labels Dec 8, 2020
@revans2 revans2 mentioned this issue Dec 8, 2020
Closed
27 tasks
@sperlingxx
Copy link
Collaborator

sperlingxx commented Dec 9, 2020 •
edited
Loading

I am willing to take the cuDF side task to support reading FIXED_LENGTH_BYTE_ARRAY decimal values if necessary.

@sperlingxx
Copy link
Collaborator

sperlingxx commented Dec 16, 2020 •
edited
Loading

@sameerz I think we can close this issue because corresponding cuDF side task has been merged. And I have manually verified this feature with spark-rapids.

@sameerz
Copy link
Collaborator

sameerz commented Dec 18, 2020

@sperlingxx do we have automated integration test cases to read FIXED_LENGTH_BYTE_ARRAY decimal values?

@revans2
Copy link
Collaborator Author

revans2 commented Jan 5, 2021

This appears to be done

@revans2 revans2 closed this as completed Jan 5, 2021
tgravescs pushed a commit to tgravescs/spark-rapids that referenced this issue Nov 30, 2023
@nvauto
Update submodule cudf to edb25a84aaafa0d65d36ebb94113393d4b6474fb (NV…
55d4d4e 
…IDIA#1324)

Signed-off-by: spark-rapids automation <70000568+nvauto@users.noreply.github.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@sperlingxx sperlingxx

Labels
feature request New feature or request P0 Must have for release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@revans2 @sperlingxx @sameerz