Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

PCBS and Parquet support ANSI year month interval type [databricks] #4993

Merged
merged 3 commits into from
Mar 25, 2022
Merged

PCBS and Parquet support ANSI year month interval type [databricks] #4993

merged 3 commits into from
Mar 25, 2022

Conversation

res-life
Copy link
Collaborator

Closes #2814

  1. Parquet support ANSI year month interval type
  2. PCBS support ANSI year month interval type

Signed-off-by: Chong Gao res_life@163.com

Chong Gao added 2 commits March 21, 2022 11:43
@res-life
Copy link
Collaborator Author

build

@res-life
Copy link
Collaborator Author

build

@res-life res-life changed the title PCBS and Parquet support ANSI year month interval type PCBS and Parquet support ANSI year month interval type [databricks] Mar 21, 2022
@res-life
Copy link
Collaborator Author

build

@res-life res-life marked this pull request as draft March 21, 2022 05:46
@res-life
Copy link
Collaborator Author

Should re-target branch-22.06, convert it to draft.

@res-life res-life changed the base branch from branch-22.04 to branch-22.06 March 22, 2022 08:42
@res-life res-life marked this pull request as ready for review March 22, 2022 08:43
@res-life
Copy link
Collaborator Author

build

@res-life res-life added the audit_3.3.0 Audit related tasks for 3.3.0 label Mar 22, 2022
@res-life res-life requested a review from razajafri March 23, 2022 06:30
@res-life
Copy link
Collaborator Author

@razajafri Help review, thanks.

@firestarman
Copy link
Collaborator

LGTM

@razajafri
Copy link
Collaborator

@res-life why are we choosing to write scalatest when as a convention we write python tests? It makes sense to do scala tests for situations where writing a python test isn't possible e.g. writing a unit test for a private method or may be you have to inject variables into a class before testing. We also use python tests in environments where it's not possible for us to run scala tests.

@res-life
Copy link
Collaborator Author

@razajafri
Currently Pyspark does not have year month type, please see
https://github.com/apache/spark/blob/branch-3.3/python/pyspark/sql/types.py
Day time type is supported in
SPARK-37279 Support DayTimeIntervalType in createDataFrame, collect and Python UDF
Is waiting the Spark to support year month type.

@res-life res-life merged commit c3731e7 into NVIDIA:branch-22.06 Mar 25, 2022
@res-life res-life deleted the cache-support branch March 25, 2022 07:57
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@firestarman firestarman firestarman left review comments

@razajafri razajafri razajafri approved these changes

Assignees
No one assigned
Labels
audit_3.2.0 audit_3.3.0 Audit related tasks for 3.3.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SPARK-35243 has introduced support for ANSI interval types
4 participants
@res-life @firestarman @razajafri @sameerz