Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

feat: expose arrow type <-> iceberg type #637

Merged
merged 2 commits into from
Sep 20, 2024
Merged

feat: expose arrow type <-> iceberg type #637

merged 2 commits into from
Sep 20, 2024

Conversation

xxchan
Copy link
Contributor

@xxchan xxchan commented Sep 19, 2024

Previously we only exposed the schema conversion.

@xxchan
feat: expose arrow type <-> iceberg type
e1663bc 
Previously we only exposed the schema conversion.

Signed-off-by: xxchan <xxchan22f@gmail.com>
Xuanwo
Xuanwo reviewed Sep 20, 2024
View reviewed changes
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly LGTM. Would you like to add some unit tests to cover the new APIs?

Xuanwo

This comment was marked as duplicate.

# to view

@xxchan
Copy link
Contributor Author

xxchan commented Sep 20, 2024

I think the tests are already covered by schema conversion tests, as type conversion is just a sub-feature of schema conversion.

@Xuanwo
Copy link
Member

Xuanwo commented Sep 20, 2024

I think the tests are already covered by schema conversion tests, as type conversion is just a sub-feature of schema conversion.

Hi, I understand that the logic might be covered by other parts, but we are adding new public APIs. How about adding unit tests directly for these new APIs?

@xxchan
Copy link
Contributor Author

xxchan commented Sep 20, 2024 via email

@Xuanwo
Copy link
Member

Xuanwo commented Sep 20, 2024

Do you think it would be enough to add one or two simple api call examples

I believe that's enough.

@xxchan
add tests
b2e97cd 
Signed-off-by: xxchan <xxchan22f@gmail.com>
@xxchan xxchan requested a review from Xuanwo September 20, 2024 06:17
@xxchan xxchan mentioned this pull request Sep 20, 2024
Closed
Xuanwo
Xuanwo approved these changes Sep 20, 2024
View reviewed changes
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, really great!

@Xuanwo Xuanwo merged commit 12e12e2 into apache:main Sep 20, 2024
16 checks passed
@xxchan xxchan deleted the xxchan/helpful-salmon branch September 20, 2024 15:30
c-thiel added a commit to hansetag/iceberg-rust that referenced this pull request Oct 2, 2024
@c-thiel
Squashed commit of the following:
12d766f 
commit cda4a0c
Author: congyi wang <58715567+wcy-fdu@users.noreply.github.com>
Date:   Mon Sep 30 11:07:36 2024 +0800

    chore: fix typo in FileIO Schemes  (apache#653)

    * fix typo

    * fix typo

commit af9609d
Author: Scott Donnelly <scott@donnel.ly>
Date:   Mon Sep 30 04:06:14 2024 +0100

    fix: page index evaluator min/max args inverted (apache#648)

    * fix: page index evaluator min/max args inverted

    * style: fix clippy lint in test

commit a6a3fd7
Author: Alon Agmon <54080741+a-agmon@users.noreply.github.com>
Date:   Sat Sep 28 10:10:08 2024 +0300

    test (datafusion): add test for table provider creation (apache#651)

    * add test for table provider creation

    * fix formatting

    * fixing yet another formatting issue

    * testing schema using data fusion

    ---------

    Co-authored-by: Alon Agmon <alon.agmon@appsflyer.com>

commit 87483b4
Author: Alon Agmon <54080741+a-agmon@users.noreply.github.com>
Date:   Fri Sep 27 04:40:08 2024 +0300

    making table provider pub (apache#650)

    Co-authored-by: Alon Agmon <alon.agmon@appsflyer.com>

commit 984c91e
Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com>
Date:   Thu Sep 26 17:56:02 2024 +0800

    avoid to create memory schema operator every time (apache#635)

    Co-authored-by: ZENOTME <st810918843@gmail.com>

commit 4171275
Author: Matheus Alcantara <mths.dev@pm.me>
Date:   Wed Sep 25 08:28:42 2024 -0300

    scan: change ErrorKind when table dont have spanshots (apache#608)

commit ab51355
Author: xxchan <xxchan22f@gmail.com>
Date:   Tue Sep 24 21:25:45 2024 +0800

    fix: compile error due to merge stale PR (apache#646)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 420b4e2
Author: Scott Donnelly <scott@donnel.ly>
Date:   Tue Sep 24 08:20:23 2024 +0100

    Table Scan: Add Row Selection Filtering (apache#565)

    * feat(scan): add row selection capability via PageIndexEvaluator

    * test(row-selection): add first few row selection tests

    * feat(scan): add more tests, fix bug where min/max args swapped

    * fix: ad test and fix for logic bug in PageIndexEvaluator in-clause handler

    * feat: changes suggested from PR review

commit b3709ba
Author: Christian <christian@hansetag.com>
Date:   Tue Sep 24 04:47:04 2024 +0200

    feat: Add NamespaceIdent.parent() (apache#641)

    * Add NamespaceIdent.parent()

    * Use split_last

commit 1533c43
Author: Alon Agmon <54080741+a-agmon@users.noreply.github.com>
Date:   Mon Sep 23 13:39:46 2024 +0300

    feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate (apache#588)

    * adding main function and tests

    * adding tests, removing integration test for now

    * fixing typos and lints

    * fixing typing issue

    * - added support in schmema to convert Date32 to correct arrow type
    - refactored scan to use new predicate converter as visitor and seperated it to a new mod
    - added support for simple predicates with column cast expressions
    - added testing, mostly around date functions

    * fixing format and lic

    * reducing number of tests (17 -> 7)

    * fix formats

    * fix naming

    * refactoring to use TreeNodeVisitor

    * fixing fmt

    * small refactor

    * adding swapped op and fixing CR comments

    ---------

    Co-authored-by: Alon Agmon <alon.agmon@appsflyer.com>

commit e967deb
Author: xxchan <xxchan22f@gmail.com>
Date:   Mon Sep 23 18:34:59 2024 +0800

    feat: expose remove_all in FileIO (apache#643)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit d03c4f8
Author: Scott Donnelly <scott@donnel.ly>
Date:   Mon Sep 23 08:28:52 2024 +0100

    Migrate to arrow-* v53 (apache#626)

    * chore: migrate to arrow-* v53

    * chore: update datafusion to 42

    * test: fix incorrect test assertion

    * chore: update python bindings to arrow 53

commit 88e5e4a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Sep 23 15:26:18 2024 +0800

    chore(deps): Bump crate-ci/typos from 1.24.5 to 1.24.6 (apache#640)

    Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6.
    - [Release notes](https://github.com/crate-ci/typos/releases)
    - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md)
    - [Commits](crate-ci/typos@v1.24.5...v1.24.6)

    ---
    updated-dependencies:
    - dependency-name: crate-ci/typos
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c354983
Author: xxchan <xxchan22f@gmail.com>
Date:   Mon Sep 23 14:50:18 2024 +0800

    doc: improve FileIO doc (apache#642)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 12e12e2
Author: xxchan <xxchan22f@gmail.com>
Date:   Fri Sep 20 19:59:55 2024 +0800

    feat: expose arrow type <-> iceberg type (apache#637)

    * feat: expose arrow type <-> iceberg type

    Previously we only exposed the schema conversion.

    Signed-off-by: xxchan <xxchan22f@gmail.com>

    * add tests

    Signed-off-by: xxchan <xxchan22f@gmail.com>

    ---------

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 3b27c9e
Author: xxchan <xxchan22f@gmail.com>
Date:   Fri Sep 20 18:32:31 2024 +0800

    feat: add Sync to TransformFunction (apache#638)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 34cb81c
Author: Xuanwo <github@xuanwo.io>
Date:   Wed Sep 18 20:18:40 2024 +0800

    chore: Bump opendal to 0.50 (apache#634)

commit cde35ab
Author: FANNG <xiaojing@datastrato.com>
Date:   Fri Sep 13 10:01:16 2024 +0800

    feat: support projection pushdown for datafusion iceberg (apache#594)

    * support projection pushdown for datafusion iceberg

    * support projection pushdown for datafusion iceberg

    * fix ci

    * fix field id

    * remove depencences

    * remove depencences

commit eae9464
Author: Xuanwo <github@xuanwo.io>
Date:   Thu Sep 12 02:06:31 2024 +0800

    refactor(python): Expose transform as a submodule for pyiceberg_core (apache#628)

commit 8a3de4e
Author: Christian <christian@hansetag.com>
Date:   Mon Sep 9 14:45:16 2024 +0200

    Feat: Normalize TableMetadata (apache#611)

    * Normalize Table Metadata

    * Improve readability & comments

commit e08c0e5
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Mon Sep 9 11:57:22 2024 +0800

    fix: Correctly calculate highest_field_id in schema (apache#590)

commit f78c59b
Author: Jack <56563911+jdockerty@users.noreply.github.com>
Date:   Mon Sep 9 03:35:16 2024 +0100

    feat: add `client.region` (apache#623)

commit a5aba9a
Author: Christian <christian@hansetag.com>
Date:   Sun Sep 8 18:36:05 2024 +0200

    feat: SortOrder methods should take schema ref if possible (apache#613)

    * SortOrder methods should take schema ref if possible

    * Fix test type

    * with_order_id should not take reference

commit 5812399
Author: Christian <christian@hansetag.com>
Date:   Sun Sep 8 18:18:41 2024 +0200

    feat: partition compatibility (apache#612)

    * Partition compatability

    * Partition compatability

    * Rename compatible_with -> is_compatible_with

commit ede4720
Author: Christian <christian@hansetag.com>
Date:   Sun Sep 8 16:49:39 2024 +0200

    fix: Less Panics for Snapshot timestamps (apache#614)

commit ced661f
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Sep 8 22:43:38 2024 +0800

    chore(deps): Bump crate-ci/typos from 1.24.3 to 1.24.5 (apache#616)

    Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5.
    - [Release notes](https://github.com/crate-ci/typos/releases)
    - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md)
    - [Commits](crate-ci/typos@v1.24.3...v1.24.5)

    ---
    updated-dependencies:
    - dependency-name: crate-ci/typos
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit cbbd086
Author: Xuanwo <github@xuanwo.io>
Date:   Sun Sep 8 10:29:31 2024 +0800

    feat: Add more fields in FileScanTask (apache#609)

    Signed-off-by: Xuanwo <github@xuanwo.io>

commit 620d58e
Author: Callum Ryan <19956159+callum-ryan@users.noreply.github.com>
Date:   Thu Sep 5 03:44:55 2024 +0100

    feat: SQL Catalog - namespaces (apache#534)

    * feat: SQL Catalog - namespaces

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * feat: use transaction for updates and creates

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: pull out query param builder to fn

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * feat: add drop and tests

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: String to str, remove pub and optimise query builder

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: nested match, remove ok()

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: remove pub, add set, add comments

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: refactor list_namespaces slightly

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: add default properties to all new namespaces

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: remove check for nested namespace

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * chore: add more comments to the CatalogConfig to explain bind styles

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: edit test for nested namespaces

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    ---------

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

commit ae75f96
Author: Søren Dalby Larsen <sdlarsen@gmail.com>
Date:   Tue Sep 3 13:46:48 2024 +0200

    chore: bump crate-ci/typos to 1.24.3 (apache#598)

commit 7aa8bdd
Author: Scott Donnelly <scott@donnel.ly>
Date:   Thu Aug 29 04:37:48 2024 +0100

    Table Scan: Add Row Group Skipping (apache#558)

    * feat(scan): add row group and page index row selection filtering

    * fix(row selection): off-by-one error

    * feat: remove row selection to defer to a second PR

    * feat: better min/max val conversion in RowGroupMetricsEvaluator

    * test(row_group_filtering): first three tests

    * test(row_group_filtering): next few tests

    * test: add more tests for RowGroupMetricsEvaluator

    * chore: refactor test assertions to silence clippy lints

    * refactor: consolidate parquet stat min/max parsing in one place

commit da08e8d
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Aug 28 14:35:55 2024 +0800

    chore(deps): Bump crate-ci/typos from 1.23.6 to 1.24.1 (apache#583)

commit ecbb4c3
Author: Sung Yun <107272191+sungwy@users.noreply.github.com>
Date:   Mon Aug 26 23:57:01 2024 -0400

    Expose Transforms to Python Binding (apache#556)

    * bucket transform rust binding

    * format

    * poetry x maturin

    * ignore poetry.lock in license check

    * update bindings_python_ci to use makefile

    * newline

    * python-poetry/poetry#9135

    * use hatch instead of poetry

    * refactor

    * revert licenserc change

    * adopt review feedback

    * comments

    * unused dependency

    * adopt review comment

    * newline

    * I like this approach a lot better

    * more tests

commit 905ebd2
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Aug 26 20:49:07 2024 +0800

    chore(deps): Update typed-builder requirement from 0.19 to 0.20 (apache#582)

    ---
    updated-dependencies:
    - dependency-name: typed-builder
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit f9c92b7
Author: FANNG <xiaojing@datastrato.com>
Date:   Sun Aug 25 22:31:36 2024 +0800

    fix: Update sqlx from 0.8.0 to 0.8.1 (apache#584)

commit ba66665
Author: FANNG <xiaojing@datastrato.com>
Date:   Sat Aug 24 12:35:36 2024 +0800

    fix: correct partition-id to field-id in UnboundPartitionField (apache#576)

    * correct partition-id to field id in PartitionSpec

    * correct partition-id to field id in PartitionSpec

    * correct partition-id to field id in PartitionSpec

    * xx
c-thiel added a commit to hansetag/iceberg-rust that referenced this pull request Oct 2, 2024
@c-thiel
Merge Builder
7b44d31 
commit 2c9d7e4
Author: Christian Thiel <christian@hansetag.com>
Date:   Wed Oct 2 07:51:50 2024 +0200

    New builder

commit fea1817
Author: Christian Thiel <christian@hansetag.com>
Date:   Tue Oct 1 12:56:10 2024 +0200

    Merge branch 'feat/safe-partition-spec'

commit 1f49c95
Author: Christian Thiel <christian@hansetag.com>
Date:   Tue Oct 1 12:55:02 2024 +0200

    Merge branch 'feat/schema-reassign-field-ids'

commit cda4a0c
Author: congyi wang <58715567+wcy-fdu@users.noreply.github.com>
Date:   Mon Sep 30 11:07:36 2024 +0800

    chore: fix typo in FileIO Schemes  (apache#653)

    * fix typo

    * fix typo

commit af9609d
Author: Scott Donnelly <scott@donnel.ly>
Date:   Mon Sep 30 04:06:14 2024 +0100

    fix: page index evaluator min/max args inverted (apache#648)

    * fix: page index evaluator min/max args inverted

    * style: fix clippy lint in test

commit a6a3fd7
Author: Alon Agmon <54080741+a-agmon@users.noreply.github.com>
Date:   Sat Sep 28 10:10:08 2024 +0300

    test (datafusion): add test for table provider creation (apache#651)

    * add test for table provider creation

    * fix formatting

    * fixing yet another formatting issue

    * testing schema using data fusion

    ---------

    Co-authored-by: Alon Agmon <alon.agmon@appsflyer.com>

commit 87483b4
Author: Alon Agmon <54080741+a-agmon@users.noreply.github.com>
Date:   Fri Sep 27 04:40:08 2024 +0300

    making table provider pub (apache#650)

    Co-authored-by: Alon Agmon <alon.agmon@appsflyer.com>

commit 984c91e
Author: ZENOTME <43447882+ZENOTME@users.noreply.github.com>
Date:   Thu Sep 26 17:56:02 2024 +0800

    avoid to create memory schema operator every time (apache#635)

    Co-authored-by: ZENOTME <st810918843@gmail.com>

commit 4171275
Author: Matheus Alcantara <mths.dev@pm.me>
Date:   Wed Sep 25 08:28:42 2024 -0300

    scan: change ErrorKind when table dont have spanshots (apache#608)

commit ab51355
Author: xxchan <xxchan22f@gmail.com>
Date:   Tue Sep 24 21:25:45 2024 +0800

    fix: compile error due to merge stale PR (apache#646)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 420b4e2
Author: Scott Donnelly <scott@donnel.ly>
Date:   Tue Sep 24 08:20:23 2024 +0100

    Table Scan: Add Row Selection Filtering (apache#565)

    * feat(scan): add row selection capability via PageIndexEvaluator

    * test(row-selection): add first few row selection tests

    * feat(scan): add more tests, fix bug where min/max args swapped

    * fix: ad test and fix for logic bug in PageIndexEvaluator in-clause handler

    * feat: changes suggested from PR review

commit b3709ba
Author: Christian <christian@hansetag.com>
Date:   Tue Sep 24 04:47:04 2024 +0200

    feat: Add NamespaceIdent.parent() (apache#641)

    * Add NamespaceIdent.parent()

    * Use split_last

commit 1533c43
Author: Alon Agmon <54080741+a-agmon@users.noreply.github.com>
Date:   Mon Sep 23 13:39:46 2024 +0300

    feat (datafusion integration): convert datafusion expr filters to Iceberg Predicate (apache#588)

    * adding main function and tests

    * adding tests, removing integration test for now

    * fixing typos and lints

    * fixing typing issue

    * - added support in schmema to convert Date32 to correct arrow type
    - refactored scan to use new predicate converter as visitor and seperated it to a new mod
    - added support for simple predicates with column cast expressions
    - added testing, mostly around date functions

    * fixing format and lic

    * reducing number of tests (17 -> 7)

    * fix formats

    * fix naming

    * refactoring to use TreeNodeVisitor

    * fixing fmt

    * small refactor

    * adding swapped op and fixing CR comments

    ---------

    Co-authored-by: Alon Agmon <alon.agmon@appsflyer.com>

commit e967deb
Author: xxchan <xxchan22f@gmail.com>
Date:   Mon Sep 23 18:34:59 2024 +0800

    feat: expose remove_all in FileIO (apache#643)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit d03c4f8
Author: Scott Donnelly <scott@donnel.ly>
Date:   Mon Sep 23 08:28:52 2024 +0100

    Migrate to arrow-* v53 (apache#626)

    * chore: migrate to arrow-* v53

    * chore: update datafusion to 42

    * test: fix incorrect test assertion

    * chore: update python bindings to arrow 53

commit 88e5e4a
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Sep 23 15:26:18 2024 +0800

    chore(deps): Bump crate-ci/typos from 1.24.5 to 1.24.6 (apache#640)

    Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.5 to 1.24.6.
    - [Release notes](https://github.com/crate-ci/typos/releases)
    - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md)
    - [Commits](crate-ci/typos@v1.24.5...v1.24.6)

    ---
    updated-dependencies:
    - dependency-name: crate-ci/typos
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit c354983
Author: xxchan <xxchan22f@gmail.com>
Date:   Mon Sep 23 14:50:18 2024 +0800

    doc: improve FileIO doc (apache#642)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 12e12e2
Author: xxchan <xxchan22f@gmail.com>
Date:   Fri Sep 20 19:59:55 2024 +0800

    feat: expose arrow type <-> iceberg type (apache#637)

    * feat: expose arrow type <-> iceberg type

    Previously we only exposed the schema conversion.

    Signed-off-by: xxchan <xxchan22f@gmail.com>

    * add tests

    Signed-off-by: xxchan <xxchan22f@gmail.com>

    ---------

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 3b27c9e
Author: xxchan <xxchan22f@gmail.com>
Date:   Fri Sep 20 18:32:31 2024 +0800

    feat: add Sync to TransformFunction (apache#638)

    Signed-off-by: xxchan <xxchan22f@gmail.com>

commit 34cb81c
Author: Xuanwo <github@xuanwo.io>
Date:   Wed Sep 18 20:18:40 2024 +0800

    chore: Bump opendal to 0.50 (apache#634)

commit cde35ab
Author: FANNG <xiaojing@datastrato.com>
Date:   Fri Sep 13 10:01:16 2024 +0800

    feat: support projection pushdown for datafusion iceberg (apache#594)

    * support projection pushdown for datafusion iceberg

    * support projection pushdown for datafusion iceberg

    * fix ci

    * fix field id

    * remove depencences

    * remove depencences

commit eae9464
Author: Xuanwo <github@xuanwo.io>
Date:   Thu Sep 12 02:06:31 2024 +0800

    refactor(python): Expose transform as a submodule for pyiceberg_core (apache#628)

commit 8a3de4e
Author: Christian <christian@hansetag.com>
Date:   Mon Sep 9 14:45:16 2024 +0200

    Feat: Normalize TableMetadata (apache#611)

    * Normalize Table Metadata

    * Improve readability & comments

commit e08c0e5
Author: Renjie Liu <liurenjie2008@gmail.com>
Date:   Mon Sep 9 11:57:22 2024 +0800

    fix: Correctly calculate highest_field_id in schema (apache#590)

commit f78c59b
Author: Jack <56563911+jdockerty@users.noreply.github.com>
Date:   Mon Sep 9 03:35:16 2024 +0100

    feat: add `client.region` (apache#623)

commit a5aba9a
Author: Christian <christian@hansetag.com>
Date:   Sun Sep 8 18:36:05 2024 +0200

    feat: SortOrder methods should take schema ref if possible (apache#613)

    * SortOrder methods should take schema ref if possible

    * Fix test type

    * with_order_id should not take reference

commit 5812399
Author: Christian <christian@hansetag.com>
Date:   Sun Sep 8 18:18:41 2024 +0200

    feat: partition compatibility (apache#612)

    * Partition compatability

    * Partition compatability

    * Rename compatible_with -> is_compatible_with

commit ede4720
Author: Christian <christian@hansetag.com>
Date:   Sun Sep 8 16:49:39 2024 +0200

    fix: Less Panics for Snapshot timestamps (apache#614)

commit ced661f
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Sun Sep 8 22:43:38 2024 +0800

    chore(deps): Bump crate-ci/typos from 1.24.3 to 1.24.5 (apache#616)

    Bumps [crate-ci/typos](https://github.com/crate-ci/typos) from 1.24.3 to 1.24.5.
    - [Release notes](https://github.com/crate-ci/typos/releases)
    - [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md)
    - [Commits](crate-ci/typos@v1.24.3...v1.24.5)

    ---
    updated-dependencies:
    - dependency-name: crate-ci/typos
      dependency-type: direct:production
      update-type: version-update:semver-patch
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit cbbd086
Author: Xuanwo <github@xuanwo.io>
Date:   Sun Sep 8 10:29:31 2024 +0800

    feat: Add more fields in FileScanTask (apache#609)

    Signed-off-by: Xuanwo <github@xuanwo.io>

commit 620d58e
Author: Callum Ryan <19956159+callum-ryan@users.noreply.github.com>
Date:   Thu Sep 5 03:44:55 2024 +0100

    feat: SQL Catalog - namespaces (apache#534)

    * feat: SQL Catalog - namespaces

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * feat: use transaction for updates and creates

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: pull out query param builder to fn

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * feat: add drop and tests

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: String to str, remove pub and optimise query builder

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: nested match, remove ok()

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: remove pub, add set, add comments

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: refactor list_namespaces slightly

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: add default properties to all new namespaces

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: remove check for nested namespace

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * chore: add more comments to the CatalogConfig to explain bind styles

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    * fix: edit test for nested namespaces

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

    ---------

    Signed-off-by: callum-ryan <19956159+callum-ryan@users.noreply.github.com>

commit ae75f96
Author: Søren Dalby Larsen <sdlarsen@gmail.com>
Date:   Tue Sep 3 13:46:48 2024 +0200

    chore: bump crate-ci/typos to 1.24.3 (apache#598)

commit 7aa8bdd
Author: Scott Donnelly <scott@donnel.ly>
Date:   Thu Aug 29 04:37:48 2024 +0100

    Table Scan: Add Row Group Skipping (apache#558)

    * feat(scan): add row group and page index row selection filtering

    * fix(row selection): off-by-one error

    * feat: remove row selection to defer to a second PR

    * feat: better min/max val conversion in RowGroupMetricsEvaluator

    * test(row_group_filtering): first three tests

    * test(row_group_filtering): next few tests

    * test: add more tests for RowGroupMetricsEvaluator

    * chore: refactor test assertions to silence clippy lints

    * refactor: consolidate parquet stat min/max parsing in one place

commit da08e8d
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Wed Aug 28 14:35:55 2024 +0800

    chore(deps): Bump crate-ci/typos from 1.23.6 to 1.24.1 (apache#583)

commit ecbb4c3
Author: Sung Yun <107272191+sungwy@users.noreply.github.com>
Date:   Mon Aug 26 23:57:01 2024 -0400

    Expose Transforms to Python Binding (apache#556)

    * bucket transform rust binding

    * format

    * poetry x maturin

    * ignore poetry.lock in license check

    * update bindings_python_ci to use makefile

    * newline

    * python-poetry/poetry#9135

    * use hatch instead of poetry

    * refactor

    * revert licenserc change

    * adopt review feedback

    * comments

    * unused dependency

    * adopt review comment

    * newline

    * I like this approach a lot better

    * more tests

commit 905ebd2
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Date:   Mon Aug 26 20:49:07 2024 +0800

    chore(deps): Update typed-builder requirement from 0.19 to 0.20 (apache#582)

    ---
    updated-dependencies:
    - dependency-name: typed-builder
      dependency-type: direct:production
    ...

    Signed-off-by: dependabot[bot] <support@github.com>
    Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

commit f9c92b7
Author: FANNG <xiaojing@datastrato.com>
Date:   Sun Aug 25 22:31:36 2024 +0800

    fix: Update sqlx from 0.8.0 to 0.8.1 (apache#584)

commit ba66665
Author: FANNG <xiaojing@datastrato.com>
Date:   Sat Aug 24 12:35:36 2024 +0800

    fix: correct partition-id to field-id in UnboundPartitionField (apache#576)

    * correct partition-id to field id in PartitionSpec

    * correct partition-id to field id in PartitionSpec

    * correct partition-id to field id in PartitionSpec

    * xx
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@Xuanwo Xuanwo Xuanwo approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@xxchan @Xuanwo