Release DataFusion `47.0.0` (April 2025) #15072

alamb · 2025-03-07T19:20:52Z

Is your feature request related to a problem or challenge?

Tracking ticket for next release, also a place to track desired inclusions

Previous release will be https://crates.io/crates/datafusion/45.0.0 (likely Feb 1, 2025) December 31, 2024 so next major release would be around March 1, 2025

Steps:

Update version and changelog: Update version to 47.0.0, add CHANGELOG #15731
Test with DataFusion Python:
Test with DataFusion Comet - chore: Upgrade to datafusion 47.0.0-rc1 and arrow-rs 55.0.0 datafusion-comet#1563
Test with delta.rs: feat: upgrade to DataFusion 47.0.0 delta-io/delta-rs#3378
Test with SailHQ: Test DataFusion 47 lakehq/sail#434
Write upgrade guide: Upgrade guide for DataFusion 47.0.0 #15707
Test with parquet viewer
Voting Thread: https://lists.apache.org/thread/zrq9x9gf51r8b6m9qokf2q75kh251rm6
Create ticket for next release: Release DataFusion 48.0.0 (June 2025) #15771

Prior release tickets:

45.0.0: Release DataFusion 45.0.0 #14008
46.0.0: Release DataFusion 46.0.0 #14123

Changes to add to upgrade guide

These PRs made changes that deserve a mention in the upgrade guide

Features to mention in the blog (if they make it)

Bugs that would be good to fix

Community Wishlist

@alamb 's wishlist

The text was updated successfully, but these errors were encountered:

xudong963 · 2025-03-11T09:34:03Z

@alamb, I'll also be in charge of this release.

alamb · 2025-03-11T11:28:43Z

@XiangpengHao also offered to test with the parquet viewer prior to 47: #15102 (comment)

xudong963 · 2025-03-12T09:40:33Z

@XiangpengHao also offered to test with the parquet viewer prior to 47: #15102 (comment)

That's great, added it to release steps

shehabgamin · 2025-03-19T03:59:17Z

I feel like this may be important enough to try to get into the release. Does anyone else have thoughts?

Comparison Operators for Decimals of Different Precisions and Scales #15174

alamb · 2025-03-19T11:15:36Z

I feel like this may be important enough to try to get into the release. Does anyone else have thoughts?

Comparison Operators for Decimals of Different Precisions and Scales #15174

Seems reasonable to me -- I have added it to the "good to get in " list

alamb · 2025-03-19T17:17:22Z

~~I believe the voting deadline has passed and I will now will prmote / publish it.~~

Update: wrong ticket (I was looking for 46.0.0

xudong963 · 2025-03-26T03:55:56Z

The PR #15266 has significantly improved performance, so I added it to the blog section.

xudong963 · 2025-03-28T05:08:59Z

@alamb I think we can start testing the 47.0.0 in the second week of April and begin the release process at the end of that week. What do you think?

alamb · 2025-03-28T18:09:10Z

@alamb I think we can start testing the 47.0.0 in the second week of April and begin the release process at the end of that week. What do you think?

I think it sounds like a great idea -- thank you @xudong963

For your planning purposes I will be away the week of April 21 -- so perhaps we can start testing a week earlier (week of April 7 so we have time to complete / fix issues prior to April 14)

shehabgamin · 2025-03-28T21:06:45Z

Happy to test whenever!

xudong963 · 2025-03-29T03:53:52Z

For your planning purposes I will be away the week of April 21 -- so perhaps we can start testing a week earlier (week of April 7 so we have time to complete / fix issues prior to April 14)

That sounds good!

rluvaton · 2025-04-02T18:47:01Z

Would really appreciate if could add the following PR to the release as well:

fix: update group by columns for merge phase after spill #15531

xudong963 · 2025-04-03T09:17:10Z

Would really appreciate if could add the following PR to the release as well:

fix: update group by columns for merge phase after spill #15531

Sure, I added it. Given that there're two approvals, I think it'll be included in DF47, and thanks for your fix

xudong963 · 2025-04-07T03:22:43Z

Hey guys, happy new week, let's start testing the incoming DF47 this week! 🚀

alamb · 2025-04-07T11:08:32Z

Makes sense. Thanks @xudong963

andygrove · 2025-04-07T21:48:58Z

We have started testing Comet with the latest DF from main. I added a link to the Comet PR in this PR's description.

apache/datafusion-comet#1563

XiangpengHao · 2025-04-08T03:52:46Z

I have tested the Parquet viewer with the latest main and found no problems.

But I hit a TPC-H panic when running on LiquidCache: XiangpengHao/liquid-cache#158

I'm working on digging into the root cause..

Other breaking changes I observed:

Need to handle DisplayFormatType::TreeRender for execution plans.
map_partial_batch is removed from schema mapper
page_pruning_predicate is removed from public api

xudong963 · 2025-04-08T07:50:46Z

Thank you @XiangpengHao , I added the breaking changes that you mentioned to the summary of the issue.

Blizzara · 2025-04-08T09:51:08Z

I tested the latest DF against our tests - the Substrait consumer is broken when it comes to renaming Struct fields' insides, due to #15239 (comment). I'll try to get a fix up.

Edit: fix here #15634

alamb · 2025-04-12T12:19:46Z

I did a little investigating, but I don't have time for a couple of days to dive in deeper. This appears to be related to #15542 @UBarney do you know why the sql unit tests needed to be changed to pass? It seems like we have a regression in datafusion-python related to this change.

I think @andygrove filed a ticket for this one

Regression in last_value functionality #15676

I didn't fully follow the discussion -- but it seems like that issue has been closed

timsaucer · 2025-04-12T12:55:37Z

I did a little investigating, but I don't have time for a couple of days to dive in deeper. This appears to be related to #15542 @UBarney do you know why the sql unit tests needed to be changed to pass? It seems like we have a regression in datafusion-python related to this change.

I think @andygrove filed a ticket for this one
* [Regression in `last_value` functionality #15676](https://github.com/apache/datafusion/issues/15676)
I didn't fully follow the discussion -- but it seems like that issue has been closed

I've read the discussion now and I think I'm in agreement that it's not an actual regression since the aggregation has no deterministic outcome without ordering assigned.

alamb · 2025-04-12T15:36:11Z

Do we want to hold DF 47 release for the arrow upgrade too?
I think it is possible (arrow will hopefully be released at the end of this week -- and we could make the DF release candidate next week...)

This would be great for the Comet project.

My only remaining question is if we want to upgrade arrow in this release as well

The upgrade PR is here:

Upgrade to arrow/parquet 55, and object_store to 0.12.0 and pyo3 to 0.24.0 #15466

Since it also upgrades object_store and pyo3 it is somewhat more disruptive.

jayzhan211 · 2025-04-13T02:00:26Z

My only remaining question is if we want to upgrade arrow in this release as well

+1 for upgrading all the dependencies

andygrove · 2025-04-13T02:51:30Z

I am also +1 for upgrading the dependencies (for selfish reasons; we are waiting on an arrow feature to help with INT96 timestamps in Parquet)

alamb · 2025-04-13T10:49:19Z

Thanks @jayzhan211 for the approval and for the discussion. I'll plan to merge #15466 tomorrow then unless we want to discuss it further.

alamb · 2025-04-14T10:45:52Z

Ok, I just merged #15466 / upgrade to dependencies (arrow/object_store/parquet) 47.0.0

I don't know of anything else we are now waiting on for this release. I suggest we make the release notes PR and generate a release candidate

BTW I will be offline for about a week starting this Friday April 18 or Saturday so I likely won't be able to help with the release until I return. Hopefully another PMC member can do the final approval / release to crates.io if it isn't ready before I leave.

alamb · 2025-04-14T10:49:27Z

I also filed Upgrade guide for DataFusion 47.0.0 #15707 to track writing the upgrade guide

Closes #1037 ### Change list - Bump `arrow` to `55` and `parquet` to `55` - Temporarily deactivates the `datafusion` integration until datafusion publishes its version `47` (apache/datafusion#15072), so that we can progress with the `arrow` 55 upgrade now. - Update JS and Python APIs for latest `parquet`. - Means we no longer need an initial `HEAD` request for Parquet files before reading metadata.

andygrove · 2025-04-14T16:40:35Z

@alamb I am hoping that we can merge #15537 for this release. It was just rebased now that the arrow-rs upgrade is merged.

gabotechs · 2025-04-15T08:45:39Z

Thanks for putting this together! If we could additionally get #14412 in, that would be awesome 🙏

xudong963 · 2025-04-15T11:34:51Z

I don't know of anything else we are now waiting on for this release. I suggest we make the release notes PR and generate a release candidate

@alamb It seems there are still one or two PRs that want to be included, so how about making release notes tomorrow(UTC+8, after I get up)

alamb · 2025-04-15T17:53:04Z

@alamb I am hoping that we can merge #15537 for this release. It was just rebased now that the arrow-rs upgrade is merged.

Thanks for putting this together! If we could additionally get #14412 in, that would be awesome 🙏

Sorry @gabotechs -- I just merged that one!

@alamb It seems there are still one or two PRs that want to be included, so how about making release notes tomorrow(UTC+8, after I get up)

SOunds like a good plan. I'll take a pass through the outstanding PRs again to see if there is anything else we can/should merge

Thank you all

alamb · 2025-04-16T16:50:53Z

I just merged the version + changelog PR from @xudong963

Update version to 47.0.0, add CHANGELOG #15731

I also created a branch-47 here for the release:

https://github.com/apache/datafusion/tree/branch-47

@xudong963, given I think it is late in your timezone, I'll plan to make an RC in a few hours unless you let me know otherwise.

alamb · 2025-04-16T20:10:16Z

I have made a release candidate and started a voting thread: https://lists.apache.org/thread/zrq9x9gf51r8b6m9qokf2q75kh251rm6

alamb · 2025-04-16T20:13:23Z

Note that I will be away starting April 18, and so likely can not complete the vote / release process until April 26. @andygrove would it be possible for you to complete the voting process / publish the release to crates.io for this release?

andygrove · 2025-04-16T20:17:23Z

Note that I will be away starting April 18, and so likely can not complete the vote / release process until April 26. @andygrove would it be possible for you to complete the voting process / publish the release to crates.io for this release?

Yes, I would be happy to do that. I will be offline on Saturday, but can take care of it on Sunday or Monday.

alamb · 2025-04-16T21:35:21Z

Awesome -- we have a plan! I hope to work on the upgrade guide and maybe even a blog post about this release, but we'll see if I have the time

alamb · 2025-04-17T12:28:16Z

Here is a draft upgrade guide:

Add DataFusion 47.0.0 Upgrade Guide #15749

alamb · 2025-04-19T10:31:47Z

I filed the following ticket for the next release:

Release DataFusion 48.0.0 (June 2025) #15771

alamb · 2025-04-28T14:28:36Z

DataFusion 47 is on crates.io https://crates.io/crates/datafusion/47.0.0

So closing this one down

Getting ready for the [datafusion 47 release](apache/datafusion#15072). ## Current issues - [x] apache/datafusion#15072 - [x] duckdb-rs depends on arrow 54, opened a PR to fix - duckdb/duckdb-rs#496. - [ ] object-store 0.12 has a regression on Azure, not sure what's the priority here but shouldn't be too hard to find the root cause if we care. apache/arrow-rs-object-store#320

alamb added the enhancement New feature or request label Mar 7, 2025

alamb mentioned this issue Mar 7, 2025

Release DataFusion 46.0.0 #14123

Closed

26 tasks

xudong963 assigned xudong963 and alamb Mar 11, 2025

alamb mentioned this issue Mar 11, 2025

Release DataFusion 46.0.1 Patch or 46.1.0 minor release (March 2025) #15151

Closed

3 tasks

This was referenced Mar 13, 2025

chore: remove deprecated variants of UDF's invoke (invoke, invoke_no_args, invoke_batch) #15123

Merged

Incorrect cast of integer columns to utf8 when comparing with utf8 constant #15161

Open

This was referenced Mar 20, 2025

feat: introduce JoinSetTracer trait for tracing context propagation in spawned tasks #14547

Merged

Refactor: add FileGroup structure for Vec<PartitionedFile> #15379

Merged

emilk mentioned this issue Apr 1, 2025

Can't use arrow datatype Duration with datafusion 46 rerun-io/rerun#9440

Open

alamb mentioned this issue Apr 7, 2025

Weekly Plan (Andrew Lamb) April 7, 2025 #15616

Closed

12 tasks

alamb mentioned this issue Apr 14, 2025

Upgrade guide for DataFusion 47.0.0 #15707

Closed

9 tasks

kylebarron mentioned this issue Apr 14, 2025

Bump arrow to 55 geoarrow/geoarrow-rs#1038

Merged

AdamGS mentioned this issue Apr 15, 2025

Upgrade to datafusion 47 vortex-data/vortex#3016

Merged

3 tasks

alamb mentioned this issue Apr 15, 2025

When datafusion.execution.parquet.coerce_int96 is set, timestamp type is still reported as Timestamp(nanoseconds) #15721

Closed

xudong963 mentioned this issue Apr 16, 2025

Update version to 47.0.0, add CHANGELOG #15731

Merged

alamb mentioned this issue Apr 16, 2025

[branch-47] Final release note touchups #15740

Merged

alamb mentioned this issue Apr 19, 2025

Release DataFusion 48.0.0 (June 2025) #15771

Open

20 tasks

alamb closed this as completed Apr 28, 2025

Release DataFusion 47.0.0 (April 2025) #15072

Release DataFusion 47.0.0 (April 2025) #15072

Comments

alamb commented Mar 7, 2025 • edited Loading

Is your feature request related to a problem or challenge?

Changes to add to upgrade guide

Features to mention in the blog (if they make it)

Bugs that would be good to fix

Community Wishlist

@alamb 's wishlist

xudong963 commented Mar 11, 2025

alamb commented Mar 11, 2025

xudong963 commented Mar 12, 2025

shehabgamin commented Mar 19, 2025 • edited by alamb Loading

alamb commented Mar 19, 2025

alamb commented Mar 19, 2025 • edited Loading

xudong963 commented Mar 26, 2025

xudong963 commented Mar 28, 2025

alamb commented Mar 28, 2025

shehabgamin commented Mar 28, 2025

xudong963 commented Mar 29, 2025

rluvaton commented Apr 2, 2025

xudong963 commented Apr 3, 2025

xudong963 commented Apr 7, 2025

alamb commented Apr 7, 2025

andygrove commented Apr 7, 2025 • edited Loading

XiangpengHao commented Apr 8, 2025

xudong963 commented Apr 8, 2025

Blizzara commented Apr 8, 2025 • edited Loading

alamb commented Apr 12, 2025

timsaucer commented Apr 12, 2025

alamb commented Apr 12, 2025

jayzhan211 commented Apr 13, 2025

andygrove commented Apr 13, 2025

alamb commented Apr 13, 2025

alamb commented Apr 14, 2025 • edited Loading

alamb commented Apr 14, 2025 • edited Loading

andygrove commented Apr 14, 2025

gabotechs commented Apr 15, 2025

xudong963 commented Apr 15, 2025 • edited Loading

alamb commented Apr 15, 2025

alamb commented Apr 16, 2025

alamb commented Apr 16, 2025

alamb commented Apr 16, 2025

andygrove commented Apr 16, 2025

alamb commented Apr 16, 2025

alamb commented Apr 17, 2025

alamb commented Apr 19, 2025

alamb commented Apr 28, 2025

Release DataFusion `47.0.0` (April 2025) #15072

Release DataFusion `47.0.0` (April 2025) #15072

alamb commented Mar 7, 2025 •

edited

Loading

shehabgamin commented Mar 19, 2025 •

edited by alamb

Loading

alamb commented Mar 19, 2025 •

edited

Loading

andygrove commented Apr 7, 2025 •

edited

Loading

Blizzara commented Apr 8, 2025 •

edited

Loading

alamb commented Apr 14, 2025 •

edited

Loading

alamb commented Apr 14, 2025 •

edited

Loading

xudong963 commented Apr 15, 2025 •

edited

Loading