[Epic] A collection of items related to processing larger than memory datasets (via spilling, externalized algorithm, etc)

[alamb]

Is your feature request related to a problem or challenge?

This epic attempts to organize attempts to improve DataFusion's ability to process datasets that are larger than fit in configured memory budget

Some of DataFusion's "pipeline blocking" operations (SortExec and HashGroupBy) already do work with datasets that are larger than fit in memory, but the performance and usability could be improved

• External sorting not working for (maybe only for string columns??) #12136
• [EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation #13123
• Optimized spill file format #14078
• A complete solution for stable and safe sort with spill #14692
• Ensure GroupByHash does not error when trying to spill (calling try_resize where error is not acceptible) #14851
• [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271

Note: Joins are another operation that can run out of memory and will error (rather than falling back to some other strategy like Sort-Merge-Join for example). If people are interested in making this better, I think we could organize another project

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

Since there was recent excitement /activity about better sorting behavior, I file an EPIC for just that:

• A complete solution for stable and safe sort with spill #14692

[alamb]

I broke this ticket into two follow on ones, one focused on hashing and one focused on sorting:

• [EPIC] A collection of tickets for improving sorting larger than memory datasets / spilling sorts #15271
• [EPIC] Improved Externalized / Spilling / Large than Memory Hash Aggregation #13123

So let's close this one