Evaluate use of selection vectors in scan-filter-join operations

[andygrove]

What is the problem the feature request solves?

It is very common to have scan -> filter as inputs to a join. The copying of data in the filter can be expensive when the batch contains strings and complex types, and the result of the filter is discarded after the join.

I believe that it would be more efficient to have the join use a selection vector to read inputs from the scanned batch rather than perform a filter.

This issue is for tracking the work to create a small prototype to demonstrate. If succesful, then we can discuss making changes in upstream DataFusion to add support for a new ColumnarValue::ArrayWithSelectionVector and then add a specialization in SortMergeJoin to take advantage of this.

Describe the potential solution

No response

Additional context

No response

[viirya]

Related issue at arrow-rs: apache/arrow-rs#3620

[andygrove]

This paper may have useful information:

"Filter Representation in Vectorized Query Execution"
https://db.cs.cmu.edu/papers/2021/ngom-damon2021.pdf