General framework to decorrelate the subqueries

[mingmwang]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In the current DataFusion, it has very limited support for correlated subqueries. It can only decorrelate the (NOT) IN/Exists predicate subqueries to Semi/Anti Joins. Even in the simplest IN/Exists cases, if the correlated expressions are not in the Filter/Join conditions, the current decorrelate rules will not support them.

In the paper "Unnesting Arbitrary Queries" by T. Neumann; A. Kemper
(http://www.btw-2015.de/res/proceedings/Hauptband/Wiss/Neumann-Unnesting_Arbitrary_Querie.pdf). It raise a mechanism to unnest arbitrary queries. This was already implemented by the Hyper DB:

For example:
select * from orders
where 1 in (select 1 from part left join (select l_partkey from lineitem where o_orderkey = 2) lineitem on p_partkey = lineitem.l_partkey)

https://hyper-db.de/interface.html#

Both SparkSQL and PostgreSQL do not support decorrelate such kind of queries.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

[duongcongtoai]

According to the discussions in this issue, i think we can list the following items to support a subqueries decorrelation framework:

• Unify the optimizor for correlated query, regardless the query type (exists query, scalar query etc)
• Support flexible decorrelation scheme (simple vs general approach), we can achieve this by following the algorithm mentioned in the 2nd paper. There is a prerequisite to introduce an index algebra during the rewrite. This index requires a pre-traversing over the whole query to detect all non-trivial subqueries, and answer the question whether simple unnesting is sufficient, or should the framework continue with the general approach
• Implement general purpose + recursive aware subquery decorrelation for the most major operators (projection, filter, group by) using the top-down algorithm mentioned in the 2nd paper
• Gradually support more complex expression (group by, order, limit, window function)

[alamb]

• I am trying to organize a join task force for planning joins / subqueries: [DISCUSSION] JOIN "task force" / project team #15885

[suibianwanwank]

Unify the optimizor for correlated query, regardless the query type (exists query, scalar query etc)

I think a crucial starting point is to transform all correlated exprs into dependent join. (I'm unsure if we need to implement its Execution Plan before completing all decorrelation).

[alamb]

In case others haven't heard, @irenjj is working on additional subquery support as part of a Google Summer of Code Project (where @jayzhan211 and I are helping mentor).

Perhaps @suibianwanwank and @duongcongtoai would be interested in being involved too

[duongcongtoai]

cool, actually in this PR: #16016 i'm trying to introduce a unified structure to do decorrelation for the following usecases:

• simple decorrelating plan with linear operators (straight forward projection/predicate pull up/ dependent join push down)
• more complex decorrelation (group by, windows, limit ...)
• recursive decorrelation/nested decorrelation

The parts that are not implemented are left with unimplemented! errors. I'm trying to make the PR mergable, so everyone can continue to contribute in parallel, but my current goal is to implement enough to satisfy existing slt tests

[alamb]

From what I can see, this ticket (a general framework to decorrelate subqueries) is going to be the core of any subsequent improvements for subqueries, so I suggest we pool our efforts together to focus on this first.

I suggest, we all first review the PR from @duongcongtoai

• [WIP] refactor: framework for subquery decorrelation #16016

cc @suibianwanwank @duongcongtoai @irenjj

[alamb]

I recommend we also do some research on existing systems. Can someone provide links to existing implementations in other systems?

• DuckDB
• Spark
• Apache Calcite ( Maybe @suibianwanwank can help, as they have significant experience with this part of calcite's optimizer in this area -- https://github.com/apache/calcite/issues?q=is%3Apr+is%3Aopen+author%3Asuibianwanwank)

[irenjj]

I recommend we also do some research on existing systems. Can someone provide links to existing implementations in other systems?

• DuckDB
• Spark
• Apache Calcite ( Maybe @suibianwanwank can help, as they have significant experience with this part of calcite's optimizer in this area -- https://github.com/apache/calcite/issues?q=is%3Apr+is%3Aopen+author%3Asuibianwanwank)

For DuckDB: basic logic is in plan_subquery.cpp file(can't find a specific pr) https://github.com/duckdb/duckdb/blob/main/src/planner/binder/query_node/plan_subquery.cpp, For "Unnesting Arbitrary Subqueries," there is a complete implementation.
And 2nd paper implementation: duckdb/duckdb#17294

[suibianwanwank]

Thanks to @irenjj remind, I found this based on your input: https://github.com/duckdb/duckdb/blob/main/src/planner/binder/query_node/plan_subquery.cpp. If I understand correctly, this should be DuckDB's logic for converting subQuery to DependentJoin.
Additionally, in Calcite, the corresponding logic is here: https://github.com/apache/calcite/blob/main/core/src/main/java/org/apache/calcite/rel/rules/SubQueryRemoveRule.java
There should be many similarities between the two.

[duongcongtoai]

In this draft, dependent join detection is implemented inside this function. But i wonder if creating an explicit LogicalPlan::DependentJoin make sense in our case? if the only place it is being used is inside the optimizor that does the decorrelation

In duckdb PR he mentioned:

With this PR, that changes. Subqueries are fully algebraized, before entering a single decorrelation pass. This potentially provides optimization opportunities, that would not have been possible before.

I can relates this to the idea in the 2nd paper

At this point the accessing list is empty and we have two choices: either we introduce a
join with 𝐷 := Π𝑎:=𝑇1.𝑎 (𝜎𝑓 =123 (𝑇1)) and use that to substitute 𝑇1.𝑎; or we exploit the fact
that all outer column references are bound in cclasses and we simply replace 𝑇1.𝑎 with
the equivalent 𝑇2.𝑎. The optimal choice depends on the selectivity of 𝜎

which corresponds to this part of the draft PR, and is still within the detail implementation of a specific optimizor

[suibianwanwank]

I'm not sure if I fully understand your point. In this paper, the decorrelate should be based on DependentJoin, as can be seen from the example relational algebra diagram.

[duongcongtoai]

There are one thing we surely know that should be implemented: detect which nodes in the LogicalPlan AST is a dependent join node. However, we don't need to create a new LogicalPlan type like LogicalPlan::DependentJoin, if this plan type is only an intermediate form during the decorrelation phase, and i expect after the rewriting completes, no more dependent join node exists.

=> We can keep this concept internally to the decorelation framework