Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
### What changes were proposed in this pull request? #36773 translate linear regression aggregate functions for pushdown. Although `REGR_AVGX` and `REGR_AVGY` are replaced to `AVG` in runtime, we can pushdown `AVG` to achieve the same result that push down `REGR_AVGX` and `REGR_AVGY`. Take `RegrAvgX` as an example, `RegrAvgX` replaced with `Average(If(And(IsNotNull(left), IsNotNull(right)), right, Literal.create(null, right.dataType)))` in runtime and then the latter will be optimized as `Average(CaseWhen(Seq[(And(IsNotNull(left), IsNotNull(right)), right)], Some(Literal.create(null, right.dataType))))` We can see `Literal.create(null, right.dataType)` here, `visitLiteral` of `JDBCSQLBuilder` cannot processing the null literal in the correct way. So we need to fix the issue too. ### Why are the changes needed? Let Aggregate pushdown supports `REGR_AVGX` and `REGR_AVGY`. ### Does this PR introduce _any_ user-facing change? 'No'. New feature. ### How was this patch tested? New test cases. Closes #37126 from beliefer/SPARK-39385_followup. Authored-by: Jiaan Geng <beliefer@163.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
- Loading branch information