fix: handle interleaved HashJoin projections in sort pushdown#23591
fix: handle interleaved HashJoin projections in sort pushdown#23591xudong963 wants to merge 1 commit into
Conversation
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing xudong963/fix-interleaved-hash-join-sort-pushdown-upstream (ad02f44) to 4b68de6 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing xudong963/fix-interleaved-hash-join-sort-pushdown-upstream (ad02f44) to 4b68de6 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing xudong963/fix-interleaved-hash-join-sort-pushdown-upstream (ad02f44) to 4b68de6 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
Which issue does this PR close?
Rationale for this change
Sort pushdown through a
HashJoinExecwith an embedded projection assumed that all projected left-side columns formed a contiguous prefix. That assumption is not valid when the projection interleaves columns from the two join inputs.For a
CollectLeftright join projected as[right.col_a, left.nullable_col], a required sort on output column 1 was therefore mistaken for a sort from the right child. The optimizer pushed a sort onright.col_abelow the join, andSanityCheckPlanrejected the resulting plan because it did not satisfy the parent sort onleft.nullable_col.What changes are included in this PR?
JoinSide, instead of comparing output positions with the number of projected left columns.SanityCheckPlanand snapshots the valid optimized plan.Are these changes tested?
Yes.
main, the new regression test fails inSanityCheckPlan: the right input is sorted bycol_a, while the parent requiresnullable_col.cargo fmt --allcargo clippy --all-targets --all-features -- -D warningsRUST_BACKTRACE=1 cargo test --profile ci --exclude datafusion-examples --exclude datafusion-benchmarks --exclude datafusion-cli --workspace --lib --tests --bins --features avro,json,backtrace,extended_tests,recursive_protection,parquet_encryptionAre there any user-facing changes?
There are no API changes. Queries whose hash join projection interleaves columns from both inputs no longer receive an invalid sort pushdown that can fail physical-plan sanity checking.