Skip to content

fix: dedup commutative AND/OR operands when canonicalize is disabled#23615

Open
1fanwang wants to merge 4 commits into
apache:mainfrom
1fanwang:fix/14943-simplify-swapped-operands-and-or
Open

fix: dedup commutative AND/OR operands when canonicalize is disabled#23615
1fanwang wants to merge 4 commits into
apache:mainfrom
1fanwang:fix/14943-simplify-swapped-operands-and-or

Conversation

@1fanwang

@1fanwang 1fanwang commented Jul 15, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Revives #21870, which was closed by the stale bot. The direction was agreed with @asolimando and sql_planner was benchmarked by @alamb there (results, no measurable change).

Rationale for this change

SimplifyExpressions disables canonicalization for LogicalPlan::Join (see #8780), so the AND/OR dedup in expr_contains_inner — which compares each conjunct/disjunct leaf with structural == — cannot recognize duplicates that differ only by commutative operand order (A = B vs B = A). Simplifying inside a join filter (the delta-rs MERGE case reported in #14943) keeps the duplicate across simplifier cycles because nothing normalizes operand order first.

@alamb pointed at the fix in the issue thread: CSE already dedups A = B / B = A via NormalizeEq; route the simplifier's leaf comparison through the same trait.

What changes are included in this PR?

expr_contains_inner compares leaves with Expr::normalize_eq instead of ==. NormalizeEq handles +, *, &, |, ^, =, != commutatively and falls back to structural == for everything else, so non-commutative rules and the existing !needle.is_volatile() guard are unchanged. A regression test covers the AND, OR, and 3-conjunct nested forms.

Are these changes tested?

Yes. The new test_simplify_swapped_operands_in_and_or_no_canonicalize fails on main (the duplicate passes through unchanged) and passes with the fix. The full datafusion-optimizer lib suite and cargo clippy are clean, and the sql_planner benchmark linked above shows no measurable change.

Are there any user-facing changes?

No public API changes. AND/OR chains containing commutative-equivalent duplicates now collapse even when the simplifier's canonicalizer is disabled (currently the LogicalPlan::Join path). Canonicalize-on paths produce the same output as before.

`SimplifyExpressions` disables canonicalization for `LogicalPlan::Join`
(see apache#8780), so `expr_contains_inner`'s structural `==` cannot recognize
duplicate predicates that differ only by operand order such as `A = B`
and `B = A`. Workloads that simplify expressions inside a join filter --
notably the delta-rs MERGE case reported in apache#14943 -- end up retaining
the duplicate even after multiple simplifier cycles.

Switching the leaf comparison to `NormalizeEq` (the same trait CSE uses
to recognize common sub-expressions for `+`, `*`, `&`, `|`, `^`, `=`,
`!=`) makes the dedup commutativity-aware while falling back to
structural `==` for everything else, so existing volatility guards and
non-commutative rules are unchanged.

Closes apache#14943.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
@github-actions github-actions Bot added the optimizer Optimizer rules label Jul 15, 2026
@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Jul 15, 2026

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @1fanwang

physical_plan
01)ProjectionExec: expr=[column1@0 = 1 as opt1, column1@0 = 2 AND column1@0 != 2 as noopt1, column1@0 = 4 as opt2, column1@0 != 5 AND column1@0 = 5 as noopt2]
02)--DataSourceExec: partitions=1, partition_sizes=[1]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took the liberty of adding some slt coverage for this feature too

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These fail like this on main

warning: `datafusion-sqllogictest` (test "sqllogictests") generated 1 warning
    Finished `ci` profile [unoptimized] target(s) in 1m 45s
     Running bin/sqllogictests.rs (target/ci/deps/sqllogictests-01bd002bb437dcb6)
Running with 16 test threads (available parallelism: 16)
Completed 493 test files in 6 seconds                                                                                          External error: 2 errors in file /Users/andrewlamb/Software/datafusion/datafusion/sqllogictest/test_files/simplify_expr.slt

1. query result mismatch:
[SQL] explain select * from t1 join t2 on t1.a = t2.b or t2.b = t1.a;
[Diff] (-expected|+actual)
    logical_plan
-   01)Inner Join: t1.a = t2.b
+   01)Inner Join:  Filter: t1.a = t2.b OR t2.b = t1.a
    02)--TableScan: t1 projection=[a]
    03)--TableScan: t2 projection=[b]
    physical_plan
-   01)HashJoinExec: mode=CollectLeft, join_type=Inner, on=[(a@0, b@0)]
+   01)NestedLoopJoinExec: join_type=Inner, filter=a@0 = b@1 OR b@1 = a@0
    02)--DataSourceExec: partitions=1, partition_sizes=[1]
    03)--DataSourceExec: partitions=1, partition_sizes=[1]
at /Users/andrewlamb/Software/datafusion/datafusion/sqllogictest/test_files/simplify_expr.slt:162


2. query result mismatch:
[SQL] explain select * from t1 join t2 on t1.a > t2.b and (t1.a + t2.b > 1 or t2.b + t1.a > 1);
[Diff] (-expected|+actual)
    logical_plan
-   01)Inner Join:  Filter: t1.a > t2.b AND t1.a + t2.b > Int32(1)
+   01)Inner Join:  Filter: t1.a > t2.b AND (t1.a + t2.b > Int32(1) OR t2.b + t1.a > Int32(1))
    02)--TableScan: t1 projection=[a]
    03)--TableScan: t2 projection=[b]
    physical_plan
-   01)NestedLoopJoinExec: join_type=Inner, filter=a@0 > b@1 AND a@0 + b@1 > 1
+   01)NestedLoopJoinExec: join_type=Inner, filter=a@0 > b@1 AND (a@0 + b@1 > 1 OR b@1 + a@0 > 1)
    02)--DataSourceExec: partitions=1, partition_sizes=[1]
    03)--DataSourceExec: partitions=1, partition_sizes=[1]
at /Users/andrewlamb/Software/datafusion/datafusion/sqllogictest/test_files/simplify_expr.slt:181

@alamb alamb added the performance Make DataFusion faster label Jul 15, 2026
@1fanwang

Copy link
Copy Markdown
Author

Thanks a lot @alamb checking now

@alamb alamb enabled auto-merge July 16, 2026 18:58
@alamb alamb disabled auto-merge July 16, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Optimizer rules performance Make DataFusion faster sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expr simplifier doesn't simplify exprs that are same if you swap lhs with rhs regardless of cycles

2 participants