Skip map_expressions rebuild for Extension nodes with empty expressions#21701
Open
zhuqi-lucas wants to merge 1 commit intoapache:mainfrom
Open
Skip map_expressions rebuild for Extension nodes with empty expressions#21701zhuqi-lucas wants to merge 1 commit intoapache:mainfrom
zhuqi-lucas wants to merge 1 commit intoapache:mainfrom
Conversation
8157855 to
90d4f6e
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes LogicalPlan::map_expressions for Extension nodes that expose no expressions, avoiding unnecessary cloning/rebuild work, and adds a Criterion micro-benchmark to quantify the improvement.
Changes:
- Add an early return in
map_expressionsforExtensionnodes whennode.expressions()is empty. - Add a Criterion benchmark measuring
map_expressionsbehavior forExtensionnodes with/without expressions across varying child counts. - Register the new benchmark in
datafusion-expr’s dev-dependencies and bench configuration.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| datafusion/expr/src/logical_plan/tree_node.rs | Skip rebuilding Extension nodes in map_expressions when there are no expressions to transform. |
| datafusion/expr/benches/map_expressions.rs | Add micro-benchmark for map_expressions on Extension nodes with/without expressions. |
| datafusion/expr/Cargo.toml | Add Criterion dev-dependency and register the new bench target. |
| Cargo.lock | Lockfile updates due to added Criterion dependency/versioning. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
When an Extension node has no expressions, `map_expressions` was still cloning all inputs and calling `with_exprs_and_inputs` to reconstruct the node — wasted work since there are no expressions to transform. This adds an early return that skips the expensive clone + rebuild when `node.expressions()` is empty, which is common for Extension nodes like view matching candidates (OneOf) that carry multiple children but no expressions. Benchmark results (criterion, `datafusion-expr` micro-benchmark): | Children | no_expr (optimized) | with_expr (rebuild) | Speedup | |----------|--------------------|--------------------|---------| | 1 | 24 ns | 167 ns | 7x | | 3 | 23 ns | 192 ns | 8x | | 5 | 23 ns | 181 ns | 8x | | 10 | 24 ns | 216 ns | 9x | The `no_expr` path is constant time regardless of children count, while `with_expr` scales with the number of children (clone cost). In a real optimizer pipeline with ~15 rules traversing a plan with OneOf(5 candidates), this saves ~2.4 us per optimization pass.
90d4f6e to
f24a39a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Closes #21700.
Rationale for this change
When an Extension node has no expressions,
map_expressionswas still cloning all inputs and callingwith_exprs_and_inputsto reconstruct the node — wasted work since there are no expressions to transform. This is common for Extension nodes like view matching candidates that carry multiple children but no expressions.What changes are included in this PR?
Code change (
datafusion/expr/src/logical_plan/tree_node.rs): Add early return whennode.expressions()is empty, skipping the clone + rebuild path.Micro-benchmark (
datafusion/expr/benches/map_expressions.rs): Criterion benchmark comparingmap_expressionson Extension nodes with and without expressions, varying the number of children (1, 3, 5, 10).Are these changes tested?
Yes — existing tests pass, and the new benchmark validates the optimization.
Benchmark results:
The
no_exprpath is constant time regardless of children count.In a real optimizer pipeline (~15 rules × 5-child Extension), this saves ~2.4 us per optimization pass.
Next step
A more general optimization would be to change
UserDefinedLogicalNode::expressions()to return references (&[Expr]) instead of clonedVec<Expr>, and only clone + rebuild when the transform actually modifies an expression. This would avoid the clone +with_exprs_and_inputsrebuild even for non-empty expression lists when the transform is a no-op. Added a TODO comment in the code for this direction. This would be a larger API change, so the empty-expressions shortcut is a pragmatic first step.Are there any user-facing changes?
No — purely internal optimization. No API changes.