Disable IN clause threshold with query parameter so we can prune at broker for large IN clauses - #19280
Merged
Jackie-Jiang merged 3 commits intoAug 19, 2026
Conversation
…roker for larger IN clauses
| /// When true, the column-value and bloom-filter segment pruners always attempt to prune segments for | ||
| /// IN predicates, regardless of the number of values in the IN clause (i.e. the server-configured | ||
| /// `inpredicate.threshold` is ignored for this query). Defaults to false. | ||
| public static final String FORCE_IN_PREDICATE_PRUNING = "forceInPredicatePruning"; |
Contributor
There was a problem hiding this comment.
To be more flexible, we can consider making it IN_PREDICATE_PRUNING_THRESHOLD, and use negative number to represent always pruning. This way it becomes a query level override. We can modify the server level one to follow the same convention
| /// doesn't mean this child filter can prune the segment. | ||
| /// 4. The specific pruners decide their own applicable predicate types. | ||
| private boolean isApplicableToFilter(FilterContext filter) { | ||
| private boolean isApplicableToFilter(FilterContext filter, boolean forceInPredicatePruning) { |
Contributor
There was a problem hiding this comment.
Pass in query.getQueryOptions() as second argument to be more flexible, and also avoid option parsing for non-IN clauses
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #19280 +/- ##
============================================
+ Coverage 66.98% 67.08% +0.10%
- Complexity 1423 1424 +1
============================================
Files 3453 3459 +6
Lines 218960 219795 +835
Branches 34805 35009 +204
============================================
+ Hits 146672 147460 +788
+ Misses 60566 60550 -16
- Partials 11722 11785 +63
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
added a commit
to pinot-contrib/pinot-docs
that referenced
this pull request
Aug 19, 2026
## Summary - add `inPredicatePruningThreshold` to the query-options reference - document positive overrides and the negative always-attempt mode - clarify the server fallback and whole-segment pruning semantics ## Upstream context Follows apache/pinot#19280. ## Validation - `git diff --check` Co-authored-by: Xiang Fu <xiangfu@Xiang-mac-mtv-2.local>
Contributor
|
The follow-up documentation is available in pinot-contrib/pinot-docs#997: pinot-contrib/pinot-docs#997 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Per #19279, it would be convenient to have a query parameter that disables the IN clause pruning threshold.
Summary
Usage
Added a per-query integer query option
inPredicatePruningThreshold:A negative value means the pruners always attempt to prune segments for IN predicates, regardless of the clause size. When unset, the server-configured inpredicate.threshold (default 10) applies, preserving existing behavior.
Pruning is per-segment and all-or-nothing. The pruner either skips a whole segment (none of the IN values exist in it) or keeps it. On a single-segment table there is nothing to skip, so no pruning is observed (my local test cluster setup); in a larger cluster, segments that contain none of the IN values are pruned.
Testing
Unit tests cover:
QueryOptionsUtilsgetterColumnValueSegmentPrunerTestandBloomFilterSegmentPrunerTest, with negative and positive overridesAlso tested in my local Pinot cluster setup and ran a few queries to validate query plans and behavior (both SSE and MSE).