Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
### What problem does this PR solve?
Issue Number: None
Problem Summary: Bounded ROW binlog queries initialize readers and open segment
footers for historical rowsets before their TSO predicates can reject them.
Initialization work therefore grows with retained history even for narrow windows.
After capturing a version-consistent read source, remove rowsets whose inclusive
commit TSO range cannot intersect the query's half-open interval. For example,
rowset [10,99] is excluded from query [100,200). Preserve unknown TSO metadata,
overlapping compaction output, and separately captured delete predicates.
Skip scanners when only empty rowsets remain so an empty source is not recaptured.
Expose pruning time and rowset, segment, and tablet counts in the query profile.
### Release note
Reduce reader initialization and segment footer IO for bounded ROW binlog queries.
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- ASAN BE and FE build completed with build.sh.
- 8 OlapScanOperatorTsoPruningTest and OlapScanOperatorBinlogPushDownTest tests passed.
- test_binlog_rowset_tso_pruning and test_binlog_changes_syntax passed.
- Generated the new regression output and reran it in comparison mode.
- An empty-window profile pruned 2 rowsets / 2 segments and created 0 scanners.
- clang-format 16, header hygiene, and clang-tidy passed.
- Behavior changed: No (query results are unchanged; profile counters are added)
- Does this need documentation: No
HappenLee
force-pushed
the
improvement/rowset-tso-pruning
branch
from
September 15, 2026 09:23
cb73d23 to
93a3cec
Compare
HappenLee
requested review from
924060929,
CalvinKirs,
dataroaring,
englefly,
gavinchou,
liaoxin01,
luwei16,
morningman,
morrySnow,
mymeiyi,
starocean999,
yiguolei and
yujun777
as code owners
September 15, 2026 09:23
HappenLee
changed the base branch from
master
to
branch-incremental-computation
September 15, 2026 09:24
Contributor
|
Codex automated review failed and did not complete. Error: Codex completed, but no new pull request review was submitted for the current head SHA. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Problem Summary:
Bounded ROW binlog queries initialize rowset readers and open segment footers before their TSO predicates reject historical data. A narrow time window can therefore pay initialization costs proportional to the retained history.
Prune the captured read source before cloning readers or creating scanners. For query
[100, 200), a rowset with inclusive commit TSO range[10, 99]can be skipped using its metadata alone.RowsetTsoPruneTime,RowsetsPrunedByTso,SegmentsPrunedByTso, andTabletsPrunedByTsoto the query profile.The filtering uses
std::erase_ifon the existing rowset vector andstd::all_offor the empty-tablet check. It adds no storage-format or protocol changes.Release note
Reduce reader initialization and segment footer IO for bounded ROW binlog queries.
Check List (For Author)
Validation:
The results below were obtained on the original master-based implementation (
cb73d23c873, based on96d0ac68e84). This PR has since been rebased ontobranch-incremental-computation(af7902c2177); the single optimization commit applies without conflicts andgit range-diffconfirms the patch is unchanged. Builds and tests have not been rerun on the new base.build.sh --be --fe -j 48.OlapScanOperatorTsoPruningTestandOlapScanOperatorBinlogPushDownTestpassed under ASAN. Coverage includes half-open boundaries, single/no bounds, unknown metadata, overlapping compaction output, delete predicates, and empty-source EOS.test_binlog_rowset_tso_pruningandtest_binlog_changes_syntaxpassed. The new output was generated by the regression runner and then checked in a second run. Before-images, deletes, empty windows, and compaction results were verified.Manual profile check: create a one-bucket UNIQUE KEY table with ROW binlog, historical values, and automatic compaction disabled; insert two rows in two separate transactions. Enable profiling and query
@incr('startTimestamp'='2099-01-01 00:00:00', 'incrementType'='MIN_DELTA'). The empty result reported:This verifies that scanner initialization is skipped. No production latency improvement is claimed; rowset metadata synchronization and version capture still occur.
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)