Skip to content

[improvement](be) Prune rowsets by TSO before row binlog scans - #68012

Open
HappenLee wants to merge 1 commit into
apache:branch-incremental-computationfrom
HappenLee:improvement/rowset-tso-pruning
Open

HappenLee wants to merge 1 commit into
apache:branch-incremental-computationfrom
HappenLee:improvement/rowset-tso-pruning

Conversation

@HappenLee

@HappenLee HappenLee commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Problem Summary:

Bounded ROW binlog queries initialize rowset readers and open segment footers before their TSO predicates reject historical data. A narrow time window can therefore pay initialization costs proportional to the retained history.

Prune the captured read source before cloning readers or creating scanners. For query [100, 200), a rowset with inclusive commit TSO range [10, 99] can be skipped using its metadata alone.

  • Preserve rowsets with unknown TSO endpoints and those that overlap the query window, including compacted rowsets. Existing segment/row predicates handle the remaining data.
  • Preserve the captured visible-version snapshot and separately captured delete predicates.
  • Skip a tablet when only empty rowsets remain, including bootstrap rowsets with no TSO. This also prevents a fully pruned source from being recaptured during scanner initialization.
  • Add RowsetTsoPruneTime, RowsetsPrunedByTso, SegmentsPrunedByTso, and TabletsPrunedByTso to the query profile.

The filtering uses std::erase_if on the existing rowset vector and std::all_of for the empty-tablet check. It adds no storage-format or protocol changes.

Release note

Reduce reader initialization and segment footer IO for bounded ROW binlog queries.

Check List (For Author)

  • Test
    • Regression test
    • Unit Test
    • Manual test (steps and results below)
    • No need to test or manual test.

Validation:

The results below were obtained on the original master-based implementation (cb73d23c873, based on 96d0ac68e84). This PR has since been rebased onto branch-incremental-computation (af7902c2177); the single optimization commit applies without conflicts and git range-diff confirms the patch is unchanged. Builds and tests have not been rerun on the new base.

  • BE (ASAN) and FE built successfully with build.sh --be --fe -j 48.
  • All 8 tests in OlapScanOperatorTsoPruningTest and OlapScanOperatorBinlogPushDownTest passed under ASAN. Coverage includes half-open boundaries, single/no bounds, unknown metadata, overlapping compaction output, delete predicates, and empty-source EOS.
  • test_binlog_rowset_tso_pruning and test_binlog_changes_syntax passed. The new output was generated by the regression runner and then checked in a second run. Before-images, deletes, empty windows, and compaction results were verified.
  • clang-format 16, header hygiene, and clang-tidy passed.

Manual profile check: create a one-bucket UNIQUE KEY table with ROW binlog, historical values, and automatic compaction disabled; insert two rows in two separate transactions. Enable profiling and query @incr('startTimestamp'='2099-01-01 00:00:00', 'incrementType'='MIN_DELTA'). The empty result reported:

RowsetsPrunedByTso: 2
SegmentsPrunedByTso: 2
TabletsPrunedByTso: 1
NumScanners: 0
RowsRead: 0

This verifies that scanner initialization is skipped. No production latency improvement is claimed; rowset metadata synchronization and version capture still occur.

  • Behavior changed:

    • No. Query results are unchanged; profile counters are added.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@HappenLee

Copy link
Copy Markdown
Contributor Author

/review

### What problem does this PR solve?

Issue Number: None

Problem Summary: Bounded ROW binlog queries initialize readers and open segment
footers for historical rowsets before their TSO predicates can reject them.
Initialization work therefore grows with retained history even for narrow windows.

After capturing a version-consistent read source, remove rowsets whose inclusive
commit TSO range cannot intersect the query's half-open interval. For example,
rowset [10,99] is excluded from query [100,200). Preserve unknown TSO metadata,
overlapping compaction output, and separately captured delete predicates.
Skip scanners when only empty rowsets remain so an empty source is not recaptured.
Expose pruning time and rowset, segment, and tablet counts in the query profile.

### Release note

Reduce reader initialization and segment footer IO for bounded ROW binlog queries.

### Check List (For Author)

- Test: Unit Test / Regression test / Manual test
    - ASAN BE and FE build completed with build.sh.
    - 8 OlapScanOperatorTsoPruningTest and OlapScanOperatorBinlogPushDownTest tests passed.
    - test_binlog_rowset_tso_pruning and test_binlog_changes_syntax passed.
    - Generated the new regression output and reran it in comparison mode.
    - An empty-window profile pruned 2 rowsets / 2 segments and created 0 scanners.
    - clang-format 16, header hygiene, and clang-tidy passed.
- Behavior changed: No (query results are unchanged; profile counters are added)
- Does this need documentation: No
@HappenLee
HappenLee force-pushed the improvement/rowset-tso-pruning branch from cb73d23 to 93a3cec Compare September 15, 2026 09:23
@HappenLee
HappenLee changed the base branch from master to branch-incremental-computation September 15, 2026 09:24
@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/34950177419

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants