Skip to content

[fix](be) Refresh V2 file predicates for each split#65498

Open
Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:codex/fix-v2-late-runtime-filter-refresh
Open

[fix](be) Refresh V2 file predicates for each split#65498
Gabriel39 wants to merge 1 commit into
apache:masterfrom
Gabriel39:codex/fix-v2-late-runtime-filter-refresh

Conversation

@Gabriel39

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: FileScannerV2 cloned its table-level conjuncts only when TableReader was
initialized. Runtime filters arriving after scanner open were visible to the scanner but never
reached subsequent split readers. Query results remained correct because scanner-level evaluation
was still applied, but Parquet and ORC metadata pruning, native file filtering, and JNI block
filtering could not benefit from late runtime filters.

This change passes a freshly cloned conjunct snapshot into every split regardless of the partition
pruning setting. TableReader replaces its initial snapshot during split preparation, and JNI
readers prepare and open the fresh snapshot per split. Hybrid readers forward the same split
options to their active native or JNI child reader. V1 behavior is unchanged.

Release note

Enable late runtime filters to participate in FileScannerV2 file-level filtering for subsequent
splits.

Check List (For Author)

  • Test: Unit Test
    • TableReaderTest.PrepareSplitReplacesInitialConjunctSnapshot
    • FileScannerV2Test.RewriteSlotRefsToGlobalIndexMatrix
  • Behavior changed: Yes. Subsequent FileScannerV2 splits use the latest runtime-filter snapshot for
    native and JNI filtering.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: FileScannerV2 cloned its table-level conjuncts only when TableReader was
initialized. Runtime filters arriving after scanner open were visible to the scanner but never
reached subsequent split readers, so Parquet and ORC metadata pruning, native file filtering, and
JNI block filtering could not benefit from late runtime filters. Pass a freshly cloned conjunct
snapshot into every split, replace TableReader's initial snapshot during split preparation, and
prepare JNI conjuncts per split.

### Release note

Enable late runtime filters to participate in FileScannerV2 file-level filtering for subsequent
splits.

### Check List (For Author)

- Test: Unit Test
    - `TableReaderTest.PrepareSplitReplacesInitialConjunctSnapshot`
    - `FileScannerV2Test.RewriteSlotRefsToGlobalIndexMatrix`
- Behavior changed: Yes. Subsequent FileScannerV2 splits use the latest runtime-filter snapshot for
  native and JNI filtering.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39

Copy link
Copy Markdown
Contributor Author

run buildall

@Gabriel39 Gabriel39 marked this pull request as ready for review July 12, 2026 07:17
@Gabriel39 Gabriel39 requested a review from yiguolei as a code owner July 12, 2026 07:17
@Gabriel39

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one correctness issue in the refreshed split-conjunct path. The change installs a fresh split snapshot, but table-level COUNT splits can still return synthetic rows before those filters are built or evaluated.

Critical checkpoint conclusions:

  • Goal/test: the PR mostly addresses late runtime-filter refresh for normal native/JNI split filtering and adds a focused unit test, but the table-level COUNT path remains uncovered.
  • Scope: the implementation is narrow and focused.
  • Concurrency: no new shared-state race found; late RF updates are still cloned through the existing scanner/local-state path.
  • Lifecycle: JNI per-split preparation was reviewed; preserved initial contexts remain effectively idempotent due concrete expression prepare guards and idempotent open.
  • Config/compatibility: no new config or serialization/storage compatibility issue found.
  • Parallel paths: native/JNI/hybrid forwarding is covered, but COUNT/table_level_row_count is a parallel fast path that bypasses the refreshed predicates.
  • Tests: added coverage validates request replacement, but not refreshed conjuncts combined with table_level_row_count.
  • Observability/performance/persistence: no additional issue found.
  • User focus: no additional user-provided focus points were supplied.

Subagent conclusions:

  • tests-session-config-001 was accepted as the inline comment below.
  • optimizer-rewrite-001 was dismissed after verifying expression prepare/open idempotence for preserved JNI snapshots.
  • Convergence round 1 ended with both live subagents reporting NO_NEW_VALUABLE_FINDINGS for this same ledger/comment set.

Validation note: I attempted ./run-be-ut.sh --run --filter=TableReaderTest.PrepareSplitReplacesInitialConjunctSnapshot, but this checkout failed during BE UT configure because thirdparty/installed/bin/protoc is missing, so the test binary did not run.

Status TableReader::prepare_split(const SplitReadOptions& options) {
SCOPED_TIMER(_profile.prepare_split_timer);
_current_split_pruned = false;
if (options.conjuncts.has_value()) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fresh split snapshot can still be bypassed by the table-level COUNT shortcut below. If a scan starts with no conjuncts, gets TPushAggOp::COUNT plus table_level_row_count, and then receives a late runtime filter before a later split, _conjuncts is replaced here but prepare_split() still sets _remaining_table_level_count and returns before any reader is opened. get_block() then emits synthetic rows from _read_table_level_count() without reaching _build_table_filters_from_conjuncts(), _supports_aggregate_pushdown()'s !_table_filters.empty() fallback, or row-level filter evaluation. That can make COUNT(*) ignore the late RF for these splits. Please disable the table-level row-count shortcut whenever the refreshed split conjunct snapshot is non-empty, or evaluate/build the filters before accepting the shortcut, and add a test for SplitReadOptions::conjuncts together with table_level_row_count.

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 27.27% (3/11) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.90% (23542/41373)
Line Coverage 40.57% (230209/567446)
Region Coverage 36.48% (181918/498746)
Branch Coverage 37.55% (81214/216286)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants