[fix](be) Refresh V2 file predicates for each split#65498
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 cloned its table-level conjuncts only when TableReader was
initialized. Runtime filters arriving after scanner open were visible to the scanner but never
reached subsequent split readers, so Parquet and ORC metadata pruning, native file filtering, and
JNI block filtering could not benefit from late runtime filters. Pass a freshly cloned conjunct
snapshot into every split, replace TableReader's initial snapshot during split preparation, and
prepare JNI conjuncts per split.
### Release note
Enable late runtime filters to participate in FileScannerV2 file-level filtering for subsequent
splits.
### Check List (For Author)
- Test: Unit Test
- `TableReaderTest.PrepareSplitReplacesInitialConjunctSnapshot`
- `FileScannerV2Test.RewriteSlotRefsToGlobalIndexMatrix`
- Behavior changed: Yes. Subsequent FileScannerV2 splits use the latest runtime-filter snapshot for
native and JNI filtering.
- Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
I found one correctness issue in the refreshed split-conjunct path. The change installs a fresh split snapshot, but table-level COUNT splits can still return synthetic rows before those filters are built or evaluated.
Critical checkpoint conclusions:
- Goal/test: the PR mostly addresses late runtime-filter refresh for normal native/JNI split filtering and adds a focused unit test, but the table-level COUNT path remains uncovered.
- Scope: the implementation is narrow and focused.
- Concurrency: no new shared-state race found; late RF updates are still cloned through the existing scanner/local-state path.
- Lifecycle: JNI per-split preparation was reviewed; preserved initial contexts remain effectively idempotent due concrete expression prepare guards and idempotent open.
- Config/compatibility: no new config or serialization/storage compatibility issue found.
- Parallel paths: native/JNI/hybrid forwarding is covered, but COUNT/table_level_row_count is a parallel fast path that bypasses the refreshed predicates.
- Tests: added coverage validates request replacement, but not refreshed conjuncts combined with table_level_row_count.
- Observability/performance/persistence: no additional issue found.
- User focus: no additional user-provided focus points were supplied.
Subagent conclusions:
- tests-session-config-001 was accepted as the inline comment below.
- optimizer-rewrite-001 was dismissed after verifying expression prepare/open idempotence for preserved JNI snapshots.
- Convergence round 1 ended with both live subagents reporting NO_NEW_VALUABLE_FINDINGS for this same ledger/comment set.
Validation note: I attempted ./run-be-ut.sh --run --filter=TableReaderTest.PrepareSplitReplacesInitialConjunctSnapshot, but this checkout failed during BE UT configure because thirdparty/installed/bin/protoc is missing, so the test binary did not run.
| Status TableReader::prepare_split(const SplitReadOptions& options) { | ||
| SCOPED_TIMER(_profile.prepare_split_timer); | ||
| _current_split_pruned = false; | ||
| if (options.conjuncts.has_value()) { |
There was a problem hiding this comment.
This fresh split snapshot can still be bypassed by the table-level COUNT shortcut below. If a scan starts with no conjuncts, gets TPushAggOp::COUNT plus table_level_row_count, and then receives a late runtime filter before a later split, _conjuncts is replaced here but prepare_split() still sets _remaining_table_level_count and returns before any reader is opened. get_block() then emits synthetic rows from _read_table_level_count() without reaching _build_table_filters_from_conjuncts(), _supports_aggregate_pushdown()'s !_table_filters.empty() fallback, or row-level filter evaluation. That can make COUNT(*) ignore the late RF for these splits. Please disable the table-level row-count shortcut whenever the refreshed split conjunct snapshot is non-empty, or evaluate/build the filters before accepting the shortcut, and add a test for SplitReadOptions::conjuncts together with table_level_row_count.
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 cloned its table-level conjuncts only when TableReader was
initialized. Runtime filters arriving after scanner open were visible to the scanner but never
reached subsequent split readers. Query results remained correct because scanner-level evaluation
was still applied, but Parquet and ORC metadata pruning, native file filtering, and JNI block
filtering could not benefit from late runtime filters.
This change passes a freshly cloned conjunct snapshot into every split regardless of the partition
pruning setting. TableReader replaces its initial snapshot during split preparation, and JNI
readers prepare and open the fresh snapshot per split. Hybrid readers forward the same split
options to their active native or JNI child reader. V1 behavior is unchanged.
Release note
Enable late runtime filters to participate in FileScannerV2 file-level filtering for subsequent
splits.
Check List (For Author)
TableReaderTest.PrepareSplitReplacesInitialConjunctSnapshotFileScannerV2Test.RewriteSlotRefsToGlobalIndexMatrixnative and JNI filtering.