Skip to content

[fix](runtime filter) do not publish an uninitialized (UNINITED) filter as disabled - #68002

Open
lide-reed wants to merge 1 commit into
apache:branch-4.1from
lide-reed:fix-runtimefilter-4.1
Open

lide-reed wants to merge 1 commit into
apache:branch-4.1from
lide-reed:fix-runtimefilter-4.1

Conversation

@lide-reed

Copy link
Copy Markdown
Contributor

Proposed changes

Fix #67997

Problem

RuntimeFilter::serialize() marks the outgoing filter as disabled for any state other
than READY:

auto state = _wrapper->get_state();
if (state != RuntimeFilterWrapper::State::READY) {
    request->set_disabled(true);
    return Status::OK();
}

UNINITED means "no filter content has been produced yet", not "this filter was
cancelled", but the consumer cannot distinguish the two. It treats disabled as an
absorbing state (set_state(DISABLED, "get disabled from remote")) and permanently gives
up the filter, so the probe side degrades to a full scan.

The merger decides readiness only by producer count, which is independent of whether any
producer really produced content:

// RuntimeFilterMerger::merge_from()
*ready = _received_producer_num == _expected_producer_num;
if (_received_producer_num == _expected_producer_num) {
    _rf_state = State::READY;
}
if (_wrapper->get_state() == RuntimeFilterWrapper::State::UNINITED) {
    _wrapper = other->_wrapper;   // may still be UNINITED
    return Status::OK();
}

So when all producers report without content, merge_from() returns ready == true while
the merged wrapper is still UNINITED, and the caller publishes it as disabled to every
consumer.

Fix

Align with the legacy (branch-3.1) semantics: only a really disabled filter
(max_in_num / join spill / rpc error -> State::DISABLED) is published as disabled; a
filter that is simply not ready is skipped and consumers wait until
runtime_filter_wait_time_ms.

File Change
runtime_filter.h serialize(): only DISABLED sets disabled=true; UNINITED publishes nothing (DCHECK + WARNING + OK)
runtime_filter_merger.h new is_wrapper_uninited() helper
runtime_filter_producer.cpp do_merge: skip send when merged wrapper is UNINITED
runtime_filter_mgr.cpp _send_rf_to_target(): skip broadcast when merged wrapper is UNINITED

Why UNINITED returns OK instead of InternalError

The old behavior for UNINITED was a silent performance degradation (filter disabled),
not an error. Turning it into InternalError would escalate a performance issue into a
query failure. Publishing nothing keeps the failure mode conservative: the filter is lost,
the consumer times out, and at worst the probe side falls back to a full scan. DCHECK is
kept so unexpected paths still fail fast in debug builds.

Checklist

  • Issue is created and linked
  • No new configuration or user-facing interface
  • No behavior change for correctly produced filters

Further comments

Silent performance regression is hard to notice in production; the fix restores the
previous (branch-3.1) semantics and makes the worst case a filter timeout rather than a
permanently disabled filter.

…er as disabled

## Proposed changes

Issue Number: close #<ISSUE_ID>

Problem:

RuntimeFilter::serialize() marks the outgoing filter as `disabled` for any
state other than READY, so a filter that is merely UNINITED (no filter
content produced yet) is broadcast to consumers as `disabled`. On the
consumer side, `disabled` is an absorbing state:

    RuntimeFilterWrapper::signal()
        -> set_state(DISABLED, "get disabled from remote")

so the consumer permanently gives up the filter and the probe side degrades
to a full scan, even though nothing actually disabled the filter.

The merger decides readiness only by producer count:

    RuntimeFilterMerger::merge_from()
        *ready = (_received_producer_num == _expected_producer_num)

which is independent of whether any producer really produced filter content.
When all producers report without content, the adopted wrapper is still
UNINITED while merge_from() already returns ready == true, and the caller
publishes it as disabled.

What this PR does:

1. be/src/exec/runtime_filter/runtime_filter.h
   In serialize(), only State::DISABLED sets `disabled=true`. State::UNINITED
   is left as "publish nothing": DCHECK in debug builds plus a WARNING log in
   release builds, and return OK. This keeps the worst case as "the filter is
   lost and the probe side falls back to a full scan", instead of returning
   InternalError and failing the whole query.

2. be/src/exec/runtime_filter/runtime_filter_merger.h
   Add is_wrapper_uninited() helper to tell "collected all products by count
   but there is no real filter content (nor a real DISABLED)" from a really
   disabled filter.

3. be/src/exec/runtime_filter/runtime_filter_producer.cpp
   In publish()/do_merge, skip sending (local and remote targets) when the
   merged wrapper is still UNINITED.

4. be/src/exec/runtime_filter/runtime_filter_mgr.cpp
   In RuntimeFilterMergeControllerEntity::_send_rf_to_target(), skip the
   broadcast when the merged wrapper is still UNINITED, so consumers keep
   waiting until runtime_filter_wait_time_ms and then time out normally.

This aligns with the legacy (branch-3.1) runtime filter semantics: only a
really disabled filter (reach max_in_num / join spill / rpc error) is
published as disabled; a filter that is simply not ready must not be.

## Further comments

No user-visible interface or configuration change. In the worst case a
runtime filter is dropped and the query falls back to a full scan, which is
the same behavior as a filter timeout, and strictly better than the current
behavior of permanently disabling the filter on the consumer side.
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@lide-reed lide-reed closed this Sep 15, 2026
@lide-reed lide-reed reopened this Sep 15, 2026
@lide-reed
lide-reed changed the base branch from master to branch-4.1 September 15, 2026 07:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Runtime filter is published as disabled when the merged filter is not ready (UNINITED), causing the probe side to degrade to a full scan

2 participants