branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary by u70b3 · Pull Request #67978 · apache/doris

u70b3 · 2026-09-15T01:18:20Z

What problem does this PR solve?

Issue Number: #66497

Related PR: #67630 (merged — this change was stacked on it; the diff is exactly the dispatcher change described below), #67754 (open — independent; shares only the job manager queries and the config block tail)

Problem Summary:

The durable Lance index job records created by the admission path (#67630) currently stay PENDING forever: nothing drives them toward a terminal state and no execution channel exists. This change adds the master-side driver and the dispatch-boundary contracts so an admitted job can be dispatched at most once, classified into its terminal state, and released — still entirely behind the disabled-by-default enable_lance_index_mutation gate.

What lands

Thrift dispatch boundary (contract only): TLanceIndexJobDispatch / TLanceIndexJobReport plus submit_lance_index_job and reportLanceIndexJobResult, four new enums with explicit value numbering and explicit struct field ids. Storage credentials are resolved from the current catalog properties at send time and travel only inside the dispatch request — never journaled, logged, or persisted. No backend consumption lands here (the isolated worker is a later release); the backend handler stub answers a clean not-implemented error, so a cluster without workers resolves a dispatched job as NOT_COMMITTED instead of UNKNOWN.
LanceIndexJobDispatcher (master-only MasterDaemon, one round in fixed order):
- deadline sweep: an expired RUNNING job converges to UNKNOWN through completeWithResult(NO_TRUSTED_RESULT); the possible-live slot, the same-name fence, and the unresolved-job quota all stay held (a deadline bounds the wait, never proves termination);
- possible-live sweep: a slot is released only when the recorded backend process epoch no longer exists; a missing backend entry or heartbeat loss proves nothing and keeps the slot held;
- refresh driver: terminal jobs with refresh REQUIRED/FAILED resume through the existing idempotent external-table refresh path; FAILED retries are throttled to one attempt per retry interval while a first REQUIRED refresh is never delayed; DONE releases the fence and quota through the existing apply-to-memory accounting;
- PENDING dispatch: durable RUNNING (invocation id, backend process epoch, deadline) is journaled before any network I/O, and the single send carries exactly the journaled identity — the backend process epoch is captured once, so a heartbeat landing between the journal and the send cannot split it; leadership/revision/invocation-id are rechecked immediately before the send; per-round and per-backend in-flight caps bound the dispatch rate; local-file datasets additionally require the separate operator assertion and a single-FE, single-alive-backend topology.
Callback handler: a typed result envelope is applied through completeWithResult (the identity check and the result classification live in the job manager from branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235); an envelope carrying a child-reap proof releases the possible-live slot; stale or malformed envelopes are logged and dropped. The refresh a terminal job still owes is driven from the daemon rather than this RPC thread, so a driver lost mid-refresh is recovered by the master-transfer downgrade.
Config: dispatch interval, execute deadline, per-round dispatch cap, per-backend in-flight cap, refresh retry interval, and the local-file mutation assertion — all mutable, master-only. The five numeric options are positive-validated through the existing validator, and the positive invariant is re-asserted at consumption, so a handcrafted fe.conf cannot kill or busy-spin the daemon thread.
Job manager: three read-only sweep queries plus a force-release filter on the refresh query. No new edit-log operation (the job upsert is reused), no new write path, no lifecycle change.

Design interpretations (please confirm)

Dedicated thrift methods instead of extending TTaskType/the agent-task queue: this avoids wire-enum drift across branches and the tablet-oriented semantics (and resend culture) of that track; FE-side single-send plus the manager compare-and-set already provide at-most-once.
PENDING retry is unbounded: there is no dispatch-exhaustion terminal state; a pre-send failure (no backend selectable, caps reached, local-file assertion off) keeps the job PENDING and retries on the next round.
Send failure classification: a transport exception after the send may have occurred converges the job to UNKNOWN immediately; only a clean error status — provably not enqueued — resolves NOT_COMMITTED as a resource rejection.
The dispatcher does not read the mutation gate: the gate guards admission; a job that is already durable must be driven to its terminal state regardless of the gate's current value.
Local file:// handling enforces the topology rejection at dispatch only; the pre/post-invocation version and identity checks belong to the worker slice.
Head-of-line blocking is accepted: dispatches are attempted in job-id order under the per-round cap, so a permanently undispatchable low-id PENDING job (say, a local-file dataset with the operator assertion off) heads every round and blocks the dispatch of later jobs. It stays visible through the job inspection SQL from branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630 and releasable through the force-release path (branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC #67754); the job-id order is what keeps each round deterministic.

Concurrency

The dispatcher holds no catalog or manager lock across any call; every durable transition goes through the job manager under its own lock, and the catalog → job-manager lock order is unchanged. The report handler runs on the RPC thread and performs no I/O beyond the manager's edit-log write. The daemon early-exits on lost mastership and on the checkpoint thread, and journal volume stays bounded: an idle round writes no record, a throttled stuck refresh costs at most two records per retry interval, and a dispatch round writes at most the per-round cap.

Validation

Focused FE run (-Dtest='Lance*'): 515 tests, 0 failures / 0 errors / 0 skipped (fe-core 504 + fe-common 11; 81 tests are new — dispatcher 33, refresh driver 10, report handler 16, frontend shim 3, thrift contract 8, manager queries +5, wiring +2, config validator +4 — alongside the existing suites), with checkstyle.
Full FE UT (clean build, fe-common + fe-core): fe-core 9612 tests; all 21 failing methods across 14 classes are the known environmental set (timezone/network/native library), identical class-by-class to the merge base — including DiskReblanceWhenSchedulerIdle, re-verified to fail with the same assertion on the merge base itself. Zero new failures.
Regression suite test_lance_index_dispatch.groovy: negative/static only (config smoke, dispatcher-inert assertions); it needs the external docker fixture and runs in the pipeline, not locally. test_lance_index_admission.groovy was made deterministic by pinning the dispatch interval for its window, because admitted jobs are now actually dispatched once the gate is open.
Backend compiles with the not-implemented stub; no backend behavior change.

Deferred

Backend consumption of the boundary: supervisor, hard-isolated worker process, and the lance-c invocation (a later release).
The mutation gate stays closed; opening it requires the verified worker memory/PID/runtime boundary.
End-to-end dispatch evidence and real UNKNOWN production (fault-injection UT covers the lifecycle contracts; the end-to-end evidence belongs to the worker slice).
Master-side IDL sync PR, if needed, lands with the worker slice.

Release note

None — fully gated behind the disabled-by-default enable_lance_index_mutation; no user-visible behavior change.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-09-15T01:18:24Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

u70b3 · 2026-09-16T01:49:57Z

run buildall

hello-stephen · 2026-09-16T02:43:03Z

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (1826/2493)
Line Coverage	61.08% (32833/53753)
Region Coverage	48.69% (31107/63884)
Branch Coverage	51.62% (9653/18700)

u70b3 · 2026-09-16T07:27:02Z

run buildall

hello-stephen · 2026-09-16T08:51:42Z

FE UT Coverage Report

Increment line coverage 78.00% (904/1159) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-16T13:12:12Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.49% (31474/42252)
Line Coverage	58.79% (351185/597382)
Region Coverage	55.53% (293222/528003)
Branch Coverage	56.38% (132334/234705)

hello-stephen · 2026-09-16T13:43:36Z

FE Regression Coverage Report

Increment line coverage 46.63% (546/1171) 🎉
Increment coverage report
Complete coverage report

… jobs

…matrix Two fault-matrix suites over the dispatcher's two test seams (the manager's edit-log capture and the sendExecuteRequest override; zero real RPC, no utframe cluster): LanceIndexJobDispatcherTest pins the five-phase round order in one round, journal-before-send as the direct durable-before-send evidence, the no-second-dispatch discipline (CAS loss skips the send and retries with a fresh identity next round; pre-send recheck failure never sends), the send outcomes (clean error -> NOT_COMMITTED, transport failure or preparation failure -> UNKNOWN with the possible-live slot retained), per-round and per-backend backpressure, no-backend and corrupt-record deferrals, the deadline sweep (expired only, callback-first warns), the epoch sweep (epoch change releases the slot, also for UNKNOWN jobs; missing entry, same epoch, and heartbeat loss never do), the local-filesystem topology gate in both directions, per-round mutable interval reload, the idle round's zero journal records, and storage options reaching the wire while never appearing in any durable record. LanceIndexJobRefreshDriverTest pins markRefreshRunning -> handleRefreshTable(catalog, db, table, ignoreIfNotExists=true) -> DONE with exact order and arguments and the fence/quota release, DdlException keeping the fence for a retried attempt, the FAILED-only retry throttle (fresh skipped, stale retried; a first REQUIRED refresh is never delayed), the silent half-orphan completion, the force-release exclusion, the master-transfer downgrade convergence, and the fail-closed missing-catalog path.

…iring

…ption Capture the backend process epoch once so the journaled RUNNING record and the wire request always carry the same dispatch identity; a heartbeat landing between the two reads must not split it. Re-assert the positive invariant at consumption for the dispatcher configs (fe.conf bypasses the validator): clamp the dispatch interval, execute deadline, and the two dispatch caps so a handcrafted fe.conf cannot kill or busy-spin the daemon thread, sweep fresh dispatches UNKNOWN, or stall dispatch. The refresh retry interval stays unclamped: a non-positive value only disengages the throttle. Also pin the thrift struct field ids in the contract test (a symmetric round-trip cannot catch renumbering), cover the leadership-loss pre-send recheck and the checkpoint-thread skip, and fix the camelCase method name in two IDL comments.

u70b3 · 2026-09-16T15:22:18Z

run buildall

hello-stephen · 2026-09-16T17:39:23Z

FE UT Coverage Report

Increment line coverage 77.35% (239/309) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-09-16T21:55:56Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.46% (31462/42251)
Line Coverage	58.73% (350855/597396)
Region Coverage	55.43% (292675/528009)
Branch Coverage	56.31% (132166/234711)

hello-stephen · 2026-09-16T22:10:57Z

FE Regression Coverage Report

Increment line coverage 37.86% (117/309) 🎉
Increment coverage report
Complete coverage report

u70b3 mentioned this pull request Sep 15, 2026

[Feature](lance) Add vector and scalar index lifecycle management #66497

Open

12 tasks

u70b3 force-pushed the pr3d-lance-index-dispatcher branch from 1342236 to 0ad75ac Compare September 16, 2026 01:49

u70b3 force-pushed the pr3d-lance-index-dispatcher branch from 0ad75ac to f07819b Compare September 16, 2026 07:26

u70b3 added 9 commits September 16, 2026 15:11

[feature](lance) Add thrift dispatch boundary for Lance index jobs

e222f35

[feature](lance) Add dispatch sweep queries and dispatcher configs

b966152

[feature](lance) Add Lance index job dispatcher with result callback

e4c184a

[feature](lance) Drive authoritative refresh for terminal Lance index…

6eb83cb

… jobs

[test](lance) Cover Lance index report handler, thrift contract and w…

7e56542

…iring

[test](lance) Cover Lance index dispatcher regression negatives

1c53cb1

[fix](lance) Converge refresh transitions and order dispatch queries

bc04118

u70b3 force-pushed the pr3d-lance-index-dispatcher branch from f07819b to 6269177 Compare September 16, 2026 15:22

u70b3 marked this pull request as ready for review September 16, 2026 15:22

u70b3 requested a review from yiguolei as a code owner September 16, 2026 15:22

Navigation Menu

Uh oh!

branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary - #67978

branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary#67978
u70b3 wants to merge 9 commits into
apache:branch-4.1from
u70b3:pr3d-lance-index-dispatcher

u70b3 commented Sep 15, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Sep 15, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

u70b3 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What lands

Design interpretations (please confirm)

Concurrency

Validation

Deferred

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 15, 2026

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

Cloud UT Coverage Report

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 16, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 16, 2026

FE Regression Coverage Report

Uh oh!

u70b3 commented Sep 16, 2026

Uh oh!

hello-stephen commented Sep 16, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Sep 16, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 16, 2026

FE Regression Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

u70b3 commented Sep 15, 2026 •

edited

Loading