branch-4.1: [feature](lance) Add Lance index job dispatcher with thrift dispatch boundary

[u70b3]

Add Lance index job dispatcher with thrift dispatch boundary

This change was stacked on #67630 (admission + job inspection SQL), which has now merged — the diff is exactly the dispatcher change described below.

Problem

The durable Lance index job records created by the admission path (#67630) currently stay PENDING forever: nothing drives them toward a terminal state and no execution channel exists. This change adds the master-side driver and the dispatch-boundary contracts so an admitted job can be dispatched at most once, classified into its terminal state, and released — still entirely behind the disabled-by-default enable_lance_index_mutation gate.

What lands

• Thrift dispatch boundary (contract only): TLanceIndexJobDispatch / TLanceIndexJobReport plus submit_lance_index_job and reportLanceIndexJobResult, four new enums with explicit value numbering and explicit struct field ids. No backend consumption lands here (the isolated worker is a later release); the backend handler stub answers a clean not-implemented error, so a cluster without workers resolves a dispatched job as NOT_COMMITTED instead of UNKNOWN.
• LanceIndexJobDispatcher (master-only MasterDaemon, one round in fixed order):
  • deadline sweep: an expired RUNNING job converges to UNKNOWN through completeWithResult(NO_TRUSTED_RESULT); the possible-live slot, fence, and quota all stay held (a deadline bounds the wait, never proves termination);
  • possible-live sweep: a slot is released only when the recorded backend process epoch no longer exists; a missing backend entry or heartbeat loss proves nothing and keeps the slot held;
  • refresh driver: terminal jobs with refresh REQUIRED/FAILED resume through the existing idempotent external-table refresh path; FAILED retries are throttled to one attempt per retry interval while a first REQUIRED refresh is never delayed; DONE releases the fence and quota through the existing apply-to-memory accounting;
  • PENDING dispatch: durable RUNNING (invocation id, backend process epoch, deadline) is journaled before any network I/O, and the single send carries exactly the journaled identity — the backend process epoch is captured once, so a heartbeat landing between the journal and the send cannot split it; leadership/revision/invocation-id are rechecked immediately before the send; per-round and per-backend in-flight caps bound the dispatch rate; local-file datasets additionally require the separate operator assertion and a single-FE, single-alive-backend topology.
• Callback handler: a typed result envelope is applied through completeWithResult (the identity check and the result classification live in the job manager from branch-4.1: [feature](lance) Durable Lance index job infrastructure with fence, quota and replay #67235); an envelope carrying a child-reap proof releases the possible-live slot; stale or malformed envelopes are logged and dropped.
• Config: dispatch interval, execute deadline, per-round dispatch cap, per-backend in-flight cap, refresh retry interval, and the local-file mutation assertion — all mutable, master-only. The five numeric options are positive-validated through the existing validator, and the positive invariant is re-asserted at consumption, so a handcrafted fe.conf cannot kill or busy-spin the daemon thread.
• Job manager: three read-only sweep queries plus a force-release filter on the refresh query. No new edit-log operation (the job upsert is reused), no new write path, no lifecycle change.

Design interpretations (please confirm)

1. Dedicated thrift methods instead of extending TTaskType/the agent-task queue: this avoids wire-enum drift across branches and the tablet-oriented semantics (and resend culture) of that track; FE-side single-send plus the manager compare-and-set already provide at-most-once.
2. PENDING retry is unbounded: there is no dispatch-exhaustion terminal state; a pre-send failure (no backend selectable, caps reached, local-file assertion off) keeps the job PENDING and retries on the next round.
3. Send failure classification: a transport exception after the send may have occurred converges the job to UNKNOWN immediately; only a clean error status — provably not enqueued — resolves NOT_COMMITTED as a resource rejection.
4. The dispatcher does not read the mutation gate: the gate guards admission; a job that is already durable must be driven to its terminal state regardless of the gate's current value.
5. Local file:// handling enforces the topology rejection at dispatch only; the pre/post-invocation version and identity checks belong to the worker slice.
6. Head-of-line blocking is accepted: dispatches are attempted in job-id order under the per-round cap, so a permanently undispatchable low-id PENDING job (say, a local-file dataset with the operator assertion off) heads every round and blocks the dispatch of later jobs. It stays visible through the job inspection SQL from branch-4.1: [feature](lance) Add Lance index admission, job inspection SQL and catalog DDL guard #67630 and releasable through the force-release path (branch-4.1: [feature](lance) Add RESOLVE LANCE INDEX JOB (FORCE_RELEASE) and job retention GC #67754); the job-id order is what keeps each round deterministic.

Concurrency

The dispatcher holds no catalog or manager lock across any call; every durable transition goes through the job manager under its own lock, and the catalog → job-manager lock order is unchanged. The report handler runs on the RPC thread and performs no I/O beyond the manager's edit-log write. The daemon early-exits on lost mastership and on the checkpoint thread, and an idle round writes no journal record.

Validation

• Focused FE run (-Dtest='Lance*'): 515 tests, 0 failures / 0 errors / 0 skipped (fe-core 504 + fe-common 11; 81 tests are new — dispatcher 33, refresh driver 10, report handler 16, frontend shim 3, thrift contract 8, manager queries +5, wiring +2, config validator +4 — alongside the existing suites), with checkstyle.
• Full FE UT (clean build, fe-common + fe-core): fe-core 9612 tests; all 21 failing methods across 14 classes are the known environmental set (timezone/network/native library), identical class-by-class to the merge base — including DiskReblanceWhenSchedulerIdle, re-verified to fail with the same assertion on the merge base itself. Zero new failures.
• Regression suite test_lance_index_dispatch.groovy: negative/static only (config smoke, dispatcher-inert assertions); it needs the external docker fixture and runs in the pipeline, not locally. test_lance_index_admission.groovy was made deterministic by pinning the dispatch interval for its window, because admitted jobs are now actually dispatched once the gate is open.
• Backend compiles with the not-implemented stub; no backend behavior change.

Deferred

• Backend consumption of the boundary: supervisor, hard-isolated worker process, and the lance-c invocation (a later release).
• The mutation gate stays closed; opening it requires the verified worker memory/PID/runtime boundary.
• End-to-end dispatch evidence and real UNKNOWN production (fake-worker fault tests in UT cover the lifecycle contracts; G2/G4 e2e evidence belongs to the worker slice).
• Master-side IDL sync PR, if needed, lands with the worker slice.

Release note

None — fully gated behind the disabled-by-default enable_lance_index_mutation; no user-visible behavior change.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[u70b3]

run buildall

[hello-stephen]

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.25% (1826/2493)
Line Coverage	61.08% (32833/53753)
Region Coverage	48.69% (31107/63884)
Branch Coverage	51.62% (9653/18700)

[u70b3]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 78.00% (904/1159) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.49% (31474/42252)
Line Coverage	58.79% (351185/597382)
Region Coverage	55.53% (293222/528003)
Branch Coverage	56.38% (132334/234705)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 46.63% (546/1171) 🎉
Increment coverage report
Complete coverage report

[u70b3]

run buildall

[hello-stephen]

FE UT Coverage Report

Increment line coverage 77.35% (239/309) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100.00% (4/4) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	74.46% (31462/42251)
Line Coverage	58.73% (350855/597396)
Region Coverage	55.43% (292675/528009)
Branch Coverage	56.31% (132166/234711)

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 37.86% (117/309) 🎉
Increment coverage report
Complete coverage report