[improvement](lance) Upgrade prewarm dependency and remove install guard (branch-4.1)

[Gabriel39]

What problem does this PR solve?

Related issue: #68692 (M1)
Related PRs: lance-format/lance-c#94, #68687

Upgrade lance-c from cd63420bfbe27f6f0a1edcc873b9191af7d52852 to 98468344bc9d56aa7ed4a4192e844afa62b34ba1 to include the upstream synchronous index-prewarm C/C++ APIs. Refresh the retained Foyer patch without changing its added/removed implementation payload, and verify that patch application preserves the prewarm APIs and shared-session regression.

Remove the Lance-specific installation fingerprint mechanism introduced by #68687: delete lance-install.sh and its test harness, remove the CI harness step, and restore the existing dependency build flow. An existing installation is no longer rejected or rebuilt because of a Lance revision fingerprint; missing APIs are reported by the normal compilation or linking process.

This delivers the dependency portion of M1. SQL parsing, BE RPCs, and shared-session integration remain outside this PR; the parent issue stays open.

Validation

After rebasing onto the current branch-4.1:

• Lance archive checksum verified; the Foyer patch's added/removed payload matches the target branch.
• lance-prefilter-patch-test.sh: passed fresh extraction, cached-source reuse, re-extraction, stale-marker refresh, invalid-patch rejection, and prewarm API/regression preservation checks.
• download-thirdparty-fallback-test.sh and adbc-jni-config-test.sh: passed.
• Affected shell syntax, workflow YAML parsing, and git diff --check: passed.
• Confirmed that the removed installation scripts and fingerprint symbols have no remaining references, and the build/CI scripts match their versions before the fingerprint mechanism was introduced.

No FE/BE source files change. Full Doris compilation and runtime suites were not rerun locally and remain for PR CI.

Release note

Prepare the Lance dependency for index prewarm and remove Lance-specific installation fingerprint checks. No new SQL interface is exposed.

Check List (For Author)

• Test:
  • Existing dependency/script regression tests
  • Manual validation described above
• Behavior changed:
  • Yes. Upgrade lance-c and restore the existing dependency build flow without Lance revision fingerprint checks.
• Does this need documentation?
  • No new SQL interface; upstream API documentation is included in the dependency.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Complete static review of PR #68698 at 52f8ad3. Two distinct [P2] findings are attached inline. There are no existing inline threads or still-applicable P0/P1 comments to carry forward.

Critical checkpoints:

• Goal and proof: The pinned lance-c archive matches the declared MD5, contains the upstream synchronous C/C++ prewarm API, and the rebased Foyer patch passes a static dry run using the downloader's patch flags. The existing Doris scan path opens through its shared Lance session. This is the dependency portion of the work; the SQL/RPC prewarm path is deferred. The added shell harnesses cover revision matching, stale input rejection, single-process publication, and cached patch refresh. They do not prevent the two installation failures below. Upstream test and CI claims were inspected, not independently rerun; this review performed no build or test suite.
• Scope and compatibility: The seven changed files keep the implementation focused on the dependency pin and installation gate. The C/C++ prewarm API is additive; no Doris FE/BE protocol field, runtime configuration, storage format, or SQL behavior changes here. Existing compilation images without the new marker and rebuild sources are rejected before deleting their artifacts. An external tree with a readable but stale builder is a separate path covered by the first inline finding.
• Concurrency and lifecycle: No new runtime thread, lock order, cross-translation-unit static initializer, or managed-memory allocation is introduced. The marker is removed before a single publisher replaces the Lance header/archive and is written last, so ordinary interruption leaves an invalid install. Two independent package builds can share a prefix without serialization; the second inline finding describes an old/new publication interleaving that can leave a current marker over the wrong archive. There is no lock protecting that multi-file publication.
• Parallel paths and conditions: The in-tree and external third-party paths, clean/default rebuilds, package-specific Lance build, Linux/Darwin gate, cached source extraction, and shared-session caller were inspected. The source/MD5/patch fingerprint and post-build check explain the new conditions, but they do not preserve the old prefix after a stale builder runs. A different archive URL/name alone did not establish another issue because the normal downloader validates MD5.
• Tests and results: No expected-result files changed. The new install fixture explicitly exercises an external builder that exits successfully without producing the marker; it checks failure, then uses a good builder to repair the now-deleted prefix. It does not assert preservation of the old dependencies or cover overlapping publishers. The prewarm patch harness checks symbol and regression-test presence, while the pinned upstream source supplies the implementation and tests. No end-to-end Doris prewarm test is expected for this dependency-only step.
• Error handling and observability: The mismatch paths return failure with diagnostics. No new Status, exception boundary, persistent metadata, EditLog, transaction, data write, visible-version, delete-bitmap, or FE-to-BE variable path is modified. Build diagnostics are sufficient to locate a revision mismatch, although the first finding shows that detection comes after destructive cleanup. No new runtime metric or dynamic configuration is needed for this change.
• Performance and other risks: Fingerprint computation is small relative to a third-party build; a stale marker triggers the existing full dependency rebuild. No query hot path changes. The Foyer patch preserves its implementation payload and the pinned prewarm path retains the shared session; no additional correctness issue was substantiated there.

Review convergence completed in two rounds: round one identified the two issues, and every round-two normal and risk-focused reviewer returned NO_NEW_VALUABLE_FINDINGS. The final changed-file and candidate sweep found no unresolved suspicious point. User focus: none was supplied.

[github-actions]

[P2] Preserve the installed prefix until a stale external builder is ruled out. A readable external build-thirdparty.sh with matching vars.sh and patch passes the preflight, but the script removes the whole installed tree before running it. The new stale-builder fixture supplies a builder that exits successfully without publishing a Lance marker, so this post-build check fails only after the previous dependencies are gone. Stage the rebuild and replace the prefix after verifying its output, or verify the builder's capability before deleting the existing install.

[github-actions]

[P2] Serialize publication of the archive and fingerprint. When two builds share this install prefix across an in-place Lance revision update, an older process can remove the marker, pause, then let a newer process publish its header, archive, and new marker. If the older process resumes and replaces liblance_c.a but stops before rewriting its marker, the new marker remains and lance_c_install_is_current accepts the old archive. A lock or versioned staging around the whole header/archive/marker publication is needed.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Complete static review: one new P2 inline finding. The fresh-source dependency pin is the upstream lance-c #94 merge and the retained Foyer patch changes only its base metadata and README context. The removed install guard leaves an interrupted targeted Lance install undetected (inline comment).

Critical checkpoints (Doris review skill Part 1.3):

1. Goal and proof: This PR prepares the synchronous C/C++ prewarm API in the pinned dependency; it adds no Doris SQL or BE caller. The exact upstream commit adds the C declaration, C++ wrapper, Rust binding, and tests. The new local harness checks their source presence, but this invocation did not execute it. The third-party script CI check passed at final sweep; Linux/macOS full builds and Doris compile check were still in progress.
2. Scope and parallel paths: The eight changed paths are limited to the pin, retained patch context, install gate, builder, tests, and workflow. Full Linux/macOS builds still download, patch, and compile Lance. Targeted Lance builds and reuse of external installed prefixes follow a different path; the reduced gate mishandles an incomplete targeted install.
3. Concurrency and lifecycle: No FE/BE runtime thread or lock changes occur. The build script copies Lance headers in place before atomically replacing only the archive. A single interruption can leave missing headers with the previous archive present. Concurrent publication was already raised in existing thread 4155397448; its particular fingerprint race no longer applies after removal of that marker. No cross-translation-unit static initialization or ownership change was introduced.
4. Conditions, errors, and observability: The new gate checks existence of the Lance archive but not its size or the headers. It can postpone an incomplete-install error until BE compilation instead of repairing the prefix (new P2). Build commands still fail through shell error propagation when invoked. No new runtime Status handling, memory allocation, metric, or log path was added.
5. Compatibility and configuration: The upstream #94 API is additive over the prior pin; existing BE C calls are unchanged. Complete older installed prefixes are intentionally reused by this PR, so later consumers requiring prewarm must detect a missing API at compile/link time. No configuration item, dynamic setting, FE/BE transmitted variable, or storage format changes.
6. Transactions, persistence, writes, and performance: There are no Doris transaction, EditLog, data-write, or query hot-path changes. The affected write lifecycle is the local third-party header/archive install and is covered by the inline finding. No new runtime performance concern was substantiated.
7. Tests and results: The deleted Lance install harness had missing-header and empty-archive cases; the remaining workflow does not exercise the build.sh reuse gate. The new patch assertions and upstream regression names are consistent with the pin, while their runtime behavior was not independently executed here. No test result file changed. This review performed static inspection only: no build, test, or product-source edit.

Existing review context and focus: Existing P2 thread 4155397437 still covers destructive rebuild with a stale external builder and is not reposted. Thread 4155397448 concerns the removed fingerprint publication race and is not carried as a new issue. Neither is P0/P1, so existing_blocking_comment_ids is empty. The user supplied no additional focus. Round 2 normal and risk-focused reviewers all returned NO_NEW_VALUABLE_FINDINGS; the final eight-file sweep found no unresolved candidate.