[improvement](lance) Update lance-c to merged upstream main

[Gabriel39]

What problem does this PR solve?

Related PR: #68687

Pin lance-c to merged upstream main commit cd63420bfbe27f6f0a1edcc873b9191af7d52852, including the distance-bounded search (#90), independent batch nearest search (#92), and boolean scalar-segment predicates (#93) changes. Update the archive checksum and refresh the existing Foyer patch against that base with an unchanged implementation payload.

Foyer PR lance-format/lance-c#73 is still open. Its patch must remain temporarily because Doris already calls its cache APIs; it can be removed after upstream merge.

This PR contains the dependency pin/Foyer refresh and its build integration: build.sh, thirdparty/build-thirdparty.sh, thirdparty/lance-install.sh, and the installation regression harness. The build checks a source/archive/patch fingerprint before reusing Lance, invalidates it during publication, and rejects stale external build output. Before removing an existing installation, it verifies required rebuild inputs and rejects external definitions that differ from the checkout. The third-party CI script job runs the installation harness under the existing workflow path filters. FE predicate changes and SQL fixtures/tests from #68687 are not included.

Validation

• Verified the downloaded archive checksum and exact patch application.
• Third-party extraction, cached-source reuse/refresh, and invalid-patch rejection tests passed on master.
• The updated dependency plus the unchanged Foyer patch passed 75 Rust unit tests, 376 C API tests, and all 3 native C/C++/static OSS consumer tests.
• The same dependency passed 27 FE-generated Substrait integration scenarios as part of [improvement](lance) Push down array membership and boolean scalar index predicates #68687 validation.
• The installation harness passed on master and branch-4.1 using independent external source definitions: legacy/matching installs, missing helper/vars/patch/downloader/builder, mismatched and synchronized pin/patch updates, incomplete artifacts, stale builder output, interrupted publication, and retry. Rejected preflight cases verify that the entire installation and builder invocation count remain unchanged. The missing-helper regression failed before the fix. Shell syntax, workflow YAML, and CI path-filter checks passed.
• Full Doris compilation and BE UT for this revision are pending CI.

Release note

Update the Lance C dependency to merged upstream search and scalar-predicate improvements.

Check List (For Author)

• Test: Dependency unit and integration tests described above.
• Behavior changed: Yes; incorporates upstream lance-c fixes and additive APIs.
• Does this need documentation: No new Doris user interface in this dependency-only update.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Review complete for head 5f8c8cbcc6d17c7c08785275a93bb4939e541bd8: no new actionable findings. No existing inline P0/P1 comments require carrying forward. The review covered both changed files, the pinned upstream delta, the Foyer patch and its download/build path. There were no additional user focus points.

Critical checkpoints (code-review skill Part 1.3):

1. Goal and proof: The archive name, source root and MD5 match the new pin; all 40 Foyer patch preimage hunks match the new upstream tree exactly once. Existing dependency and C/C++ tests cover the intended APIs, but I did not execute them.
2. Scope: The two-file change is focused on the Lance pin and Foyer patch context. The Foyer implementation payload remains unchanged.
3. Concurrency and locks: No Doris concurrency entry point changes. The existing Foyer wrapper's shared cache, origin namespace mutex and atomic counters were traced; no new lock-order or race issue was substantiated.
4. Lifecycle and static initialization: Session, dataset, restored-handle, wrapper and cache ownership paths remain compatible. No new cross-translation-unit initializer was added.
5. Configuration: No Doris configuration item was added or changed.
6. Compatibility: The upstream C/C++ additions are additive; no current Doris FE/BE/Cloud query code calls the Lance dataset/scanner API. The old and new pins have identical upstream Cargo.toml and Cargo.lock blobs, so nested Lance/object_store/opendal versions do not change. No Doris protocol or storage-format migration is involved.
7. Parallel paths: Fresh and cached extraction, stale patch markers, strict patch failure, whole-object and range cache reads, and C/C++ API surfaces were reviewed. No missed corresponding path was found.
8. Conditional checks: Strict zero-fuzz patching, cache-safe request selection, scanner mode limits and scalar predicate fallback were inspected; no incorrect branch was substantiated.
9. Test coverage: Existing third-party extraction/patch checks and upstream Rust, C and C++ tests cover the new search modes and retained Foyer behavior. This review is static only; no build or test was run here. The author reports separate dependency and integration results, which were not independently reproduced.
10. Test results: No Doris regression .out file changed. Embedded upstream test context was adjusted for the new base; I make no claim about unrun test results.
11. Observability: Existing cache statistics and upstream scan metrics remain available; this pin introduces no new Doris operation needing separate logs or metrics.
12. Transactions and persistence: No Doris transaction, EditLog or visible-version path changes. The existing Foyer disk cache identity and recovery paths were checked.
13. Data writes and crashes: No new Doris write path is introduced. Patch markers are written only after successful application; cache behavior remains scoped to immutable Lance data filenames.
14. FE/BE state propagation: No new variable or wire field is passed between FE and BE.
15. Performance and other risks: The new upstream scan controls and prefilter path were reviewed with their bounds and tests. No current Doris caller or substantiated performance regression was found.

Disposition: All three review passes returned NO_NEW_VALUABLE_FINDINGS in round 1, including the separate Foyer risk pass. The final changed-file and unresolved-candidate sweep found no remaining issue. This is a completed static review; full Doris compile and BE UT were not performed by this reviewer.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Review complete for head c2858482cf4c35d67df6e8f6b9eac3f13d6dfd33: three P2 inline findings. No existing P0/P1 inline finding needs to be carried forward. The PR pins Lance C to the newer upstream commit, rebases the retained Foyer patch, and adds an install fingerprint. The exact archive MD5 matches vars.sh, and the installer's strict zero-fuzz patch command passes a dry run. The findings concern external rebuild preflight and the new regression harness. There were no additional user focus points.

Critical checkpoints (code-review skill Part 1.3):

1. Goal and proof: The pin, archive checksum, strict patch application, and ordinary fingerprint comparison agree. The external-tree failure and test gaps are detailed inline; the new harness alone does not prove a full Lance build.
2. Scope and clarity: The six changed files stay within the dependency pin, patch refresh, install gate, and its test. The three suggested fixes are localized to that flow.
3. Concurrency and locks: No Doris thread entry, shared runtime state, or lock was changed. The shell publication sequence was traced; no separate supported concurrent-build race was substantiated.
4. Lifecycle: The builder removes the marker before replacing headers/archive and publishes it last, so an interrupted publication is not certified as current. The external-tree preflight can delete an existing prefix before discovering missing rebuild inputs (inline finding).
5. Configuration: No new Doris dynamic configuration item was added. DORIS_THIRDPARTY remains an existing local or external source selection.
6. Compatibility: The rebased C/C++ API changes are additive, with the patched Rust handle constructors accounted for. No Doris wire or storage-format migration is introduced. Older installed-only compilation images require refreshed dependencies and exit before deletion when no builder exists.
7. Parallel paths: Local and external trees, installed-only images, Linux/Darwin sentinels, fresh/cached source extraction, and direct/full third-party build paths were checked. The incomplete external-tree path is the outstanding defect.
8. Conditional checks: The stamp predicate requires nonempty archive, both headers, and a matching source/archive/patch fingerprint. Checking only for a builder script before deleting installed is insufficient for an external tree.
9. Test coverage: The new shell harness covers legacy, stale, incomplete, and interrupted publication cases with stubs. It is neither invoked by CI nor executable as a direct workflow command, and its external builder stub stamps checkout definitions instead of external definitions (two inline findings). The separate archive patch test covers patch extraction when supplied the archive.
10. Test results: No regression .out file changed. Reviewers confirmed the archive checksum and a strict patch dry run; one reviewer ran the stub shell harness successfully. No Doris or Lance product build, Rust test suite, or BE unit test was run by this review. Author-reported test results and pending CI are not independent runtime proof here.
11. Observability: The gate emits a mismatch error and no new Doris runtime operation requires logging or metrics. The missing-helper failure occurs after deletion, as described inline.
12. Transactions and persistence: No Doris transaction, EditLog, visible-version, or persistent metadata path changes. The Foyer runtime cache payload is carried unchanged.
13. Writes and crash handling: Third-party installation writes are protected by marker invalidation and archive staging; the external prefix removal before source verification is the identified destructive path. No Doris data write path changed.
14. FE/BE state propagation: No FE-to-BE variable or protocol field was added.
15. Performance and remaining issues: Fingerprinting is small relative to a build; a mismatched external tree can incur a full rebuild before the final rejection. The final sweep covered all six changed files and all candidates: three accepted inline, one pre-existing Foyer concern dismissed with base-code evidence, and no unresolved suspicious point.

This is a completed review of the pinned head. Full third-party and Doris CI builds were still pending at the last check; this review does not claim their results.

[github-actions]

[P2] Validate external rebuild inputs before deleting installed. When DORIS_THIRDPARTY points to a partially updated source tree containing this builder but missing lance-install.sh, the new fingerprint check enters this branch, the existing builder-presence check passes, and rm -rf installed runs. The builder then exits at its unconditional source of the missing helper, leaving the entire prefix erased. Check the required external helper and matching definitions before deletion, or stage the replacement; cover this case in the harness.

[github-actions]

[P2] Run the Lance install regression harness in CI. This new script is not referenced by the third-party script_test job (or any other workflow), so that job can pass without exercising the fingerprint gate, stale external builder, or interrupted publication cases added here. The file is also mode 100644, whereas that job invokes its other shell tests directly. Add it to the job and make it executable, or invoke it with bash.

[github-actions]

[P2] Make this builder stub use its external source definitions. The stub stamps ${DORIS_HOME}/thirdparty, while the real external builder sources its own helper and stamps ${TP_DIR}. Since this fixture gives ${DORIS_THIRDPARTY} only a builder script, its pin and patch mutation cases pass even when an actual external tree cannot produce the requested revision. Populate the external vars, patch, and helper; compute the stamp from that tree; and add an incomplete-source case.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Complete static review — 717a851

NO_NEW_VALUABLE_FINDINGS. I reviewed all seven changed files, the Lance download/patch/build/link chain, the installation gate and harness, and the three existing inline threads. The change achieves its stated dependency-pin and guarded-reuse goal at the code level: the pinned archive checksum matches, the Foyer patch's 13 existing-file preimages match the pinned upstream tree, and the builder publishes the header/archive pair before stamping its fingerprint. The gate checks that fingerprint before reuse and after an external rebuild.

Critical checkpoints:

• Goal, focus, scope: The pin, Foyer refresh, build gate, and CI harness are focused on the dependency update. There were no additional user focus points. The shell harness covers matching and stale installs, missing external inputs, fingerprint changes, incomplete artifacts, stale builder output, and interrupted publication.
• Concurrency and lifecycle: This adds no runtime threads or locks. In the serial build path, marker removal precedes publication and a new marker follows both header and archive publication; interrupted installs fail the next gate. Independent builders sharing one install prefix are not serialized by this change.
• Configuration and compatibility: No dynamic configuration, FE/BE variable, protocol, persistent format, or transaction state changes. The new upstream wrapper interfaces remain compatible with the retained Foyer patch by static inspection. The downloader verifies the archive MD5 and applies the patch strictly before recording its checksum marker.
• Parallel paths and conditions: I traced local and external third-party trees, Linux and Darwin sentinels, clean builds, cached source refresh, preflight checks, and post-build rejection. The five required external inputs and matching source/MD5/patch fingerprints are checked before deleting an old prefix.
• Tests and results: The new harness is invoked by the third-party script job for thirdparty/** and build.sh changes; that CI job passed on this head. Full Linux and macOS third-party build jobs were still in progress at review time. This review ran no builds or tests, so runtime validation is not claimed. No generated test-result files were changed.
• Data, errors, observability, performance: No Doris query data, visibility versions, memory ownership, or EditLog paths change. Filesystem publication errors fail the build; gate errors identify missing or mismatched inputs. The fingerprint check is outside runtime hot paths, and no additional metrics are needed for this build flow.

Existing P2 threads 4151729478, 4151729484, and 4151729488 cover the earlier external-input and harness concerns; the exact reported missing-input, CI-routing, and external-stub cases are addressed on this head. A stale external builder can still fail after an old prefix is removed, but that preservation concern substantially overlaps thread 4151729478, and the post-build gate rejects its output. I am not reposting any of those concerns. There are no existing P0/P1 comments to carry forward and no new inline comments. The required full-review and focused reviews converged after one round, followed by a final changed-file and unresolved-candidate sweep.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Complete review of PR #68689 at head 3dcc69710f8bcda8cb99b83d382014c1437af0e6. I inspected all seven changed paths and the related download, source-patch, build, install, reuse, and CI paths. Two full-review rounds and separate risk scans converged with three new P2 findings below.

Critical checkpoints

• Goal and proof: The new Lance pin's archive checksum matches vars.sh, and the refreshed Foyer patch applies to that archive with the production zero-fuzz patch command. The revision gate handles matching and visibly incomplete installs, but M1 and M2 show external-tree states for which its result is not reliable. No full Doris build or runtime validation was performed in this review; author-reported validation was not treated as independently run.
• Scope and clarity: The changes are focused on the dependency pin, its build gate, and its shell harness. The fingerprint and publication conditions need the two corrections below.
• Concurrency: This change adds no Doris runtime threads, shared variables, or locks. The third-party package loop runs sequentially; no new lock-order or static-initialization issue was identified.
• Lifecycle: The marker is removed before header/archive publication and written after both, so the inspected interrupted-copy path forces a rebuild. M2 can still publish a matching marker for unpatched source.
• Configuration and compatibility: No new runtime configuration, FE-to-BE variable, storage format, or wire protocol is introduced. The external source-tree path and additive Lance cache API depend on using the intended patch; M1/M2 expose compatibility gaps in that path. Direct third-party builds, cached extraction, and compilation-image rebuilds were reviewed.
• Conditions and errors: Missing external files and differing checked definitions fail before deletion, with actionable error text. The archive locator and downloader implementation remain unchecked; these are M1/M2. A false current marker in M2 also weakens observability.
• Tests and results: The new harness exercises legacy/matching installs, missing inputs, stale builder output, incomplete artifacts, and interrupted publication; it lacks the mixed-input and stale-downloader cases. M3 leaves build.sh-only changes outside its CI trigger. No regression result files were changed. Review-only validation included the archive checksum and patch dry run; no Doris compilation, BE/FE unit test, or cluster test was run here.
• Transactions, writes, and performance: No Doris transaction, EditLog, persisted data, or runtime write path changes. The installation prefix is rewritten during rebuild, as considered in M1/M2. Fingerprint calculation is build-time work; no separate runtime hot-path concern was found.

The three earlier P2 inline threads address scenarios now fixed on this head: missing external helper, no CI invocation, and a builder fixture using checkout definitions. The findings below concern distinct remaining paths. No previously reported P0/P1 finding still applies, so existing_blocking_comment_ids is empty. No additional user review focus was supplied. The changed-file and unresolved-candidate sweep is complete; review status: complete.

[hello-stephen]

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.65% (35087/45778)
Line Coverage	61.83% (396341/641054)
Region Coverage	58.25% (334365/574038)
Branch Coverage	59.13% (153696/259947)

[yiguolei]

一个thridparty 为啥要有install 脚本？