Repository navigation
feat: add synchronous index prewarm C and C++ APIs - #94
Merged
yanghua merged 1 commit intoOct 1, 2026
Merged
Conversation
4 of 12 tasks
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The thin binding preserves the SDK’s snapshot, error, and shared-session cache semantics. Local regression and C/C++ runtime tests confirm multi-segment prewarm and cache reuse after reopening handles, with correct query results and zero partition loads. No blocking issues found.
lance-gatekeeper
Bot
added
the
K-approved
yanghua
approved these changes
Oct 1, 2026
| let version = lance_dataset_version(ds); | ||
| let historical = lance_dataset_open(c_str(&uri).as_ptr(), ptr::null(), version); | ||
| assert!(!historical.is_null()); | ||
| assert_eq!(lance_dataset_prewarm_index(ds, c"key_idx".as_ptr()), 0); |
There was a problem hiding this comment.
[P2] Assert the scalar index was actually warmed
This test currently proves that the B-tree call succeeds for a multi-segment snapshot, but not that either segment was loaded into the session cache. Could we also query both segments through the same session and assert the relevant cache-hit/load statistics? That would prevent a no-op or existence-only implementation from satisfying the test.
4 of 8 tasks
Gabriel39
added a commit
to apache/doris
that referenced
this pull request
Oct 8, 2026
…ard (branch-4.1) (#68698) ### What problem does this PR solve? Related issue: #68692 (M1) Related PRs: lance-format/lance-c#94, #68687 Upgrade lance-c from `cd63420bfbe27f6f0a1edcc873b9191af7d52852` to `98468344bc9d56aa7ed4a4192e844afa62b34ba1` to include the upstream synchronous index-prewarm C/C++ APIs. Refresh the retained Foyer patch without changing its added/removed implementation payload, and verify that patch application preserves the prewarm APIs and shared-session regression. Remove the Lance-specific installation fingerprint mechanism introduced by #68687: delete `lance-install.sh` and its test harness, remove the CI harness step, and restore the existing dependency build flow. An existing installation is no longer rejected or rebuilt because of a Lance revision fingerprint; missing APIs are reported by the normal compilation or linking process. This delivers the dependency portion of M1. SQL parsing, BE RPCs, and shared-session integration remain outside this PR; the parent issue stays open. ### Validation After rebasing onto the current `branch-4.1`: - Lance archive checksum verified; the Foyer patch's added/removed payload matches the target branch. - `lance-prefilter-patch-test.sh`: passed fresh extraction, cached-source reuse, re-extraction, stale-marker refresh, invalid-patch rejection, and prewarm API/regression preservation checks. - `download-thirdparty-fallback-test.sh` and `adbc-jni-config-test.sh`: passed. - Affected shell syntax, workflow YAML parsing, and `git diff --check`: passed. - Confirmed that the removed installation scripts and fingerprint symbols have no remaining references, and the build/CI scripts match their versions before the fingerprint mechanism was introduced. No FE/BE source files change. Full Doris compilation and runtime suites were not rerun locally and remain for PR CI. ### Release note Prepare the Lance dependency for index prewarm and remove Lance-specific installation fingerprint checks. No new SQL interface is exposed. ### Check List (For Author) - Test: - [x] Existing dependency/script regression tests - [x] Manual validation described above - Behavior changed: - [x] Yes. Upgrade lance-c and restore the existing dependency build flow without Lance revision fingerprint checks. - Does this need documentation? - [x] No new SQL interface; upstream API documentation is included in the dependency. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Oct 10, 2026
…ard (branch-4.1) (#68698) ### What problem does this PR solve? Related issue: #68692 (M1) Related PRs: lance-format/lance-c#94, #68687 Upgrade lance-c from `cd63420bfbe27f6f0a1edcc873b9191af7d52852` to `98468344bc9d56aa7ed4a4192e844afa62b34ba1` to include the upstream synchronous index-prewarm C/C++ APIs. Refresh the retained Foyer patch without changing its added/removed implementation payload, and verify that patch application preserves the prewarm APIs and shared-session regression. Remove the Lance-specific installation fingerprint mechanism introduced by #68687: delete `lance-install.sh` and its test harness, remove the CI harness step, and restore the existing dependency build flow. An existing installation is no longer rejected or rebuilt because of a Lance revision fingerprint; missing APIs are reported by the normal compilation or linking process. This delivers the dependency portion of M1. SQL parsing, BE RPCs, and shared-session integration remain outside this PR; the parent issue stays open. ### Validation After rebasing onto the current `branch-4.1`: - Lance archive checksum verified; the Foyer patch's added/removed payload matches the target branch. - `lance-prefilter-patch-test.sh`: passed fresh extraction, cached-source reuse, re-extraction, stale-marker refresh, invalid-patch rejection, and prewarm API/regression preservation checks. - `download-thirdparty-fallback-test.sh` and `adbc-jni-config-test.sh`: passed. - Affected shell syntax, workflow YAML parsing, and `git diff --check`: passed. - Confirmed that the removed installation scripts and fingerprint symbols have no remaining references, and the build/CI scripts match their versions before the fingerprint mechanism was introduced. No FE/BE source files change. Full Doris compilation and runtime suites were not rerun locally and remain for PR CI. ### Release note Prepare the Lance dependency for index prewarm and remove Lance-specific installation fingerprint checks. No new SQL interface is exposed. ### Check List (For Author) - Test: - [x] Existing dependency/script regression tests - [x] Manual validation described above - Behavior changed: - [x] Yes. Upgrade lance-c and restore the existing dependency build flow without Lance revision fingerprint checks. - Does this need documentation? - [x] No new SQL interface; upstream API documentation is included in the dependency. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
C and C++ callers currently cannot invoke the Rust SDK's index prewarm operation before serving queries. Add
lance_dataset_prewarm_index(dataset, index_name)andDataset::prewarm_index(index_name)as synchronous bindings to the existing SDK method.The operation uses one dataset snapshot and its existing session, prewarms all segments of the named logical index, and preserves the dataset version. It uses the established FFI error/panic handling and rejects NULL, empty, and invalid UTF-8 arguments. The README and public header explain shared-session reuse, eviction, partial failures, snapshot scope, and the distinction between index prewarming and result-row reads. No Lance Rust or dependency changes are required.
Tests cover invalid inputs, missing indexes, multi-segment B-tree and IVF-Flat indexes, historical snapshots after index removal, repeated calls, and dataset/session handle lifetimes. An independent cold query records partition loads; queries from a newly opened handle sharing the prewarmed session return the expected nearest neighbors with zero partition loads. C and C++ runtime tests exercise the exported symbol and wrapper.
Validation:
cargo fmt -- --checkcargo check --all-targetscargo clippy --all-targets -- -D warningscargo test— 479 passed, 0 failed (3 separately executed tests ignored by default)cargo test --test compile_and_run_test -- --ignored --test-threads=1— 3 passed, 0 failedImplements the upstream binding portion of M1 in apache/doris#68692. The Doris dependency update and synchronous SQL integration remain separate follow-up work after this merges.