feat(index): compose FRI mapping readers with planner safety boundaries - #9068
Draft
LuQQiu wants to merge 25 commits into
Draft
feat(index): compose FRI mapping readers with planner safety boundaries#9068LuQQiu wants to merge 25 commits into
LuQQiu wants to merge 25 commits into
Conversation
jackye1995
force-pushed
the
lu/fri-query
branch
from
September 9, 2026 17:28
9f98168 to
4bf7c5f
Compare
LuQQiu
removed this pull request from stack #9066
September 9, 2026 19:15
LuQQiu
added this pull request to stack #9108
September 9, 2026 19:16
# Conflicts: # rust/lance-table/src/feature_flags.rs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add the index-version-1 FRI reader behind explicit metadata dispatch. No-FRI and index-version-0 datasets continue through the existing legacy read blocks; this PR makes no scalar/vector loader changes.
Compose mapping readers along validated fragment lineage, resolve each live fragment backward to the nearest complete source coverage, and stop address translation at current live fragments. Unknown mapping paths scan; unsupported whole index versions require an upgrade.
Keep snapshot state separate from immutable cached contents: share decoded ledgers by FRI UUID and reuse unchanged mapping readers across ledger changes using content fingerprints and storage bindings. Current live fragments remain snapshot-specific. Lazily loaded counts are included in cache weights; a TODO records the cache accuracy versus latency tradeoff without adding a new interface. New label-block caching and prewarm optimizations remain later work.
Set/preserve paired FRI feature flags and validate tagged metadata against those flags before publishing a manifest. Reject unsupported maintenance. Restore follows the existing snapshot-publication path, preserving index metadata and external references; it is not blocked merely because a FRI flag is present. Clone checks share one helper and avoid reading index metadata when the flag is absent. Compaction is rejected before entering the legacy rewrite flow. Group segments by logical index before selecting usable contributors and deriving destination coverage. Drop user segments with empty query coverage. Failure to cover one destination does not discard another destination’s coverage. The batch API uses physical
u64row IDs throughout; callers can exclude superseded intermediate fragments to prevent duplicated contributions after branches reconverge. Until #9107 installs asynchronous consumers, affected index segments scan and independent segments retain their existing loading path. Dead provenance cannot receive an identity remapper just because an unknown mapping was omitted from the graph; projected legacy coverage still requires conservative lineage checks.Validation: 31 common-reader/boundary tests and 17 existing deferred-remap regression tests pass. Tests cover concurrent cache reuse, append snapshot isolation, retained mappings after simulated pruning, changed source order/storage bindings, and cache weight growth. The historical fixture verifies legacy-reader reuse across append; pure-version-0 operations fail the test if they enter the new reader. Five transaction-level tests pin rejection of Delete, Update, CreateIndex, UpdateConfig, and MemWAL maintenance. Additional tests cover Restore, missing flags at publication, and the clone flag fast path. Formatting and workspace Clippy pass.
No format documentation changes. The core writer remains separate work.