Skip to content

feat(index): compose FRI mapping readers with planner safety boundaries - #9068

Draft
LuQQiu wants to merge 25 commits into
lu/fri-readerfrom
lu/fri-query
Draft

feat(index): compose FRI mapping readers with planner safety boundaries#9068
LuQQiu wants to merge 25 commits into
lu/fri-readerfrom
lu/fri-query

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Add the index-version-1 FRI reader behind explicit metadata dispatch. No-FRI and index-version-0 datasets continue through the existing legacy read blocks; this PR makes no scalar/vector loader changes.

Compose mapping readers along validated fragment lineage, resolve each live fragment backward to the nearest complete source coverage, and stop address translation at current live fragments. Unknown mapping paths scan; unsupported whole index versions require an upgrade.

Keep snapshot state separate from immutable cached contents: share decoded ledgers by FRI UUID and reuse unchanged mapping readers across ledger changes using content fingerprints and storage bindings. Current live fragments remain snapshot-specific. Lazily loaded counts are included in cache weights; a TODO records the cache accuracy versus latency tradeoff without adding a new interface. New label-block caching and prewarm optimizations remain later work.

Set/preserve paired FRI feature flags and validate tagged metadata against those flags before publishing a manifest. Reject unsupported maintenance. Restore follows the existing snapshot-publication path, preserving index metadata and external references; it is not blocked merely because a FRI flag is present. Clone checks share one helper and avoid reading index metadata when the flag is absent. Compaction is rejected before entering the legacy rewrite flow. Group segments by logical index before selecting usable contributors and deriving destination coverage. Drop user segments with empty query coverage. Failure to cover one destination does not discard another destination’s coverage. The batch API uses physical u64 row IDs throughout; callers can exclude superseded intermediate fragments to prevent duplicated contributions after branches reconverge. Until #9107 installs asynchronous consumers, affected index segments scan and independent segments retain their existing loading path. Dead provenance cannot receive an identity remapper just because an unknown mapping was omitted from the graph; projected legacy coverage still requires conservative lineage checks.

Validation: 31 common-reader/boundary tests and 17 existing deferred-remap regression tests pass. Tests cover concurrent cache reuse, append snapshot isolation, retained mappings after simulated pruning, changed source order/storage bindings, and cache weight growth. The historical fixture verifies legacy-reader reuse across append; pure-version-0 operations fail the test if they enter the new reader. Five transaction-level tests pin rejection of Delete, Update, CreateIndex, UpdateConfig, and MemWAL maintenance. Additional tests cover Restore, missing flags at publication, and the clone flag fast path. Formatting and workspace Clippy pass.

No format documentation changes. The core writer remains separate work.

@LuQQiu
LuQQiu added this pull request to stack #9066 September 9, 2026 01:11
@github-actions github-actions Bot added the enhancement New feature or request label Sep 9, 2026
@github-actions github-actions Bot added the A-index Vector index, linalg, tokenizer label Sep 9, 2026
@LuQQiu
LuQQiu removed this pull request from stack #9066 September 9, 2026 19:15
@LuQQiu LuQQiu changed the title feat(index): query tagged fragment reuse histories feat(index): compose FRI mapping readers with planner safety boundaries Sep 9, 2026
@LuQQiu
LuQQiu added this pull request to stack #9108 September 9, 2026 19:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant