Skip to content

feat(index): apply shared FRI remapping to scalar and vector queries - #9107

Open
LuQQiu wants to merge 37 commits into
lu/fri-queryfrom
lu/fri-index-consumers
Open

feat(index): apply shared FRI remapping to scalar and vector queries#9107
LuQQiu wants to merge 37 commits into
lu/fri-queryfrom
lu/fri-index-consumers

Conversation

@LuQQiu

@LuQQiu LuQQiu commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

Stack reordered: the standalone format proposal is now #9136 (replacing #9065), followed by #9106#9064#9067#9068#9107 in native stack #9137. This PR’s diff is byte-identical to its pre-reorder diff. All branches include main 31d78d170; no history was rewritten. Formatting checks pass; Clippy and tests remain blocked by the unavailable object_store_opendal 0.60.1 dependency.

Connect scalar and vector index loading to the common FRI reader from #9068. Mapping algorithms and lineage traversal remain in earlier PRs; this PR adapts index consumers and enables derived coverage only for supported consumers.

  • Share asynchronous batch remapping across B-tree, bitmap, zonemap, bloom, FTS, ngram, label-list, RTree, and vector partition loading. Preserve vector layouts and deletion handling without a partition-sized per-row lookup table.
  • Keep index-version-0 FRI on the existing synchronous reader/writer path: scalar cold loads call the original plugin API, and vector storage receives the original synchronous remapper. Only external mappings enter the new asynchronous loading branch. Unrelated FM/custom plugins retain legacy loading; unsupported affected consumers scan.
  • Retain legacy translated-index cache isolation and support the existing vector prewarm entry point. Immutable ledger/mapping reuse is in feat(index): compose FRI mapping readers with planner safety boundaries #9068. New label-block caching, row-map prewarm optimizations, and the core FRI writer are later work.

Validation:

  • A two-source/two-destination end-to-end matrix verifies every contributing segment is queried, and that missing segments, unsupported index versions, or unsupported async consumers force scan fallback without missing or duplicated rows.
  • Vector fragment-restricted search falls back to scanning when explicit segment selection omits a contributor; segment-only partial-search semantics remain unchanged. Regressions cover default, complete, and partial segment selection, including a single destination with a prefilter.
  • Queries prefer direct destination coverage and combine indexed fragments with scans for uncovered fragments. When branches reconverge, each segment excludes paths entering fragments supplied by other selected segments, preventing duplicate vector results.
  • 79 common-reader and index-consumer tests pass, including scalar/vector index-version-0 lifecycles, released legacy fixtures, inline/external history, eager/deferred compaction, append/delete/checkout, RTree, unrelated FM, unknown mappings, and maintenance boundaries.
  • Default-budget local run: cargo test -p lance frag_reuse_reader --lib -- --test-threads=1.
  • Run serially with the default memory budget. The default 150 MB process-shared DataFusion pool can exhaust sort reservations with concurrent tests; CI uses nextest process isolation.
  • Shared remapper unit tests pass (batch bounds, deletion/layout preservation, legacy delegation).
  • cargo fmt --all and workspace cargo clippy --all --tests --benches -- -D warnings pass.

No format documentation changes. Draft, last layer of the six-PR reader stack.

Latest main synchronization (2026-09-10): merged d7b031f9c through the stack without rewriting history. cargo fmt --all and whitespace checks pass. Workspace Clippy could not run: main now requires object_store_opendal 0.60.1, but the crates.io index currently resolves only up to 0.60.0. Tests were not rerun after this synchronization because dependency resolution is blocked. Earlier validation above predates this merge.

@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer enhancement New feature or request labels Sep 9, 2026
@LuQQiu
LuQQiu added this pull request to stack #9108 September 9, 2026 19:16
LuQQiu and others added 16 commits September 9, 2026 14:31
# Conflicts:
#	rust/lance/src/index/frag_reuse_reader.rs
The v0 read path keeps its original signatures and remapper type end to
end; asynchronous batch translation lives in sibling entry points used
only by tagged histories. The bridging RowIdRemapping enum is removed,
and cache scoping is owned by one helper.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
Which rows a segment owns under a tagged history was decided twice at
two times: reader planning computed coverage, and every index open
re-ran load_indices plus a whole-dataset sibling scan to reconstruct
the excluded fragments. Opening N segments repeated snapshot-wide
planning N times.

The tagged path now builds one snapshot-level plan in a single pass
over the same load_indices output: per segment UUID, the rewritten
query coverage and the fragments owned by other selected siblings,
with identity and missing-coverage outcomes recorded so per-segment
error semantics are unchanged. The plan is cached under the
manifest-path (tagged) namespace via FriQueryPlanKey and built through
get_or_insert so concurrent opens coalesce; each open then only looks
its segment up. The v0/no-FRI path never touches the plan machinery.

Also assert in debug builds when a plugin advertises batch row-ID
remapping but falls into the default load_index_with_remapping.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
…maps

Warm tagged opens still called load_indices before the plan lookup, so
every open re-ran the tagged post-processing (capability checks and
segment_coverage) even on a plan cache hit. The generation decision now
reads the cheap cached stored listing, and load_indices moves inside
the FriQueryPlanKey loader: a warm open touches only the stored
listing, the cached mapping, and the plan lookup. Unsupported FRI
versions keep the exact load_indices error.

Plan construction is also de-quadratized: one pass over the filtered
listing groups selected segments by logical index name and unions each
group's direct coverage once; each segment derives its exclusions as
the group union minus its own stored bitmap. Behavior is identical.

The new batch-remapping bitmap load path no longer unwraps
RowAddrTreeMap deserialization; malformed null-bitmap bytes now
surface a corrupt-file error naming the lookup file. The legacy V0
path is untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
@LuQQiu
LuQQiu removed this pull request from stack #9108 September 10, 2026 03:15
@LuQQiu
LuQQiu changed the base branch from lu/fri-query to main September 10, 2026 03:15
@LuQQiu
LuQQiu changed the base branch from main to lu/fri-query September 10, 2026 03:15
@LuQQiu
LuQQiu added this pull request to stack #9117 September 10, 2026 03:15
LuQQiu and others added 4 commits September 10, 2026 09:18
Resolve FTS document-state sharing and scalar cache rebind API conflicts. Keep batch remappers store-bound and reload on binding changes. Formatting passes; Clippy is blocked by main requiring object_store_opendal 0.60.1, which the registry does not currently resolve.
The FRI version picks the interface and the segment's need picks the
behavior. ResolvedRemapping now has three states: V0 keeps the compact
synchronous handle exactly as before tagged histories existed;
V1Identity carries no remapper at all, so an untouched segment on a
tagged dataset loads through every plugin's original entry point,
including plugins that only implement the legacy API; V1Translate
feeds the additive batch entry points. The hollow identity
CompactFragReuseIndexHandle is gone; cache identity is carried by the
resolver, and the cache-scoping rule is uniformly "tagged history"
(V1Identity | V1Translate).

Tests pin the resolver dispatch: no FRI resolves to nothing, v0 to the
compact handle, an untouched segment to identity, a rewritten segment
to translation, and an uncommitted UUID to the existing error. An
FM-index segment (legacy-API-only plugin) stays available when
untouched and serves queries, while its translation-needing sibling is
excluded from coverage and queries fall back to scans with correct
results.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
Generalize the legacy-reader task-local guard to every batch-remapping
entry point this PR adds: the free helpers in lance-index-core's
remapping module, each index type's load_with_remapping body, the
vector storage batch branches, the QueryRowIdRemapper, and the tagged
branch of open_row_id_remapping. The LEGACY_TRAFFIC_ONLY scope lives
beside the helpers so every crate in the stack checks the same guard;
the check has teeth in debug builds (every test profile) and compiles
away in release.

The v0 lifecycle matrix arms both guards end to end: datasets without
an FRI and with a legacy v0 FRI run scans, filtered queries, appends,
deletes, eager and deferred compaction, index optimization, historical
checkouts, index remapping, and FRI cleanup, asserting the guard never
trips and every result matches its pre-operation baseline. A teeth
test proves the guard rejects both the resolver's tagged branch and
the shared checker inside a legacy-only scope.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
@github-actions github-actions Bot added the A-deps Dependency updates label Sep 10, 2026
LuQQiu and others added 5 commits September 10, 2026 10:29
The parent merge changed InvertedPartition.tokens to Arc<TokenSet>; the
FRI loading paths added by this branch now convert at construction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
fri_query_plan no longer computes sibling exclusions itself: each
filtered name-group's stored provenance goes through the reader's
segment_plans, whose backtrack derives coverage and exclusions in one
pass, and the plan keeps only the exclusions. The sibling-union code
this replaces produced the same bitmaps, so behavior is unchanged.

Also pin the persistence boundary: after an Append commit on a tagged
dataset, every stored segment's fragment_bitmap is byte-identical to
the previous manifest's, proving snapshot-derived coverage never leaks
into stored provenance.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
Every paired remapper field now documents its role: the legacy
synchronous remapper serves index_version 0 and is mutually exclusive
with the batch sibling, which serves tagged histories; both None means
no translation is needed. Each dual-dispatch site labels its legacy
synchronous and tagged asynchronous branches, and each construction
site that sets either field debug-asserts the exclusivity invariant.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RynpAxtwGB9Q9CL4JCvR4
lance-gatekeeper[bot]

This comment was marked as outdated.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 10, 2026
@lance-gatekeeper lance-gatekeeper Bot removed the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 10, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The synchronized base and head only move the tagged FRI capability flag to its non-conflicting bit; this PR retains the same stable patch ID as the reviewed revision. The shared batched remapping boundary still keeps v0 on the legacy path, scopes tagged caches by snapshot, and falls back to scans for unsupported or incomplete coverage; focused scalar, vector, lifecycle, and prewarm coverage remains successful.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-deps Dependency updates A-index Vector index, linalg, tokenizer enhancement New feature or request K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant