feat(index): implement stable partition mapping readers - #9064
Open
LuQQiu wants to merge 14 commits into
Open
Conversation
A reordered rewrite (reclustering) distributes live rows of n source fragments across m destination fragments in scan order, so unlike compaction the destination of a row cannot be derived from row order. This adds the standalone format capability that records and replays that mapping, with no transaction or read-path integration yet: - lance-core/utils/stable_partition: pure translation arithmetic. CountsMatrix stores cumulative per-destination row counts at every 64K-row block boundary; a point lookup is counts base + label rank in one block, a sweep is counter[label]++ per row seeded from any block boundary. Encode/decode for the on-disk form plus content validation. - lance-index/frag_reuse/row_map: the row map file. One Lance file with a single nullable u16 label column (one row per physical source row, NULL = deleted at source) and the encoded counts in a global buffer, so open costs one tail read. RowMapWriter interleaves NULLs from the source deletion vectors while the caller streams live-row labels; RowMapReader offers point, coalesced-batch and sweep translation. - benches/stable_partition_row_map: encoded size and translation costs. 2M rows / 1000 destinations on V2_1: 11.15 bits/row uniform-random labels (worst case, nominal 10) and 6.27 bits/row with 16-destination block locality; sweep ~170M rows/s, full-block label rank 3.8us. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-ups on the stable-partition row map: - The counts header now carries a representation tag. Only the dense grid is written (exact, data-independent size: ~600KB for a 50M-row rewrite across 500 destinations, ~61MB at 1B rows across 1000 — trivial beside the rewrite either way); unknown tags are rejected with a clear error, so sparser encodings can be added later without breaking readers. - RowMapReader::open() now fails loudly on a bad file instead of translating rows to wrong addresses or panicking: it checks the label column's schema, decodes with exact structural checks (magic, version, supported representation, shape, precise payload length), runs the full counts consistency validation, and reconciles label row count against the counts. Batch column casts return errors instead of panicking. - Documented the stable-partition ordering contract the arithmetic rests on (labels in source physical-row order, destinations filled in that same order and never re-sorted, destination list fixed), mirroring the Ordering section of row_addr_remap.rs. - translate_many now subtracts block starts in u64 like translate. - Replaced three copies of a hand-rolled LCG with seeded StdRng; rand is already a workspace dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Deduplicate the ordering-contract paragraph in the module docs and drop a redundant explicit rustdoc link. - open() enforces the full schema contract it claims: exactly one column, named label, u16, nullable. - Document the remaining public writer/reader methods. - Deterministic NULL edge-case test: fully-deleted source, zero-row source, deleted tail drained by finish(), empty translate_many and sweep inputs. - State that sweep's one-block-at-a-time IO is intentional (bounded memory); prefetch belongs to read integration. - Fix the 50M x 500 counts size in docs (1.5 MB, not 600 KB) and allow the size-probe printlns in the bench. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review follow-up: make explicit that the row map represents stable partitions only. A rewrite that sorts rows within a destination cannot be expressed by destination labels (equal labels would rank in source order, not output order) and would need a per-row final-offset encoding as a separate format. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
A-index
This was referenced Sep 5, 2026
LuQQiu
removed this pull request from stack #9066
September 9, 2026 19:15
LuQQiu
changed the title
LuQQiu
added this pull request to stack #9108
September 9, 2026 19:16
LuQQiu
removed this pull request from stack #9108
September 10, 2026 03:15
LuQQiu
added this pull request to stack #9117
September 10, 2026 03:15
LuQQiu
removed this pull request from stack #9117
September 10, 2026 18:17
LuQQiu
added this pull request to stack #9137
September 10, 2026 18:17
This was referenced Sep 10, 2026
LuQQiu
marked this pull request as ready for review
September 10, 2026 18:21
![lance-gatekeeper[bot]](https://avatars.githubusercontent.com/in/4433141?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Contributor
There was a problem hiding this comment.
✅ Gate recommendation: approve.
The label-plus-cumulative-count encoding fits the stable-partition contract: it preserves per-destination input order without an explicit destination/offset entry per row. The immutable writer/reader and lazy MappingReader implementation keep validation and block traversal within this layer. This stacks cleanly after #9106 supplies the reader contract; downstream integration remains isolated to #9067, #9068, and #9107.
lance-gatekeeper
Bot
added
the
K-approved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack reordered: the standalone format proposal is now #9136 (replacing #9065), followed by #9106 → #9064 → #9067 → #9068 → #9107 in native stack #9137. This PR’s diff is byte-identical to its pre-reorder diff. All branches include main
31d78d170; no history was rewritten. Formatting checks pass; Clippy and tests remain blocked by the unavailableobject_store_opendal 0.60.1dependency.Implement immutable stable-partition row-map files and the
MappingReadercontract introduced in #9106. Single and batch physical-row-ID remapping lazily opens the Lance row map and sweeps each touched block once, validating labels against counts. Invalid input is rejected before opening the file. The reader accepts the existing protobuf FragmentDigest directly; there is no duplicate FragmentLayout conversion. Coverage is derived by the common lineage reader.This PR owns stable-partition mapping semantics. Dataset lineage traversal is in #9068. The legacy compaction reader/writer is unchanged. No format documentation changes; format review documentation will follow separately.
Validation: row-map/mapping tests and workspace Clippy passed;
cargo fmt --allpassed.Latest main synchronization (2026-09-10): merged
d7b031f9cthrough the stack without rewriting history.cargo fmt --alland whitespace checks pass. Workspace Clippy could not run: main now requiresobject_store_opendal 0.60.1, but the crates.io index currently resolves only up to 0.60.0. Tests were not rerun after this synchronization because dependency resolution is blocked. Earlier validation above predates this merge.