feat(format): add the MinHash LSH scalar index specification

[zhangyue19921010]

Add MinHashLshIndexDetails, the persisted parameters of the MinHash LSH index (num_hashes, num_bands, shingle_size, the tokenizer recorded as the full text search index records it, and signature_version), and the format specification of its two files: the fixed-width signature table and the sorted band table with its page table, together with the signature procedure, the band key, reader navigation, multi-segment semantics and versioning.

The message references lance.table.InvertedIndexDetails from another proto package, so the build script maps that package to the module that already includes its generated code.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer) and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals (this commit)	none (0/3)
Vetoes	none
Voting period	ends Mon 2026-09-14 06:51 UTC (Sun 23:51 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

❌ Gate recommendation: request changes.

1 fixed / 3 remain. Restricting version 0 to built-in tokenizers removes deployment-resource drift, but the durable tokenizer contract still does not uniquely determine token streams across readers and releases.

A viable version 0 would use canonical tokenizer details and a small, fully specified tokenizer subset with conformance vectors for every admitted configuration. Broader Unicode, ICU, stemming, or stop-word support must freeze the exact algorithms and data it depends on. Once that contract is accepted, #9042 should rebase onto it and remove its overlapping schema change.

[lance-gatekeeper]

Version 0 still does not define the token streams it persists by reference. simple and code depend on Rust Unicode tables, icu on ICU4X data, and filters on Unicode normalization, frostem, and stop-word tables; repository history has already changed a stop-word stream without changing these persisted details. The only vector covers default simple ASCII text. An independent reader or later Lance release can therefore produce different signatures from equal details. Freeze the exact algorithms and data and add conformance vectors across every admitted tokenizer, filter, and configuration, or narrow version 0 to a fully specified table-free subset. This continues the tokenizer-identity finding.

[lance-gatekeeper]

The inner InvertedIndexDetails presence and default contract is ambiguous. A present empty message satisfies the outer presence requirement, but the existing FTS decoder expands it to simple, English, length 40, with stemming and stop-word removal enabled; protobuf scalar defaults instead expose false, zero, or absent values, while the MinHash default disables stemming and stop words. Readers can therefore reject or tokenize the same persisted bytes differently. Require canonical explicit shaping values, including base_tokenizer, and reject empty or partial legacy encodings, or define exact absence semantics.

Reproducer

Executed on this head in a temporary Cargo binary depending on rust/lance-index; cargo run completed successfully:

use lance_index::{pbold::InvertedIndexDetails, scalar::InvertedIndexParams};

fn main() {
    let params =
        InvertedIndexParams::try_from(&InvertedIndexDetails::default()).unwrap();
    assert_eq!(params.base_tokenizer, "simple");
    assert_eq!(params.max_token_length, Some(40));
    assert!(params.stem);
    assert!(params.remove_stop_words);
}

[lance-gatekeeper]

Admitting raw makes the empty-input rule contradictory. The current Lance RawTokenizer emits one token containing the entire input for both "" and " ", while Signature Generation says these inputs have no token or signature and must not be indexed. Readers can therefore hash and index them or skip them. Define an explicit preprocessing rule before tokenization, define no-signature behavior solely from emitted token count and correct the examples, or remove raw from version 0.

Reproducer

Executed on this head in a temporary Cargo binary depending on rust/lance-tokenizer; cargo run exited successfully:

use lance_tokenizer::{RawTokenizer, TokenStream, Tokenizer};

fn main() {
    let mut tokenizer = RawTokenizer::default();
    for text in ["", "   "] {
        let mut stream = tokenizer.token_stream(text);
        assert!(stream.advance());
        assert_eq!(stream.token().text, text);
        assert!(!stream.advance());
    }
}