Skip to content

fix(index): reject RabitQ metadata without a code dimension - #9025

Open
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/rq-degenerate-metadata
Open

fix(index): reject RabitQ metadata without a code dimension#9025
LuciferYang wants to merge 1 commit into
lance-format:mainfrom
LuciferYang:fix/rq-degenerate-metadata

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Problem

The code-width check in RabitQuantizationStorage::try_from_batch is gated on the expected width being positive, so metadata that carries no code dimension (code_dim absent and no rotation matrix to recover it from) makes the check compare against zero and skip itself, and a code column of any width loads.

Fixes #9024.

What this changes

Reject rotated_dim() == 0 before the width check, as CorruptFile, and drop the > 0 gate so the width comparison always runs.

No in-tree writer produces that metadata: since #6024 the writer always sets code_dim, and before that it always stored the rotation matrix that parse_buffer recovers the dimension from. So this is corrupt-metadata handling rather than a fix for a state a lance version can write, which is also why the > 0 gate was not protecting a real case: it arrived in #7021, after every writer already emitted code_dim.

Test plan

test_degenerate_metadata_rejects_any_code_width deserializes {"num_bits":1,"packed":true}, checks rotated_dim() is 0, and asserts that an 8-byte code column is rejected with CorruptFile. Restoring the > 0 gate makes it fail: the call returns a storage with code_dim: 0 alongside an 8-byte code column.

test_metadata_without_code_dim_loads_from_the_rotation_matrix covers the shape the rejection must not touch, which is the pre-#6024 one: no code_dim, dimension recovered from the rotation matrix that parse_buffer installs.

  • cargo test -p lance-index --lib vector::bq 232 passed
  • cargo clippy --all --tests --benches -- -D warnings clean
  • cargo fmt --all --check clean

The code-width check in try_from_batch was gated on a positive expected width, so metadata carrying no code dimension (code_dim absent and the rotation matrix not loaded) made the check compare against zero and skip itself, and a code column of any width loaded.
@github-actions github-actions Bot added A-index Vector index, linalg, tokenizer bug Something isn't working labels Sep 7, 2026

@lance-gatekeeper lance-gatekeeper Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gate recommendation: approve.

The change rejects dimensionless RabitQ metadata at the hydrated storage boundary while preserving historical matrix-backed metadata whose dimension is recovered before storage construction. Rejecting this inconsistent state is preferable to inferring a dimension from code bytes, which would make corrupt data authoritative.

@lance-gatekeeper lance-gatekeeper Bot added the K-approved Latest Gatekeeper recommendation permits acceptance. label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-index Vector index, linalg, tokenizer bug Something isn't working K-approved Latest Gatekeeper recommendation permits acceptance.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: RabitQ metadata without a code dimension loads a code column of any width

1 participant