Skip to content

Add V7 raw forward index format for codec pipelines - #19307

Open
xiangfu0 wants to merge 3 commits into
xiangfu0/codex/codec-stack/04-t64-gorillafrom
xiangfu0/codex/codec-stack/05-v7-format
Open

Add V7 raw forward index format for codec pipelines#19307
xiangfu0 wants to merge 3 commits into
xiangfu0/codex/codec-stack/04-t64-gorillafrom
xiangfu0/codex/codec-stack/05-v7-format

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Stacked on #19306 and part of the split of #18229.

Summary

This is the first slice that consumes codecSpec in production forward-index I/O.

  • Adds the self-describing V7 fixed-byte SV INT/LONG writer and reader with an explicit 0xC0DEC0DE discriminator and canonical codec spec in the header.
  • Routes a single legacy-compatible compression invocation through the established raw formats; transforms, chains, and non-default options route to V7.
  • Recreates the executor from the persisted canonical spec on read.
  • Keeps all legacy fixed-byte versions, including historical version 7, on their existing reader path unless the new discriminator is present.

Hardening

  • The writer truncates reused output files, validates actual and composed bounds, caps cumulative pipeline work, and rejects executor/type/entry-size or putInt/putLong mismatches before corrupting state.
  • The reader enforces the 4096-byte spec cap, exact contiguous chunk extents from the offset table through EOF, chunk-count/document-count consistency, per-frame bounds, and exact decoded sizes.
  • The production factory passes segment metadata totalDocs; the reader rejects a mismatched header during construction rather than at first query.
  • Each reader context owns reusable ping-pong decode scratch and closes it with the context.

V7 can now be created and loaded. Until #19308, reload does not reconcile codecSpec changes.

Verification

  • V7 round trips for all runtime codecs and representative chains.
  • Corrupt header, gap/trailing-byte, chunk-count, bounds, type mismatch, and wrong-writer-method tests.
  • Production factory metadata-mismatch coverage.
  • A hand-authored transform-only V7 golden fixture that pins header fields, offsets, decoded values, and exact writer reproduction.
  • Legacy reader compatibility fixtures for prior fixed-byte writer versions.
  • Spotless, Checkstyle, license format, and license check on the affected modules.

Stack

#19284#19285#19305#19306#19307 (this PR)#19308#19309

Review and merge parent-first.

@xiangfu0 xiangfu0 added feature New functionality index Related to indexing (general) index-spi Related to index SPI interfaces serialization Related to data serialization and deserialization needs-attention Used for sensitive changes - allows searching PRs post release to narrow down causes for regression. labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from 1924ed6 to 7014dcc Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from d906d6d to 04dd66f Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from 7014dcc to bdc7c33 Compare August 21, 2026 00:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from 04dd66f to 669848c Compare August 21, 2026 00:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/04-t64-gorilla branch from bdc7c33 to d251cca Compare August 21, 2026 04:16
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/05-v7-format branch from 669848c to e784709 Compare August 21, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality index Related to indexing (general) index-spi Related to index SPI interfaces needs-attention Used for sensitive changes - allows searching PRs post release to narrow down causes for regression. serialization Related to data serialization and deserialization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant