Skip to content

Add codec pipeline integration tests and design doc - #19309

Open
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/06-reload-enablefrom
xiangfu0/codex/codec-stack/07-integration-docs
Open

Add codec pipeline integration tests and design doc#19309
xiangfu0 wants to merge 1 commit into
xiangfu0/codex/codec-stack/06-reload-enablefrom
xiangfu0/codex/codec-stack/07-integration-docs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Stacked on #19308 and the final test/documentation slice of the split of #18229.

Integration coverage

CodecPipelineIntegrationTest builds and queries an offline table with representative pipelines covering all eight built-in codecs on INT/LONG raw columns:

  • Legacy-compatible compression-only specs: LZ4, ZSTD(3), SNAPPY, GZIP.
  • V7 compression-handler chains: DELTA,SNAPPY and DELTA,GZIP.
  • Transform/packing pipelines including DELTA,LZ4, DELTA,ZSTD(3), DELTADELTA,LZ4, T64, GORILLA, T64,LZ4, and GORILLA,ZSTD(3).

Both SSE and MSE exercise sums, filters, batched point lookups across real chunk boundaries, and cross-codec consistency. The test also verifies dictionary coexistence and a raw STRING column with ZSTD(3).

The test opens every generated segment to prove the config is not ignored: V7 columns must use FixedByteChunkSVForwardIndexReaderV7 and expose the exact canonical spec; legacy-compatible numeric/string columns must retain the expected legacy reader and ChunkCompressionType; the dictionary column must remain dictionary encoded.

Design documentation

Adds docs/design/codec-pipeline-v7.md covering:

  • The exact bounded grammar (4096-character cap and no leading-zero numeric arguments).
  • Structural config normalization versus semantic executor/V7 canonicalization.
  • Actual runtime placement in pinot-segment-local.
  • V7 framing, exact byte/bit order, codec-library framing, corruption defenses, and golden fixtures.
  • Offline and committed-realtime support versus the mutable consuming representation.
  • Migration/rollback limits, mixed-version safety, resource bounds, and context-owned reusable decode scratch.

Safety and verification

This slice changes only integration tests and documentation. The final reactor run builds all 63 required modules and runs all 88 integration cases. Spotless, Checkstyle, license format, and license check cover every affected module in the stack.

Stack

#19284#19285#19305#19306#19307#19308#19309 (this PR)

Review and merge parent-first.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/06-reload-enable branch from 9737c0c to aafe8c6 Compare August 19, 2026 09:51
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 88e1d40 to 026ac0b Compare August 19, 2026 09:51
@xiangfu0 xiangfu0 added documentation Improvements or additions to documentation testing Related to tests or test infrastructure test-coverage Adds or improves test coverage index Related to indexing (general) labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/06-reload-enable branch from aafe8c6 to 37dd99f Compare August 20, 2026 09:08
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 026ac0b to 4f0c32d Compare August 20, 2026 09:08
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/06-reload-enable branch from 37dd99f to aedae05 Compare August 21, 2026 00:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from 4f0c32d to c8810ab Compare August 21, 2026 00:49
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/06-reload-enable branch from aedae05 to affc903 Compare August 21, 2026 04:16
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/07-integration-docs branch from c8810ab to 948d51c Compare August 21, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation index Related to indexing (general) test-coverage Adds or improves test coverage testing Related to tests or test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant