Add codec pipeline integration tests and design doc - #19309
Open
xiangfu0 wants to merge 1 commit into
Open
Conversation
This was referenced Aug 19, 2026
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/06-reload-enable
branch
from
August 19, 2026 09:51
9737c0c to
aafe8c6
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 19, 2026 09:51
88e1d40 to
026ac0b
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/06-reload-enable
branch
from
August 20, 2026 09:08
aafe8c6 to
37dd99f
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 20, 2026 09:08
026ac0b to
4f0c32d
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/06-reload-enable
branch
from
August 21, 2026 00:49
37dd99f to
aedae05
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 21, 2026 00:49
4f0c32d to
c8810ab
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/06-reload-enable
branch
from
August 21, 2026 04:16
aedae05 to
affc903
Compare
xiangfu0
force-pushed
the
xiangfu0/codex/codec-stack/07-integration-docs
branch
from
August 21, 2026 04:16
c8810ab to
948d51c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Stacked on #19308 and the final test/documentation slice of the split of #18229.
Integration coverage
CodecPipelineIntegrationTestbuilds and queries an offline table with representative pipelines covering all eight built-in codecs on INT/LONG raw columns:LZ4,ZSTD(3),SNAPPY,GZIP.DELTA,SNAPPYandDELTA,GZIP.DELTA,LZ4,DELTA,ZSTD(3),DELTADELTA,LZ4,T64,GORILLA,T64,LZ4, andGORILLA,ZSTD(3).Both SSE and MSE exercise sums, filters, batched point lookups across real chunk boundaries, and cross-codec consistency. The test also verifies dictionary coexistence and a raw STRING column with
ZSTD(3).The test opens every generated segment to prove the config is not ignored: V7 columns must use
FixedByteChunkSVForwardIndexReaderV7and expose the exact canonical spec; legacy-compatible numeric/string columns must retain the expected legacy reader andChunkCompressionType; the dictionary column must remain dictionary encoded.Design documentation
Adds
docs/design/codec-pipeline-v7.mdcovering:pinot-segment-local.Safety and verification
This slice changes only integration tests and documentation. The final reactor run builds all 63 required modules and runs all 88 integration cases. Spotless, Checkstyle, license format, and license check cover every affected module in the stack.
Stack
#19284 → #19285 → #19305 → #19306 → #19307 → #19308 → #19309 (this PR)
Review and merge parent-first.