GH-3772: Bulk skip in RunLengthBitPackingHybridDecoder / DictionaryValuesReader - #3773
Open
abstractdog wants to merge 1 commit into
Open
GH-3772: Bulk skip in RunLengthBitPackingHybridDecoder / DictionaryValuesReader#3773abstractdog wants to merge 1 commit into
abstractdog wants to merge 1 commit into
Conversation
abstractdog
changed the title
…naryValuesReader Add skipInts(int n) to RunLengthBitPackingHybridDecoder. It re-uses the existing readNext() to load each run and then advances currentCount by min(n, currentCount) instead of returning values one-by-one via readInt(). Runs are decoded the same way as before -- the win is dropping the per-value mode switch, array-index arithmetic and method-call overhead that readInt() pays for each value the caller is going to throw away. Propagate to the two ValuesReader wrappers (DictionaryValuesReader, RunLengthBitPackingHybridValuesReader) so callers going through the public ValuesReader.skip(int) contract get the fast path. Motivation: dictionary-encoded columns are ubiquitous in production Parquet, and the default ValuesReader.skip(int) is a naive loop over skip() -- which for dict columns is a RunLengthBitPackingHybridDecoder readInt(). Filter-driven read paths (column-index row ranges, Hive ProbeDecode, arbitrary row-skip) pay that cost per skipped row even when the values are being thrown away. Bench (parquet-benchmarks / RleSkipBenchmark, thrpt, 1 fork, 3x1s warmup, 5x1s measure, 100k values/op, ops/s of individual values): pattern=rle bitWidth=8: 1.38 B/s -> 103.2 B/s (~75x) pattern=packed bitWidth=8: 0.89 B/s -> 4.12 B/s (~4.6x) pattern=mixed bitWidth=8: 0.96 B/s -> 6.85 B/s (~7.1x) RLE runs dominate because a whole run is consumed with a single currentCount decrement; bit-packed runs still get fully unpacked, so the gain there is just the readInt() overhead avoided per value. Tests: TestRunLengthBitPackingHybridDecoderSkip covers RLE-only, PACKED-only, mixed, zero-skip, full-skip, partial-run skip, bitWidth=0, and randomized skip/read alternation across 4K values. All 700 parquet-column tests still pass.
abstractdog
force-pushed
the
probe-decode-support-parquet-java
branch
from
September 4, 2026 19:31
5179bd9 to
f3246b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
ValuesReader.skip(int n)defaults to a naive loop ofskip()calls. For dictionary-encoded columns eachskip()bottoms out inRunLengthBitPackingHybridDecoder.readInt()— a mode switch, array indexing, and a value the caller immediately discards. Any filter-then-skip path (column-index row ranges, hash-join probe filtering, runtime filters) pays this cost per skipped row.RleSkipBenchmark, JMH throughput, JDK 17, 100 k values/op:readInt()loopskipIntsWhat changes are included in this PR?
RunLengthBitPackingHybridDecoder.skipInts(int)— re-usesreadNext()per run, then advancescurrentCountbymin(n, currentCount)instead of walking every value throughreadInt().skip(int)overrides onDictionaryValuesReaderandRunLengthBitPackingHybridValuesReaderdelegating todecoder.skipInts(n).parquet-benchmarks / RleSkipBenchmarkJMH benchmark.readNext()and theValuesReader.skip(int)default are unchanged.Are these changes tested?
TestRunLengthBitPackingHybridDecoderSkipcovers RLE-only, PACKED-only, mixed, zero-skip, full-skip, mid-run partial skip,bitWidth = 0, and a randomised 4 K-value read/skip interleaving cross-checked against areadInt()reference. Allparquet-columntests pass.Are there any user-facing changes?
No. Additive and binary-compatible: no signatures change;
skipInts(n)is semantically identical to N discardedreadInt()s. Existing callers ofValuesReader.skip(int)pick up the fast path with no code change.