[WIP][POC] Add support for PFOR encoding with delta - #3775
Draft
prtkgaur wants to merge 14 commits into
Draft
Conversation
Implements the PFOR (Patched Frame of Reference) integer compression encoding for INT32 and INT64 columns in the pfor package: - PforConstants: header/vector sizes, max exceptions (65535) - PforEncoderDecoder: histogram-based cost model for optimal bit width - PforValuesWriter: IntPforValuesWriter + LongPforValuesWriter with vector-buffered encoding and interleaved page layout - PforValuesReader: abstract base with lazy per-vector decoding - PforValuesReaderForInt: INT32 decoder using BytePacker - PforValuesReaderForLong: INT64 decoder using BytePackerForLong
Wires PFOR encoding into the parquet-java read/write pipeline: - Encoding.java: add PFOR enum with INT32/INT64 reader dispatch - ParquetProperties.java: add pforEnabled column property with isPforEnabled() and builder methods withPforEncoding() - DefaultV2ValuesWriterFactory.java: PFOR takes priority over BYTE_STREAM_SPLIT and DELTA_BINARY_PACKED for INT32/INT64 - ParquetMetadataConverter.java: guard for PFOR until thrift spec is merged upstream
64 tests covering: - PforEncoderDecoderTest: bit width utilities and histogram-based cost model - PforBitPackingTest: round-trip correctness across bit widths 0-64, partial groups, page header format - PforValuesEndToEndTest: full writer→reader pipeline including reset/reuse, skip, edge cases, random data
Benchmarks encode/decode throughput for int32/int64 across 8 data distributions inspired by Snowflake's NumericComprBenchmark: constant, sequential, small range, high-base-small-range (timestamps), with outliers (exception path), random, TPC-DS date keys, TPC-DS quantity. Uses junit-benchmarks (matches existing delta encoding benchmarks). Prints compression ratios for all distributions during setup. Excluded from normal test runs by surefire's benchmark exclusion.
Writer: - Pre-allocate reusable buffers (deltasBuffer, excPosBuffer, excValBuffer, metadataBuf, packBuf, packPadBuf) in constructor instead of allocating new arrays on every encodeAndFlushVector call - Replace ByteBuffer.allocate().order(LITTLE_ENDIAN) with manual byte shifts into reusable metadataBuf for vector info and exception writes - Emit valid header for totalCount==0 (reader can distinguish empty page from missing encoding) instead of BytesInput.empty() Reader: - Add numElements > valuesCount validation (handles nullable columns where page row count > encoded values) - Move getShortLE/getIntLE/getLongLE from private static in concrete readers to protected static in PforValuesReader base class
Tests cover: - Bad packing mode, log vector size out of range, bad value byte width - Negative num_elements, numElements > valuesCount - Header-only page, truncated offset array, truncated vector data - Corrupted offset pointing past buffer end - Skip past end, negative skip, read past end - Skip across vector boundaries (correctness check)
Pre-allocate reusable decode buffers (deltasBuffer, excPositionsBuffer, unpackPadBuf, unpackTempBuf) in allocateDecodedBuffer instead of allocating new arrays on every decodeVector call. Mirrors the writer-side improvement from the previous commit.
getBytes() now emits a valid 7-byte header even when totalCount==0, so the reader can distinguish an empty PFOR page from a missing encoding. Update assertions from size==0 to size==PFOR_HEADER_SIZE.
They were added unformatted, so spotless:check fails on the branch as it stands.
Bit width, exception count, and exception positions all came off the wire and sized reads and writes unchecked, so a corrupt page raised raw index errors.
parquet.enable.pfor turns PFOR on for INT32 and INT64 columns, following how parquet.enable.bytestreamsplit exposes BYTE_STREAM_SPLIT: a constant, a getter reading the key against the ParquetProperties default, an entry in the class documentation, and a line in the properties the record writer builds. The default is unchanged, so PFOR stays off unless a job asks for it. Also wraps the long line the formatter rejects in ParquetMetadataConverter.
A PFOR vector can now hold the differences between its successive values instead of the values, chosen per vector by costing both with the same model and keeping the cheaper one. Bit 7 of the bit width byte carries the choice, and a delta vector stores its own first value between the vector info and the packed residuals, so it still decodes without reading the vector before it. Differencing and the prefix sum are both modular, and exception values in a delta vector are differences, so the reader patches them in before summing. The decision runs a sampled estimate first and drops the mode where the estimate cannot beat the plain cost, which skips writing the differences out and searching them. The mode is on by default and can be turned off globally or per column with ParquetProperties.withPforDeltaEncoding; the writer keeps whichever mode costs fewer bits, so leaving it on cannot make a page larger.
parquet.enable.pfor.delta allows a PFOR vector to hold the differences between its successive values, and is only consulted where PFOR itself is enabled. It follows parquet.enable.pfor at the same four sites in ParquetOutputFormat, and the default is unchanged, so the mode stays on wherever PFOR is. The key is what lets a job configured only through a Configuration decline differencing; with parquet.enable.pfor alone it could turn PFOR on but would always get the mode the cost model preferred.
prtkgaur
force-pushed
the
pforEncodingDelta
branch
from
September 5, 2026 01:31
c99bf98 to
2380751
Compare
CurtHagenlocher
added a commit
to clast-project/engineered-wood
that referenced
this pull request
Sep 5, 2026
FSST's proposal asks for encoding 10 and does not get it. ALP claimed 10 too, shipped here first, and has since been merged into parquet.thrift on parquet-format main -- so 10 is settled and not FSST's. FSST took 11 here, which is what the arrow-rs proof-of-concept predicted would happen once ALP landed. 11 is no longer free either. apache/parquet-format#617 proposes PFOR (Patched Frame of Reference) as encoding 11, and unlike the FSST proposal it arrives with two implementations behind it: apache/parquet-java#3775 and apache/arrow-rs#10977, both of which write 11. So FSST moves to 12 and 11 is reserved. The collision is not one a reader can detect and report: a decoder reads the encoding byte, believes it, and misreads the page body -- there is no magic or length that disagrees. That makes it worth vacating the slot now rather than after files exist. The number lives only on the enum member; every other site goes through Encoding.Fsst, so this is a one-line format change plus its documentation. Breaking for anyone who has persisted the numeric value, which the [Experimental] attribute on the member has always warned about. Adds EncodingWireNumberTests to pin all of the numbers, including that nothing answers to 11. Round-trip tests cannot see a renumber: this library would write and read its own files happily either way and only disagree with other implementations, which is exactly the failure mode that produced the move. Also corrects a stale README claim that FSST_16 is unimplemented and rejected; both symbol table widths have shipped since 2026-08-13. Claude-Session: https://claude.ai/code/session_01UX8ZZxcXf5q4EqqwhDntNA Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The frame was the vector minimum, which leaves one value far below a tight cluster forcing a bit width wide enough to reach it. Exceptions could not help, because with the frame at the minimum every residual is non-negative and only values above the packed window ever exceed it. The frame is now searched, and may sit anywhere in the column's type. A value below it wraps under the modular subtraction to a residual too large for the width, which is the same unsigned test a value above the window fails, so it becomes an ordinary exception carrying its unreduced value. There is no sign, no direction, and no second kind of exception. Nothing changes on the wire or in the reader: the frame already travels in the vector info at full width, and the reader only adds it back before patching. Pages written this way were always readable. The search matches the C++ and Rust implementations bucket for bucket, so the three writers agree on the frame for the same input. It walks min and max, returns early on a constant vector, then builds 256 shift-bucketed counts in the same pass as the width histogram. The minimum is costed exactly as candidate zero, and a sliding window over the buckets is seeded with that cost, so the scan can decline a winner but never accept a loser. Only a winning window pays the second walk that lowers the frame onto the smallest real value it covers, and that frame is costed exactly and taken only if it is strictly cheaper. The delta mode searches the frame of the differences too, which is what turns a constant-step column into a width-0 vector with the leading zero difference as its one exception. PforFrameSearchTest covers the cluster-plus-outlier shapes, patching on both sides of the window, the type extremes, every vector size, a frame searched per vector, and randomized shapes held against an independently computed minimum-frame cost so the search can never lose. Six assertions in PforDeltaModeTest move to the cheaper answers the search now finds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?