Support PostgreSQL hex bytea literals - #19263
Draft
xiangfu0 wants to merge 5 commits into
Draft
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #19263 +/- ##
============================================
+ Coverage 67.09% 67.12% +0.03%
Complexity 1424 1424
============================================
Files 3459 3460 +1
Lines 219789 219938 +149
Branches 35007 35043 +36
============================================
+ Hits 147457 147627 +170
+ Misses 60543 60506 -37
- Partials 11789 11805 +16
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiangfu0
force-pushed
the
xiangfu0/postgresql-bytea-literals
branch
2 times, most recently
from
August 17, 2026 09:10
5b84662 to
ebc4124
Compare
Pinot can ingest and project multi-value BYTES columns, but neither query engine could reliably construct an equivalent SQL literal. Single-stage parsing lacked a native MV BYTES representation, while multi-stage execution returned external arrays where DataBlock expects ByteArray[]. Add the Thrift literal arm and the conversions needed by both engines. Keep ordinary single-stage broker-to-server requests encoded as arrayValueConstructor with scalar binary literals so older Literal readers can still decode them. Servers still need the new execution support before using the feature, but mixed-version decoding remains compatible. Cache constant multi-stage byte-array operands in both internal and external form to avoid per-row allocation. Exercise Avro array<bytes> ingestion, dictionary and raw segment storage, literal projection, and arraysOverlap through both query engines.
Directly exercise TypeUtils conversion from external byte[][] values to the internal ByteArray[] representation required for multi-stage block serialization. Cover empty, single-byte, multi-byte, and unsigned byte content.
Use a BYTES_ARRAY-specific transform operand for mixed literal and dynamic values. Reuse existing ByteArray wrappers and backing bytes while allocating only the required outer result array per row.
Normalize PostgreSQL bytea hex constants to Pinot binary literals before single- and multi-stage planning. Cover parser validation, legacy wire encoding, broker/runtime execution, and Avro BYTES_ARRAY ingestion queries.
xiangfu0
force-pushed
the
xiangfu0/postgresql-bytea-literals
branch
from
August 18, 2026 09:08
ebc4124 to
ab14771
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on
This is intentionally a draft stacked follow-up to keep PostgreSQL parser compatibility separate from the core BYTES_ARRAY feature. Until #19247 merges, GitHub's diff against master also includes that parent PR. After #19247 lands, this branch will be rebased so only the PostgreSQL parser commit remains.
Summary
Usage
PostgreSQL infix-cast form:
Standard CAST spelling:
Construct and query a BYTES array:
Result values remain hex strings, and an array result reports BYTES_ARRAY metadata.
Compatibility scope
This change supports quoted PostgreSQL hex-format bytea constants. Values must start with \x; upper- and lowercase hex are accepted, as is whitespace between complete byte pairs.
It does not add a general expr::type cast operator, dynamic STRING-to-BYTES conversion, or PostgreSQL's historical octal escape-format bytea input. Normalizing only quoted constants keeps both query engines aligned and avoids per-row decoding.
No Thrift or protobuf schema changes are introduced here. Supported constants normalize to the same scalar binary literals used by X'...' syntax.
Validation