Skip to content

Support PostgreSQL hex bytea literals - #19263

Draft
xiangfu0 wants to merge 5 commits into
apache:masterfrom
xiangfu0:xiangfu0/postgresql-bytea-literals
Draft

Support PostgreSQL hex bytea literals#19263
xiangfu0 wants to merge 5 commits into
apache:masterfrom
xiangfu0:xiangfu0/postgresql-bytea-literals

Conversation

@xiangfu0

Copy link
Copy Markdown
Contributor

Depends on

This is intentionally a draft stacked follow-up to keep PostgreSQL parser compatibility separate from the core BYTES_ARRAY feature. Until #19247 merges, GitHub's diff against master also includes that parent PR. After #19247 lands, this branch will be rebased so only the PostgreSQL parser commit remains.

Summary

  • Accept PostgreSQL hex bytea constants in both '\x0102'::bytea and CAST('\x0102' AS BYTEA) forms.
  • Normalize supported bytea constants to Pinot's existing binary-literal representation before either the single-stage or multi-stage engine plans the query.
  • Preserve the existing SQL-standard X'0102' form.
  • Cover scalar literals, BYTES array construction, ingested multi-value BYTES predicates, planner type inference, and runtime execution.

Usage

PostgreSQL infix-cast form:

SELECT '\x0102'::bytea;

Standard CAST spelling:

SELECT CAST('\x0102' AS BYTEA);

Construct and query a BYTES array:

SELECT ARRAY['\x00'::bytea, '\x0102'::bytea, '\xFF'::bytea];

SELECT id, byte_values
FROM events
WHERE ARRAYS_OVERLAP(
  byte_values,
  ARRAY['\x0102'::bytea, '\xCAFE'::bytea]
);

Result values remain hex strings, and an array result reports BYTES_ARRAY metadata.

Compatibility scope

This change supports quoted PostgreSQL hex-format bytea constants. Values must start with \x; upper- and lowercase hex are accepted, as is whitespace between complete byte pairs.

It does not add a general expr::type cast operator, dynamic STRING-to-BYTES conversion, or PostgreSQL's historical octal escape-format bytea input. Normalizing only quoted constants keeps both query engines aligned and avoids per-row decoding.

No Thrift or protobuf schema changes are introduced here. Supported constants normalize to the same scalar binary literals used by X'...' syntax.

Validation

  • Parser/request/Thrift tests: 64/64 passed.
  • Single-stage literal-only broker tests: 9/9 passed.
  • Multi-stage bytea type-inference test: 1/1 passed.
  • BinaryTypes runtime/H2 queries: 4/4 passed across both optimizer variants.
  • BytesMvTypeTest: 14/14 passed for real Avro array ingestion and both query engines.
  • Full integration-test reactor: 63/63 modules and 956 goals passed under JDK 25.
  • spotless:apply, checkstyle:check, license:format, and license:check passed for all five affected modules.

@codecov-commenter

codecov-commenter commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 82.91139% with 27 lines in your changes missing coverage. Please review.
✅ Project coverage is 67.12%. Comparing base (b6cd43b) to head (ab14771).

Files with missing lines Patch % Lines
...nsform/function/ArrayLiteralTransformFunction.java 77.77% 7 Missing and 1 partial ⚠️
...ot/sql/parsers/PostgreSqlByteaLiteralRewriter.java 82.92% 2 Missing and 5 partials ⚠️
...sthandler/BaseSingleStageBrokerRequestHandler.java 75.00% 1 Missing and 2 partials ⚠️
...pache/pinot/common/utils/request/RequestUtils.java 86.36% 2 Missing and 1 partial ⚠️
...e/pinot/common/request/context/LiteralContext.java 83.33% 0 Missing and 2 partials ⚠️
.../pinot/query/runtime/operator/utils/TypeUtils.java 75.00% 0 Missing and 2 partials ⚠️
...e/pinot/common/function/scalar/ArrayFunctions.java 80.00% 0 Missing and 1 partial ⚠️
.../parsers/rewriter/CompileTimeFunctionsInvoker.java 94.44% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #19263      +/-   ##
============================================
+ Coverage     67.09%   67.12%   +0.03%     
  Complexity     1424     1424              
============================================
  Files          3459     3460       +1     
  Lines        219789   219938     +149     
  Branches      35007    35043      +36     
============================================
+ Hits         147457   147627     +170     
+ Misses        60543    60506      -37     
- Partials      11789    11805      +16     
Flag Coverage Δ
integration 100.00% <ø> (ø)
integration1 100.00% <ø> (ø)
integration2 0.00% <ø> (ø)
java-25 67.12% <82.91%> (+0.03%) ⬆️
lane-a 100.00% <ø> (ø)
lane-b 0.00% <ø> (ø)
temurin 67.12% <82.91%> (+0.03%) ⬆️
unittests 67.11% <82.91%> (+0.03%) ⬆️
unittests1 57.79% <82.19%> (+0.05%) ⬆️
unittests2 39.19% <33.54%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@xiangfu0
xiangfu0 force-pushed the xiangfu0/postgresql-bytea-literals branch 2 times, most recently from 5b84662 to ebc4124 Compare August 17, 2026 09:10
Pinot can ingest and project multi-value BYTES columns, but neither query engine could reliably construct an equivalent SQL literal. Single-stage parsing lacked a native MV BYTES representation, while multi-stage execution returned external arrays where DataBlock expects ByteArray[].

Add the Thrift literal arm and the conversions needed by both engines. Keep ordinary single-stage broker-to-server requests encoded as arrayValueConstructor with scalar binary literals so older Literal readers can still decode them. Servers still need the new execution support before using the feature, but mixed-version decoding remains compatible.

Cache constant multi-stage byte-array operands in both internal and external form to avoid per-row allocation. Exercise Avro array<bytes> ingestion, dictionary and raw segment storage, literal projection, and arraysOverlap through both query engines.
Directly exercise TypeUtils conversion from external byte[][] values to the internal ByteArray[] representation required for multi-stage block serialization. Cover empty, single-byte, multi-byte, and unsigned byte content.
Use a BYTES_ARRAY-specific transform operand for mixed literal and dynamic values. Reuse existing ByteArray wrappers and backing bytes while allocating only the required outer result array per row.
Normalize PostgreSQL bytea hex constants to Pinot binary literals before single- and multi-stage planning. Cover parser validation, legacy wire encoding, broker/runtime execution, and Avro BYTES_ARRAY ingestion queries.
@xiangfu0
xiangfu0 force-pushed the xiangfu0/postgresql-bytea-literals branch from ebc4124 to ab14771 Compare August 18, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants