Skip to content

Add DELTA and DELTADELTA transform codecs - #19305

Open
xiangfu0 wants to merge 3 commits into
xiangfu0/codex/codec-stack/02-runtimefrom
xiangfu0/codex/codec-stack/03-delta-codecs
Open

Add DELTA and DELTADELTA transform codecs#19305
xiangfu0 wants to merge 3 commits into
xiangfu0/codex/codec-stack/02-runtimefrom
xiangfu0/codex/codec-stack/03-delta-codecs

Conversation

@xiangfu0

@xiangfu0 xiangfu0 commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Context

Stacked on #19285 and part of the split of #18229.

What changed

  • Adds DELTA and DELTADELTA as typed-layout-preserving INT/LONG transforms.
  • Keeps the output headerless and the same width/count as the input so transforms can be chained before packing or compression.
  • Defines intentional two's-complement modular arithmetic for overflow-adjacent values.
  • Registers both frozen codec names in the closed runtime registry.
  • Makes persisted decoding explicitly big-endian, independent of a caller view's byte order.

Safety and scope

Registering DELTA/DELTADELTA makes them available to the bounded runtime API, but production forward-index creation/loading is not wired until #19307. This slice changes no segment format or production I/O path.

Verification

  • INT/LONG round trips for empty, boundary, monotonic, negative, sign-changing, and overflow-adjacent data.
  • Allocating and bounded decodeInto paths, transform chains, and malformed-width rejection.
  • Stable hand-authored decode fixtures that pin DELTA and DELTADELTA wire bytes independently of the encoders.
  • Little-endian caller-view regressions proving big-endian persisted decoding.
  • Spotless, Checkstyle, license format, and license check for pinot-segment-local.

Stack

#19284#19285#19305 (this PR)#19306#19307#19308#19309

Review and merge parent-first.

@xiangfu0 xiangfu0 added feature New functionality index Related to indexing (general) serialization Related to data serialization and deserialization performance Related to performance optimization labels Aug 19, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/02-runtime branch from d4f9863 to 2c64261 Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 7aa46f4 to 6de5c07 Compare August 20, 2026 09:07
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/02-runtime branch from 2c64261 to bb596e6 Compare August 21, 2026 00:48
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 6de5c07 to 432f34d Compare August 21, 2026 00:49
@xiangfu0 xiangfu0 added the timeseries-compression Related to time-series data compression label Aug 21, 2026
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/02-runtime branch from bb596e6 to e99c0e8 Compare August 21, 2026 04:15
@xiangfu0
xiangfu0 force-pushed the xiangfu0/codex/codec-stack/03-delta-codecs branch from 432f34d to e4af8ac Compare August 21, 2026 04:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New functionality index Related to indexing (general) performance Related to performance optimization serialization Related to data serialization and deserialization timeseries-compression Related to time-series data compression

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant