Skip to content

Make L2Denorm an array encoding instead of a scalar function - #9138

Draft
connortsui20 wants to merge 1 commit into
developfrom
ct/l2-denorm-encoding
Draft

Make L2Denorm an array encoding instead of a scalar function#9138
connortsui20 wants to merge 1 commit into
developfrom
ct/l2-denorm-encoding

Conversation

@connortsui20

@connortsui20 connortsui20 commented Aug 2, 2026

Copy link
Copy Markdown
Member

Rationale for this change

(semi-related) Tracking Issue: #9129

I would like to propose a rule we hold going forward: a scalar function never verifies its inputs, an encoding may. A scalar function takes anything well-typed and checks nothing past dtypes, which is all return_dtype can see anyway. An encoding owns a physical layout, so it gets VTable::validate and its own constructors, and it can hold invariants between its children. The corollary is the useful half: if you want to scan the data at construction time, you want an encoding.

Note that the rule only runs one way. FoR is the edge case I can think of, since encoded + reference is an ordinary scalar function over any integer column and its validate is purely structural. Nothing checks that the encoded values really are offsets from the minimum, because that is a compression heuristic and not a correctness requirement. So needing verification forces an encoding, but not needing it does not force a scalar function.

L2Denorm broke the rule in the direction that matters: its constructor took an ExecutionCtx and scanned both children for the unit-norm invariant, which ScalarFnFactoryExt has nowhere to put. Everything else follows from that. L2Norm reads its stored norms instead of recomputing, CosineSimilarity and InnerProduct reach into its physical children, L2DenormScheme already listed it in produced_encodings, and persisting it needed a ScalarFnArrayPlugin behind an env var. The operation itself (normalized * norm) is valid for any tensor and any float column, so the invariant only ever existed to justify encoding-specific optimizations.

What changes are included in this PR?

L2Denorm becomes a VTable in vortex-tensor/src/encodings/l2_denorm/ with two slots (normalized, norms).

  • try_new scans for the exact invariants, try_new_trusted skips the scan for lossy normalized children whose stored norms stay authoritative. Neither is unsafe, since breaking the contract gives wrong answers rather than undefined behavior.
  • Slice and filter push down into both children. Filter is new, since the generic ScalarFnArray rule only fires when at most one child is non-constant.
  • L2DenormScheme cascades its children like TemporalScheme, so both HACK TO SUPPORT L2 DENORMALIZATION cases come out of CascadingCompressor. They existed because a scheme producing a ScalarFnArray had no compressible children to be sized against.

What APIs are changed? Are there any user-facing changes?

Before After
scalar_fns::l2_denorm::* encodings::l2_denorm::*
L2Denorm::try_new_array(n, s, ctx) L2Denorm::try_new(n, s, ctx)
unsafe L2Denorm::new_array_unchecked(n, s) L2Denorm::try_new_trusted(n, s)
L2Denorm::new() removed, there is no scalar function
session.scalar_fns().register(L2Denorm) session.arrays().register(L2Denorm), unconditional
ScalarFnArrayPlugin::new(L2Denorm) removed

VX_SCALAR_FN_ARRAY_TENSOR_PLUGIN no longer gates L2Denorm, since the compressor can emit it and readers have to be able to open those files. It still gates the other three tensor plugins. The array ID and the L2DenormMetadata message are unchanged, so the on-disk bytes match what the plugin wrote. L2DenormScheme stays out of ALL_SCHEMES, because wiring it in changes default compression for every tensor column and wants its own numbers.

`L2Denorm` was registered as a `ScalarFnVTable`, but it never behaved like
one. Its constructor took an `ExecutionCtx` and scanned both children to
enforce a unit-norm invariant, `L2Norm` read its stored norms instead of
recomputing, `CosineSimilarity` and `InnerProduct` reached into its physical
children, and the compressor scheme named it as a produced encoding. Those
are all properties of a physical decomposition, not of an operation over
arbitrary well-typed values.

Moves it to `vortex-tensor/src/encodings/l2_denorm/` as a real `VTable` with
two slots (`normalized`, `norms`). Structural validation runs on construction
and on deserialization, `try_new` additionally scans for the exact unit-norm
invariant, and `try_new_trusted` skips that scan for lossy normalized
children whose stored norms stay authoritative. Neither constructor is
`unsafe`, since violating the contract produces wrong answers rather than
undefined behavior.

The encoding keeps the `vortex.tensor.l2_denorm` array ID and the same
two-field metadata message, so the wire format is unchanged.

Slice and filter now push down into both children through `reduce_parent`.
The generic `ScalarFnArray` filter rule only fired when at most one child was
non-constant, which for this encoding was almost never.

Also makes `L2DenormScheme` cascade its two children like `TemporalScheme`
does, which lets both `HACK TO SUPPORT L2 DENORMALIZATION` special cases come
out of `CascadingCompressor`. The scheme now competes on measured size like
every other scheme.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XEo4wnqKfAf1QLJbdrz76j
Signed-off-by: Claude <noreply@anthropic.com>
@connortsui20
connortsui20 force-pushed the ct/l2-denorm-encoding branch from 80daa53 to 63e459b Compare August 2, 2026 18:03
@claude claude Bot added the changelog/break A breaking API change label Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/break A breaking API change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants