Skip to content

Add Map logical type and scalar support - #9107

Merged
AdamGS merged 1 commit into
developfrom
adamg/maptype-1
Aug 4, 2026
Merged

Add Map logical type and scalar support#9107
AdamGS merged 1 commit into
developfrom
adamg/maptype-1

Conversation

@AdamGS

@AdamGS AdamGS commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Rationale for this change

Vortex currently has no native logical map type, so Arrow maps must be represented as lists of structs and lose map-specific semantics such as non-null keys and the keys_sorted assertion. This provides the type-level foundation for native Map support; #9108 builds on it with canonical arrays and builders. Related to #428.

What changes are included in this PR?

  • Adds DType::Map and MapDType, including key/value dtypes, outer nullability, sortedness, validation, coercion, display, equality, and hashing.
  • Adds map scalar construction and typed access through Scalar::map, Scalar::try_map, and MapScalar.
  • Adds map dtype and scalar serialization through Serde, Protobuf, and FlatBuffers.
  • Adds Arrow dtype/schema round-tripping, Vortex-to-Arrow map scalar conversion, and DataFusion physical-schema reconciliation.
  • Tests cover dtype and scalar validation, null and empty maps, coercion, serialization round-trips, Arrow schema conversion, and nested extension metadata.

What APIs are changed? Are there any user-facing changes?

This adds the public DType::Map and ScalarValue::Map variants, MapDType, MapScalar, and their constructors and accessors. Consumers that exhaustively match DType or ScalarValue must handle the new variants. The Protobuf and FlatBuffers dtype schemas also gain Map variants.

This PR supports Map dtypes, schemas, and scalars, but not full Arrow MapArray transport.

@AdamGS
AdamGS force-pushed the adamg/maptype-1 branch from e1dfc4b to db2f918 Compare July 31, 2026 11:25
@codspeed-hq

codspeed-hq Bot commented Jul 31, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 1838 untouched benchmarks
⏩ 44 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation compact_sliced[(4096, 90)] 780.3 ns 867.8 ns -10.08%
Simulation new_raw_prim_test_between[i32, 2048] 39.1 µs 33.9 µs +15.35%
Simulation execute_scalar_struct_simple 483.2 µs 427.2 µs +13.12%
Simulation execute_scalar_struct_wide 2.3 ms 2 ms +11.36%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing adamg/maptype-1 (6889de1) with develop (02f89d6)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@AdamGS AdamGS added the changelog/feature A new feature label Jul 31, 2026
@AdamGS
AdamGS force-pushed the adamg/maptype-1 branch 4 times, most recently from a54e5f6 to bad889a Compare July 31, 2026 13:17
@AdamGS
AdamGS force-pushed the adamg/maptype-1 branch 2 times, most recently from 5b965da to 7821711 Compare August 4, 2026 11:52
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
@AdamGS
AdamGS marked this pull request as ready for review August 4, 2026 14:47
@AdamGS
AdamGS merged commit bc9aa10 into develop Aug 4, 2026
72 of 73 checks passed
@AdamGS
AdamGS deleted the adamg/maptype-1 branch August 4, 2026 14:49
AdamGS added a commit that referenced this pull request Aug 4, 2026
## Rationale for this change

Building on the logical type and scalar support in #9107, this makes
map-typed arrays constructible, canonicalizable, and serializable. A
dedicated canonical representation preserves Map semantics instead of
treating maps as ordinary lists of structs.

## What changes are included in this PR?

- Adds the `vortex.map` canonical encoding, backed by
`ListView<Struct<key, value>>` storage while retaining the Map dtype and
`keys_sorted` assertion.
- Adds `MapArray`, validated construction, entry access, validity
delegation, scalar access, serialization, and session registration.
- Adds `MapBuilder` and integrates Map values with generic builder and
canonicalization paths, including empty, constant, and chunked arrays.
- Tests cover valid and malformed storage, nullable and empty values,
duplicate and unsorted keys, builder append behavior, canonicalization,
scalar access, and serialization round-trips.

## What APIs are changed? Are there any user-facing changes?

This adds the public `Map`, `MapArray`, `MapArrayExt`, `MapDataParts`,
and `MapBuilder` APIs. `Canonical` and `CanonicalView` gain Map variants
and corresponding accessors, so exhaustive matches over those enums must
handle Map arrays.

Map arrays can now be built, canonicalized, inspected, and serialized.
Compute paths such as take, filter, masking, compression, and Arrow
array transport continue to reject Map arrays at this point in the
stack.

Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants