Skip to content

[common] Keep blank variant object keys unshredded during schema inference - #9813

Open
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:variant-infer-blank-key
Open

zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:variant-infer-blank-key

Conversation

@zhuxiangyi

Copy link
Copy Markdown
Contributor

Purpose

This is a bug fix. With variant.inferShreddingSchema=true, a Variant object whose key is empty or whitespace-only fails the whole file write:

CREATE TABLE T (id INT, v VARIANT) TBLPROPERTIES ('variant.inferShreddingSchema' = 'true');
INSERT INTO T VALUES (1, parse_json('{"":1,"a":2}'));
-- Job aborted ... java.lang.IllegalArgumentException: Field names must contain at least one non-whitespace character.
--   at org.apache.paimon.types.RowType.validateFields(RowType.java:283)
--   at org.apache.paimon.data.variant.InferVariantShreddingSchema.schemaOf(InferVariantShreddingSchema.java:318)

InferVariantShreddingSchema.schemaOf turns every key of a sampled object into a DataField and builds a RowType from them, and RowType rejects a blank field name. An empty key is a valid Variant object key (the same row writes and reads back fine with inference off), so inference must not turn it into a write failure. Other unusual keys (a.b, keys with spaces, typed_value / value / metadata, non-ASCII) already work.

The fix skips blank keys when building the inferred object type, so they stay in the unshredded value while the remaining keys are still shredded; an object made only of blank keys stays unshredded. The alphabetical-order check now iterates the kept fields rather than the original object size.

Tests

The new tests reproduce the bug: on master they fail with the IllegalArgumentException above, and pass with this change.

  • InferVariantShreddingSchemaTest#testInferSchemaSkipsBlankKeys: {"":1," ":2,"a":3} infers to a typed object with only a; {"":1} alone stays unshredded.
  • InferVariantShreddingWriteTest#testInferSchemaSkipsBlankKeys: writes such rows with inference on, checks the physical Parquet schema, the reconstructed Variant (blank keys preserved) and typed extraction of $.a.
  • VariantTestBase: inserts {"":1,"a":2} and reads it back with to_json and variant_get; runs under all four Spark 4.x configurations. Verified on Spark 4.1.2: 116 tests pass.

API and Format

No.

Documentation

No.

…rence

InferVariantShreddingSchema turns every key of a sampled object into a
RowType field, and RowType rejects an empty or whitespace-only field name.
A variant like {"":1,"a":2} is valid, but with
variant.inferShreddingSchema enabled the whole file failed to write with
"Field names must contain at least one non-whitespace character".

Skip such keys when building the inferred object type so they stay in the
unshredded value; the other keys are still shredded, and an object made
only of blank keys stays unshredded.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant