[common] Keep blank variant object keys unshredded during schema inference - #9813
Open
zhuxiangyi wants to merge 1 commit into
Open
zhuxiangyi wants to merge 1 commit into
zhuxiangyi wants to merge 1 commit into
Conversation
…rence
InferVariantShreddingSchema turns every key of a sampled object into a
RowType field, and RowType rejects an empty or whitespace-only field name.
A variant like {"":1,"a":2} is valid, but with
variant.inferShreddingSchema enabled the whole file failed to write with
"Field names must contain at least one non-whitespace character".
Skip such keys when building the inferred object type so they stay in the
unshredded value; the other keys are still shredded, and an object made
only of blank keys stays unshredded.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This is a bug fix. With
variant.inferShreddingSchema=true, a Variant object whose key is empty or whitespace-only fails the whole file write:InferVariantShreddingSchema.schemaOfturns every key of a sampled object into aDataFieldand builds aRowTypefrom them, andRowTyperejects a blank field name. An empty key is a valid Variant object key (the same row writes and reads back fine with inference off), so inference must not turn it into a write failure. Other unusual keys (a.b, keys with spaces,typed_value/value/metadata, non-ASCII) already work.The fix skips blank keys when building the inferred object type, so they stay in the unshredded
valuewhile the remaining keys are still shredded; an object made only of blank keys stays unshredded. The alphabetical-order check now iterates the kept fields rather than the original object size.Tests
The new tests reproduce the bug: on
masterthey fail with theIllegalArgumentExceptionabove, and pass with this change.InferVariantShreddingSchemaTest#testInferSchemaSkipsBlankKeys:{"":1," ":2,"a":3}infers to a typed object with onlya;{"":1}alone stays unshredded.InferVariantShreddingWriteTest#testInferSchemaSkipsBlankKeys: writes such rows with inference on, checks the physical Parquet schema, the reconstructed Variant (blank keys preserved) and typed extraction of$.a.VariantTestBase: inserts{"":1,"a":2}and reads it back withto_jsonandvariant_get; runs under all four Spark 4.x configurations. Verified on Spark 4.1.2: 116 tests pass.API and Format
No.
Documentation
No.