[common] Reject out-of-range integral casts when extracting a variant scalar - #9809
Open
zhuxiangyi wants to merge 1 commit into
Open
[common] Reject out-of-range integral casts when extracting a variant scalar#9809zhuxiangyi wants to merge 1 commit into
zhuxiangyi wants to merge 1 commit into
Conversation
… scalar Extracting a variant number into a narrower integral type went through the generic cast rules, which wrap or saturate a value that does not fit: try_variant_get(v, '$.n', 'int') on 99999999999 returned 1215752191 and 1e30 as bigint returned Long.MAX_VALUE. Spark's own evaluation returns NULL (TRY cast), so with pushVariantIntoScan enabled the same query silently produced different numbers. The shredded typed_value reader shared the problem through the same CastExecutors. Check the truncated value against the target range before casting, and treat a null cast result as an invalid cast, in both VariantGet and the scalar reader of shredded files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This is a bug fix. Extracting a variant number into a narrower integral type returns a wrong number instead of
NULL/an error when the value does not fit.The cast goes through the generic
CastExecutorsrules (Number::intValue,(int) DecimalUtils.castToIntegral(...)), which wrap or saturate an out-of-range value. Withspark.sql.variant.pushVariantIntoScan=trueon Spark 4.1 the cast runs inside the Paimon reader, so the same query returns different numbers depending on the session flag:The shredded
typed_valuepath (BaseVariantReader.ScalarReader) has the same problem through the same cast rules, e.g. aBIGINTtyped value read asINT.This PR checks the truncated value against the target range before delegating to the cast rules, in a helper shared by
VariantGetand the scalar reader, so an out-of-range number becomes an invalid cast (NULLfortry_variant_get, an error forvariant_get), matching Spark. A cast rule returningnull(e.g. aBIGINTthat overflows aDECIMAL(p, s)target) is now also reported as an invalid cast instead of being returned as a silentNULLunderfailOnError=true.In-range behaviour is unchanged: fractional parts are still truncated (
1.5asint→1) and widening casts still succeed.Tests
The new tests reproduce the bug: on
masterthey fail withexpected: null but was: 1215752191, and pass with this change.GenericVariantTest#testVariantGetIntegralOverflow: LONG / DOUBLE / DECIMAL inputs into TINYINT..BIGINT, try mode returnsNULL, strict mode throws; boundary values still cast.VariantShreddingReadTest#testReadIntegralOverflowAsInvalidCast: parameterised over a plain file and a file shredded withn BIGINT, d DOUBLE, covering bothVariantGetand the shredded scalar reader.VariantTestBase:try_variant_getout-of-range extraction returnsNULL; runs under all four Spark 4.x configurations (with/without pushdown, with/without inferred shredding). Verified on Spark 4.1.2: 116 tests pass.API and Format
No.
Documentation
No.