Skip to content

[common] Reject out-of-range integral casts when extracting a variant scalar - #9809

Open
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:variant-get-integral-overflow
Open

[common] Reject out-of-range integral casts when extracting a variant scalar#9809
zhuxiangyi wants to merge 1 commit into
apache:masterfrom
zhuxiangyi:variant-get-integral-overflow

Conversation

@zhuxiangyi

@zhuxiangyi zhuxiangyi commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Purpose

This is a bug fix. Extracting a variant number into a narrower integral type returns a wrong number instead of NULL/an error when the value does not fit.

The cast goes through the generic CastExecutors rules (Number::intValue, (int) DecimalUtils.castToIntegral(...)), which wrap or saturate an out-of-range value. With spark.sql.variant.pushVariantIntoScan=true on Spark 4.1 the cast runs inside the Paimon reader, so the same query returns different numbers depending on the session flag:

CREATE TABLE T (id INT, v VARIANT);
INSERT INTO T VALUES (1, parse_json('{"n":99999999999,"d":1e30}')), (2, parse_json('{"n":7,"d":1.5}'));

SELECT id, try_variant_get(v, '$.n', 'int'), try_variant_get(v, '$.d', 'bigint') FROM T ORDER BY id;
-- pushVariantIntoScan=true : (1, 1215752191, 9223372036854775807), (2, 7, 1)   <- wrong
-- pushVariantIntoScan=false: (1, NULL, NULL), (2, 7, 1)                        <- Spark's TRY cast

The shredded typed_value path (BaseVariantReader.ScalarReader) has the same problem through the same cast rules, e.g. a BIGINT typed value read as INT.

This PR checks the truncated value against the target range before delegating to the cast rules, in a helper shared by VariantGet and the scalar reader, so an out-of-range number becomes an invalid cast (NULL for try_variant_get, an error for variant_get), matching Spark. A cast rule returning null (e.g. a BIGINT that overflows a DECIMAL(p, s) target) is now also reported as an invalid cast instead of being returned as a silent NULL under failOnError=true.

In-range behaviour is unchanged: fractional parts are still truncated (1.5 as int1) and widening casts still succeed.

Tests

The new tests reproduce the bug: on master they fail with expected: null but was: 1215752191, and pass with this change.

  • GenericVariantTest#testVariantGetIntegralOverflow: LONG / DOUBLE / DECIMAL inputs into TINYINT..BIGINT, try mode returns NULL, strict mode throws; boundary values still cast.
  • VariantShreddingReadTest#testReadIntegralOverflowAsInvalidCast: parameterised over a plain file and a file shredded with n BIGINT, d DOUBLE, covering both VariantGet and the shredded scalar reader.
  • VariantTestBase: try_variant_get out-of-range extraction returns NULL; runs under all four Spark 4.x configurations (with/without pushdown, with/without inferred shredding). Verified on Spark 4.1.2: 116 tests pass.

API and Format

No.

Documentation

No.

… scalar

Extracting a variant number into a narrower integral type went through the
generic cast rules, which wrap or saturate a value that does not fit:
try_variant_get(v, '$.n', 'int') on 99999999999 returned 1215752191 and
1e30 as bigint returned Long.MAX_VALUE. Spark's own evaluation returns NULL
(TRY cast), so with pushVariantIntoScan enabled the same query silently
produced different numbers. The shredded typed_value reader shared the
problem through the same CastExecutors.

Check the truncated value against the target range before casting, and
treat a null cast result as an invalid cast, in both VariantGet and the
scalar reader of shredded files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant