Repository navigation
Conversation
DecimalAverager computed 10^sum_scale and 10^target_scale with pow_wrapping(scale as u32). A negative scale wraps to a zero factor and avg() then divided by it. Store the single 10^(target_scale - sum_scale) multiplier instead, computed with checked exponentiation. Closes apache#24898.
| } | ||
|
|
||
| let Some(scale_mul) = T::Native::from_usize(10_usize) | ||
| .and_then(|b| b.pow_checked(scale_diff as u32).ok()) |
There was a problem hiding this comment.
doesn't the ok() here loses some error information?
There was a problem hiding this comment.
Yes, it did. pow_checked returns an ArrowError that says what overflowed, and .ok() discarded it. Fixed in ac0e791: try_new now keeps the error.
let scale_mul = ten.pow_checked(scale_diff as u32).map_err(|e| {
exec_datafusion_err!(
"Arithmetic Overflow in AvgAccumulator: cannot rescale from scale {sum_scale} to {target_scale}: {e}"
)
})?;The message still starts with Arithmetic Overflow in AvgAccumulator. No .slt or Rust test matches on the old text. The from_usize(10) case (can't happen for i32/i64/i128/i256) is reported as an internal error again, as it was before this PR.
I also added unit tests in functions-aggregate-common/src/utils.rs:
decimal_averager_negative_scale:Decimal128(10, -2)→(14, 2)rescales a sum of 300 at scale -2 by 10^4 and averages to 1500000.decimal_averager_unrepresentable_multiplier_keeps_cause:try_new(-128, 38, 127)needs 10^255, which doesn't fit in an i128. The error now reads... cannot rescale from scale -128 to 127: Arithmetic overflow: Overflow happened on: 10 ^ 255. On the previous head (60067ef) the same test fails with justExecution error: Arithmetic Overflow in AvgAccumulator.
cargo test -p datafusion-functions-aggregate-common: 62 passed. cargo clippy -p datafusion-functions-aggregate-common --all-targets --all-features -- -D warnings and cargo fmt --all -- --check are clean, and the branch still merges cleanly into main.
Review follow-up: `pow_checked(..).ok()` dropped the ArrowError. Propagate it, together with the source and target scales, in the existing "Arithmetic Overflow in AvgAccumulator" execution error, and report the impossible `from_usize(10)` failure as an internal error as before. Add unit tests for the negative-scale multiplier and for the error detail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Which issue does this PR close?
avgover a decimal with a negative scale panics with "attempt to divide by zero" #24898.Rationale for this change
avgover a decimal column with a negative scale panics withattempt to divide by zeroinstead of returning a result. For example, averaging10000and20000stored asDecimal128(10, -2)should return15000.00asDecimal128(14, 2)(the typeAvg::return_typealready declares), but the query task panics. The plain, grouped andDISTINCTaggregates all go through the same helper.What changes are included in this PR?
DecimalAverager::try_newpreviously computed10^sum_scaleand10^target_scaleseparately withpow_wrapping(scale as u32). A negative scale becomes a hugeu32exponent, the wrapped power is0, andavgthen dividestarget_mulby that zerosum_mul.The two factors were only ever used as the ratio
10^(target_scale - sum_scale), so the averager now stores that single multiplier:i16, so it is exact for any pair ofi8scales;Arithmetic Overflow in AvgAccumulatorerror, as the oldtarget_mul >= sum_mulcheck did;pow_checked, so an unrepresentable multiplier also returns that error instead of wrapping.avgmultiplies the sum by the stored multiplier; the precision validation is unchanged.What is the testing strategy for this PR?
Added
avg_decimal_negative_scaletodatafusion/sqllogictest/test_files/aggregate.slt, covering the plain aggregate (value andDecimal128(14, 2)result type), a grouped aggregate andavg(DISTINCT ...)overDecimal128(10, -2)values.main(1d9be2e10) with only the new test:cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- aggregate.sltfails at the new query withtask 11 panicked with message "attempt to divide by zero".cargo test --profile ci -p datafusion-sqllogictest --test sqllogictestsrun passes (524/524 files).cargo test --profile ci -p datafusion-functions-aggregate-common -p datafusion-functions-aggregatepasses,cargo clippy -p datafusion-functions-aggregate-common -p datafusion-functions-aggregate --all-targets --all-features -- -D warningsis clean, andcargo fmt --all -- --checkpasses.Are there any user-facing changes?
Yes:
avgover negative-scale decimals returns the average instead of panicking. No public API changes (the changed fields are private).AI disclosure: this fix was found, written and tested by the breken-ai agent (Breken). Happy to answer questions about the scale arithmetic during review.