Skip to content

fix: avoid divide-by-zero in avg for negative-scale decimals - #25888

Open
breken-ai wants to merge 2 commits into
apache:mainfrom
breken-ai:fix/avg-negative-decimal-scale
Open

breken-ai wants to merge 2 commits into
apache:mainfrom
breken-ai:fix/avg-negative-decimal-scale

Conversation

@breken-ai

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

avg over a decimal column with a negative scale panics with attempt to divide by zero instead of returning a result. For example, averaging 10000 and 20000 stored as Decimal128(10, -2) should return 15000.00 as Decimal128(14, 2) (the type Avg::return_type already declares), but the query task panics. The plain, grouped and DISTINCT aggregates all go through the same helper.

What changes are included in this PR?

DecimalAverager::try_new previously computed 10^sum_scale and 10^target_scale separately with pow_wrapping(scale as u32). A negative scale becomes a huge u32 exponent, the wrapped power is 0, and avg then divides target_mul by that zero sum_mul.

The two factors were only ever used as the ratio 10^(target_scale - sum_scale), so the averager now stores that single multiplier:

  • the scale difference is computed in i16, so it is exact for any pair of i8 scales;
  • a negative difference (target scale smaller than the input scale) returns the existing Arithmetic Overflow in AvgAccumulator error, as the old target_mul >= sum_mul check did;
  • the power uses pow_checked, so an unrepresentable multiplier also returns that error instead of wrapping.

avg multiplies the sum by the stored multiplier; the precision validation is unchanged.

What is the testing strategy for this PR?

Added avg_decimal_negative_scale to datafusion/sqllogictest/test_files/aggregate.slt, covering the plain aggregate (value and Decimal128(14, 2) result type), a grouped aggregate and avg(DISTINCT ...) over Decimal128(10, -2) values.

  • On main (1d9be2e10) with only the new test: cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- aggregate.slt fails at the new query with task 11 panicked with message "attempt to divide by zero".
  • With the fix, the same command passes, and the full cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests run passes (524/524 files).
  • cargo test --profile ci -p datafusion-functions-aggregate-common -p datafusion-functions-aggregate passes, cargo clippy -p datafusion-functions-aggregate-common -p datafusion-functions-aggregate --all-targets --all-features -- -D warnings is clean, and cargo fmt --all -- --check passes.

Are there any user-facing changes?

Yes: avg over negative-scale decimals returns the average instead of panicking. No public API changes (the changed fields are private).

AI disclosure: this fix was found, written and tested by the breken-ai agent (Breken). Happy to answer questions about the scale arithmetic during review.

DecimalAverager computed 10^sum_scale and 10^target_scale with
pow_wrapping(scale as u32). A negative scale wraps to a zero factor and
avg() then divided by it. Store the single 10^(target_scale - sum_scale)
multiplier instead, computed with checked exponentiation.

Closes apache#24898.
@github-actions github-actions Bot added sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Sep 30, 2026
}

let Some(scale_mul) = T::Native::from_usize(10_usize)
.and_then(|b| b.pow_checked(scale_diff as u32).ok())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

doesn't the ok() here loses some error information?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it did. pow_checked returns an ArrowError that says what overflowed, and .ok() discarded it. Fixed in ac0e791: try_new now keeps the error.

let scale_mul = ten.pow_checked(scale_diff as u32).map_err(|e| {
    exec_datafusion_err!(
        "Arithmetic Overflow in AvgAccumulator: cannot rescale from scale {sum_scale} to {target_scale}: {e}"
    )
})?;

The message still starts with Arithmetic Overflow in AvgAccumulator. No .slt or Rust test matches on the old text. The from_usize(10) case (can't happen for i32/i64/i128/i256) is reported as an internal error again, as it was before this PR.

I also added unit tests in functions-aggregate-common/src/utils.rs:

  • decimal_averager_negative_scale: Decimal128(10, -2) → (14, 2) rescales a sum of 300 at scale -2 by 10^4 and averages to 1500000.
  • decimal_averager_unrepresentable_multiplier_keeps_cause: try_new(-128, 38, 127) needs 10^255, which doesn't fit in an i128. The error now reads ... cannot rescale from scale -128 to 127: Arithmetic overflow: Overflow happened on: 10 ^ 255. On the previous head (60067ef) the same test fails with just Execution error: Arithmetic Overflow in AvgAccumulator.

cargo test -p datafusion-functions-aggregate-common: 62 passed. cargo clippy -p datafusion-functions-aggregate-common --all-targets --all-features -- -D warnings and cargo fmt --all -- --check are clean, and the branch still merges cleanly into main.

Review follow-up: `pow_checked(..).ok()` dropped the ArrowError. Propagate
it, together with the source and target scales, in the existing
"Arithmetic Overflow in AvgAccumulator" execution error, and report the
impossible `from_usize(10)` failure as an internal error as before. Add
unit tests for the negative-scale multiplier and for the error detail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

functions Changes to functions implementation sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

avg over a decimal with a negative scale panics with "attempt to divide by zero"

2 participants