Skip to content

Aggregation coherence with ANSI SQL and query engine integrations #9081

Description

@myrrc

Vortex has some specified behavior for aggregation functions, but it's not formalized. Duckdb and Datafusion as our main query engines have other specified behaviour. Our vortex-duckdb and vortex-datafusion integrations have a third set of behavior. Goal of this epic is to bring our integrations to the main query engine processing.

NaN means both + and - nan since both Duckdb and Datafusion treat both the same.

Integer (BIGINT / i64)

Query ANSI DuckDB DataFusion Vortex Vortex + Duckdb
sum([1,2,NULL]) 3 3 3 3 3 (pushed)
min / max([1,2,NULL]) 1 / 2 1 / 2 1 / 2 1 / 2 1 / 2 (pushed)
avg([1,2,NULL]) 1.5 1.5 1.5 1.5 1.5 (pushed)
count([1,2,NULL]) 2 2 2 2 2 (pushed)
sum(all-null) NULL NULL NULL 0 MISMATCH 0 (#9084)
min / max(all-null) NULL NULL NULL NULL NULL
avg(all-null) NULL NULL NULL NaN MISMATCH NaN (#9083)
count(all-null) 0 0 0 0 0

Floating point

Floating point pushdown is turned off since #9069 but we can revert that for all functions but max.
Therefore last column is the situation where we push to floats

Query ANSI DuckDB DataFusion Vortex (skip_nans) Vortex + Duckdb
count([NaN,1,2,NULL]) impl-def 3 3 2 3
sum([NaN,1,2]) impl-def NaN NaN 3 NaN
min([NaN,1,2]) impl-def 1 1 1 1 (not pushed)
max([NaN,1,2]) impl-def NaN NaN 2 NaN
avg([NaN,1,2]) impl-def NaN NaN 1.5 NaN
count([NaN,NaN]) impl-def 2 2 0 2
sum([NaN,NaN]) impl-def NaN NaN 0 NaN
min([NaN,NaN]) impl-def NaN NaN NULL NaN (not pushed)
max([NaN,NaN]) impl-def NaN NaN NULL NaN
avg([NaN,NaN]) impl-def NaN NaN NaN NaN
sum([inf,-inf,0]) impl-def NaN NaN NaN NaN
min / max([inf,-inf,0]) impl-def -inf / inf -inf / inf -inf / inf -inf (not pushed) / inf
sum([inf,1,2]) impl-def inf inf inf inf
sum(all-null) NULL NULL NULL 0 MISMATCH 0
min / max(all-null) NULL NULL NULL NULL NULL
avg(all-null) NULL NULL NULL NaN MISMATCH NaN
count(all-null) 0 0 0 0 0

String (VARCHAR / Utf8)

Query ANSI DuckDB DataFusion Vortex Vortex + DuckDB
min / max(['banana','apple',NULL]) apple / banana apple / banana apple / banana apple / banana apple / banana (pushed)
count(['banana','apple',NULL]) 2 2 2 2 2 (pushed)
min / max(all-null) NULL NULL NULL NULL NULL
count(all-null) 0 0 0 0 0
sum(x) / avg(x) type error error error unsupported not pushed

Metadata

Metadata

Assignees

Labels

epicPublic roadmap umbrella for a major initiative, with work tracked in sub-issues.ext/datafusionRelates to the DataFusion integrationext/duckdbRelates to the DuckDB integration

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions