perf: Concrete TopK array storage by MassivePizza · Pull Request #23609 · apache/datafusion

MassivePizza · 2026-07-15T15:13:27Z

Which issue does this PR close?

N/A

Rationale for this change

Eliminate the overhead of casting with dyn Any for every item.

What changes are included in this PR?

Cast once per batch and store the typed array.
PrimitiveHashTables was already generic, so this was a big and simple win.
StringHashTable was made to use an enum for consistency/readability.

Are these changes tested?

Should be covered by existing tests.
For benching try cargo bench -p datafusion --bench topk_aggregate -- "top k=10 aggregate".

top k=10 aggregate 10000000 time-series rows
                        time:   [10.787 ms 10.940 ms 11.093 ms]
                        change: [−38.879% −36.987% −35.111%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high mild

top k=10 aggregate 10000000 worst-case rows
                        time:   [17.414 ms 17.797 ms 18.203 ms]
                        change: [−46.131% −44.095% −41.958%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  2 (2.00%) high mild
  2 (2.00%) high severe

Benchmarking top k=10 aggregate 10000000 time-series rows [Utf8View]: Collecting 100 samples in estimated 5.9278 s (500 itera
top k=10 aggregate 10000000 time-series rows [Utf8View]
                        time:   [11.777 ms 11.954 ms 12.147 ms]
                        change: [−37.279% −35.443% −33.760%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

Benchmarking top k=10 aggregate 10000000 worst-case rows [Utf8View]: Collecting 100 samples in estimated 6.3472 s (300 iterat
top k=10 aggregate 10000000 worst-case rows [Utf8View]
                        time:   [20.337 ms 20.786 ms 21.301 ms]
                        change: [−37.920% −35.713% −33.401%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe

Are there any user-facing changes?

None.

alamb · 2026-07-16T17:26:44Z

run benchmark topk_aggregate

adriangbot · 2026-07-16T17:27:53Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4994752917-1116-k6xvg 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing concrete-topk-storage (8a7b678) to 95de385 (merge-base) diff using: topk_aggregate
Results will be posted here when complete

File an issue against this benchmark runner

codecov-commenter · 2026-07-16T17:51:47Z

Codecov Report

❌ Patch coverage is 96.36364% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@95de385). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...on/physical-plan/src/aggregates/topk/hash_table.rs	95.45%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23609   +/-   ##
=======================================
  Coverage        ?   80.65%           
=======================================
  Files           ?     1086           
  Lines           ?   366091           
  Branches        ?   366091           
=======================================
  Hits            ?   295262           
  Misses          ?    53235           
  Partials        ?    17594

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

adriangbot · 2026-07-16T18:05:44Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                             HEAD                                   concrete-topk-storage
-----                                                             ----                                   ---------------------
aggregate 10000000 time-series rows                               1.00     63.7±6.94ms        ? ?/sec    1.00     63.5±6.93ms        ? ?/sec
aggregate 10000000 worst-case rows                                1.01     65.1±8.58ms        ? ?/sec    1.00     64.3±8.09ms        ? ?/sec
distinct 10000000 rows asc [TopK]                                 1.41      5.8±0.14ms        ? ?/sec    1.00      4.1±0.07ms        ? ?/sec
distinct 10000000 rows asc [no TopK]                              1.05     55.7±3.17ms        ? ?/sec    1.00     53.0±3.16ms        ? ?/sec
distinct 10000000 rows desc [TopK]                                1.41      5.8±0.08ms        ? ?/sec    1.00      4.1±0.06ms        ? ?/sec
distinct 10000000 rows desc [no TopK]                             1.07     56.6±2.91ms        ? ?/sec    1.00     53.0±2.95ms        ? ?/sec
string aggregate 10000000 time-series rows [Utf8View]             1.00     51.1±2.32ms        ? ?/sec    1.06     54.2±2.90ms        ? ?/sec
string aggregate 10000000 time-series rows [Utf8]                 1.00     49.1±1.39ms        ? ?/sec    1.06     52.0±2.38ms        ? ?/sec
string aggregate 10000000 worst-case rows [Utf8View]              1.00   299.2±16.08ms        ? ?/sec    1.04   311.5±17.93ms        ? ?/sec
string aggregate 10000000 worst-case rows [Utf8]                  1.00   289.7±10.10ms        ? ?/sec    1.07   310.2±11.00ms        ? ?/sec
top k=10 aggregate 10000000 time-series rows                      1.30     11.7±0.35ms        ? ?/sec    1.00      9.0±0.24ms        ? ?/sec
top k=10 aggregate 10000000 time-series rows [Utf8View]           1.26     13.0±0.47ms        ? ?/sec    1.00     10.4±0.34ms        ? ?/sec
top k=10 aggregate 10000000 worst-case rows                       1.29     18.2±0.97ms        ? ?/sec    1.00     14.2±0.73ms        ? ?/sec
top k=10 aggregate 10000000 worst-case rows [Utf8View]            1.27     19.8±1.15ms        ? ?/sec    1.00     15.6±0.77ms        ? ?/sec
top k=10 string aggregate 10000000 time-series rows [Utf8View]    1.01     11.6±0.25ms        ? ?/sec    1.00     11.5±0.21ms        ? ?/sec
top k=10 string aggregate 10000000 time-series rows [Utf8]        1.00      9.8±0.18ms        ? ?/sec    1.00      9.8±0.17ms        ? ?/sec
top k=10 string aggregate 10000000 worst-case rows [Utf8View]     1.00     11.6±0.28ms        ? ?/sec    1.01     11.7±0.41ms        ? ?/sec
top k=10 string aggregate 10000000 worst-case rows [Utf8]         1.00      9.8±0.26ms        ? ?/sec    1.00      9.8±0.21ms        ? ?/sec

Resource Usage

topk_aggregate — base (merge-base)

Metric	Value
Wall time	1140.3s
Peak memory	2.1 GiB
Avg memory	323.3 MiB
CPU user	2089.2s
CPU sys	118.7s
Peak spill	0 B

topk_aggregate — branch

Metric	Value
Wall time	1120.2s
Peak memory	2.1 GiB
Avg memory	319.2 MiB
CPU user	1979.1s
CPU sys	126.4s
Peak spill	0 B

File an issue against this benchmark runner

alamb · 2026-07-16T18:10:53Z

Benchmarks look good -- I am running them once more to see if htey are reproducable

alamb · 2026-07-16T18:10:54Z

run benchmark topk_aggregate

adriangbot · 2026-07-16T18:13:38Z

🤖 Benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4995135144-1119-ltwqj 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing concrete-topk-storage (8a7b678) to 95de385 (merge-base) diff using: topk_aggregate
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-07-16T18:40:05Z

🤖 Benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                                             HEAD                                   concrete-topk-storage
-----                                                             ----                                   ---------------------
aggregate 10000000 time-series rows                               1.00     61.5±8.15ms        ? ?/sec    1.06     65.1±6.29ms        ? ?/sec
aggregate 10000000 worst-case rows                                1.00     59.5±6.85ms        ? ?/sec    1.03     61.5±5.50ms        ? ?/sec
distinct 10000000 rows asc [TopK]                                 1.40      5.8±0.08ms        ? ?/sec    1.00      4.1±0.07ms        ? ?/sec
distinct 10000000 rows asc [no TopK]                              1.00     31.8±0.48ms        ? ?/sec    1.65     52.4±2.91ms        ? ?/sec
distinct 10000000 rows desc [TopK]                                1.39      5.8±0.06ms        ? ?/sec    1.00      4.1±0.06ms        ? ?/sec
distinct 10000000 rows desc [no TopK]                             1.00     31.8±0.30ms        ? ?/sec    1.70     53.9±2.74ms        ? ?/sec
string aggregate 10000000 time-series rows [Utf8View]             1.01     52.3±2.59ms        ? ?/sec    1.00     51.9±2.41ms        ? ?/sec
string aggregate 10000000 time-series rows [Utf8]                 1.01     50.2±2.42ms        ? ?/sec    1.00     49.7±1.68ms        ? ?/sec
string aggregate 10000000 worst-case rows [Utf8View]              1.08   323.4±16.09ms        ? ?/sec    1.00   300.1±13.64ms        ? ?/sec
string aggregate 10000000 worst-case rows [Utf8]                  1.00   299.5±11.02ms        ? ?/sec    1.00   299.2±10.60ms        ? ?/sec
top k=10 aggregate 10000000 time-series rows                      1.31     11.7±0.42ms        ? ?/sec    1.00      9.0±0.30ms        ? ?/sec
top k=10 aggregate 10000000 time-series rows [Utf8View]           1.26     13.0±0.46ms        ? ?/sec    1.00     10.3±0.35ms        ? ?/sec
top k=10 aggregate 10000000 worst-case rows                       1.29     18.2±1.02ms        ? ?/sec    1.00     14.1±0.71ms        ? ?/sec
top k=10 aggregate 10000000 worst-case rows [Utf8View]            1.27     19.7±0.94ms        ? ?/sec    1.00     15.5±0.77ms        ? ?/sec
top k=10 string aggregate 10000000 time-series rows [Utf8View]    1.00     11.6±0.24ms        ? ?/sec    1.02     11.8±0.75ms        ? ?/sec
top k=10 string aggregate 10000000 time-series rows [Utf8]        1.00      9.8±0.13ms        ? ?/sec    1.01      9.9±0.31ms        ? ?/sec
top k=10 string aggregate 10000000 worst-case rows [Utf8View]     1.00     11.6±0.27ms        ? ?/sec    1.01     11.7±0.45ms        ? ?/sec
top k=10 string aggregate 10000000 worst-case rows [Utf8]         1.00      9.8±0.15ms        ? ?/sec    1.01      9.9±0.31ms        ? ?/sec

Resource Usage

topk_aggregate — base (merge-base)

Metric	Value
Wall time	790.2s
Peak memory	2.1 GiB
Avg memory	463.9 MiB
CPU user	2138.8s
CPU sys	106.2s
Peak spill	0 B

topk_aggregate — branch

Metric	Value
Wall time	785.2s
Peak memory	2.1 GiB
Avg memory	470.4 MiB
CPU user	1976.8s
CPU sys	116.1s
Peak spill	0 B

File an issue against this benchmark runner

alamb · 2026-07-16T19:41:29Z

🤔

These two seem to reproducibly get worse

distinct 10000000 rows asc [no TopK]                              1.00     31.8±0.48ms        ? ?/sec    1.65     52.4±2.91ms        ? ?/sec
distinct 10000000 rows desc [no TopK]                             1.00     31.8±0.30ms        ? ?/sec    1.70     53.9±2.74ms        ? ?/sec

Can you look into that @MassivePizza ?

MassivePizza added 4 commits July 15, 2026 16:58

store concrete type in TopK structures

2a56ad5

store concrete string type in TopK hash-table

14a5c02

swap in-place in TopK heap

293a0e8

fix error from cherry-pick

3b28922

github-actions Bot added the physical-plan Changes to the physical-plan crate label Jul 15, 2026

jayzhan211 approved these changes Jul 16, 2026

View reviewed changes

MassivePizza added 2 commits July 16, 2026 17:40

fmt and clippy

6214dfa

Merge branch 'main' into concrete-topk-storage

8a7b678

alamb added the performance Make DataFusion faster label Jul 16, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf: Concrete TopK array storage#23609

perf: Concrete TopK array storage#23609
MassivePizza wants to merge 6 commits into
apache:mainfrom
massive-com:concrete-topk-storage

MassivePizza commented Jul 15, 2026 •

edited

Loading

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

codecov-commenter commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

MassivePizza commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

codecov-commenter commented Jul 16, 2026

Codecov Report

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

adriangbot commented Jul 16, 2026

Uh oh!

alamb commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

MassivePizza commented Jul 15, 2026 •

edited

Loading