Skip to content

Add scalar function performance baselines - #9136

Draft
connortsui20 wants to merge 1 commit into
ct/scalar-fn-factory-extfrom
ct/scalar-fn-baselines
Draft

Add scalar function performance baselines#9136
connortsui20 wants to merge 1 commit into
ct/scalar-fn-factory-extfrom
ct/scalar-fn-baselines

Conversation

@connortsui20

@connortsui20 connortsui20 commented Aug 1, 2026

Copy link
Copy Markdown
Member

Tracking Issue: #9128

Adds stable scalar function performance baselines for byte_length, l2_norm, l2_denorm, inner_product, cosine_similarity, GeoContains, and GeoIntersects.

The benchmarks use the public construction and execution paths, with no RowFn internals or forced execution strategies. The tensor arms vary width and nullability. The cosine arms also cover both constant-vector representations. The geo arms cover column and constant operands, mostly-disjoint and mostly-overlapping polygons, and nullable inputs.

The l2_denorm arms measure decoding the L2Denorm array encoding rather than a scalar function, since #9138 reclassified it. The benchmark and arm names are unchanged so CodSpeed can still line them up against develop.

The benchmark sizes are all comfortably above the roughly 146-instruction Divan harness floor from #9011. Each benchmark binary also uses vendored mimalloc, since scalar function execution allocates its output inside the timed trace and glibc differences between runner images caused the flakes fixed in #8861. vortex-tensor and vortex-geo run in a dedicated CodSpeed shard.

CodSpeed can only compare a later implementation change when the same benchmark name already exists on develop, so these baselines need to land before the RowFn ports. This PR is stacked on the constructor standardization because the ordinary scalar functions use ScalarFnFactoryExt.

Checks:

  • cargo check -p vortex-array -p vortex-tensor -p vortex-geo --benches
  • cargo bench -p vortex-array --bench byte_length -- --test
  • cargo bench -p vortex-tensor --bench cosine_similarity --bench l2_norm --bench l2_denorm --bench inner_product -- --test
  • cargo bench -p vortex-geo --bench binary_predicates -- --test
  • two repeated sample runs for byte_length and all tensor benchmarks (byte-length medians stayed within 0.4%, tensor medians stayed within 3.3%)
  • cargo clippy -p vortex-array -p vortex-tensor -p vortex-geo --all-targets --all-features
  • cargo clippy --all-targets --all-features
  • cargo +nightly fmt --all
  • yamllint --strict -c .yamllint.yaml .github/workflows/codspeed.yml
  • git diff --check

@codspeed-hq

codspeed-hq Bot commented Aug 1, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1885 untouched benchmarks
🆕 53 new benchmarks
⏩ 1 skipped benchmark1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
🆕 Simulation column_x_column_points N/A 14.1 ms N/A
🆕 Simulation column_x_column_polygons N/A 23.6 ms N/A
🆕 Simulation constant_x_nullable_points N/A 12.6 ms N/A
🆕 Simulation constant_x_nullable_polygons_disjoint N/A 17.9 ms N/A
🆕 Simulation constant_x_points N/A 13.7 ms N/A
🆕 Simulation constant_x_polygons_disjoint N/A 17.8 ms N/A
🆕 Simulation constant_x_polygons_overlapping N/A 2 s N/A
🆕 Simulation column_x_column_polygons N/A 22.1 ms N/A
🆕 Simulation nullable_points_x_constant N/A 12.7 ms N/A
🆕 Simulation nullable_polygons_disjoint_x_constant N/A 16 ms N/A
🆕 Simulation points_x_constant N/A 13.7 ms N/A
🆕 Simulation polygons_disjoint_x_constant N/A 15.6 ms N/A
🆕 Simulation polygons_overlapping_x_constant N/A 26.8 ms N/A
🆕 Simulation multipoint_non_nullable N/A 65.5 ms N/A
🆕 Simulation multipolygon_mixed_validity N/A 77.4 ms N/A
🆕 Simulation multipolygon_non_nullable N/A 81.7 ms N/A
🆕 Simulation multipolygon_random_nulls N/A 45 ms N/A
🆕 Simulation point_mixed_validity N/A 34.1 µs N/A
🆕 Simulation point_non_nullable N/A 32.4 µs N/A
🆕 Simulation point_random_nulls N/A 34 µs N/A
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing ct/scalar-fn-baselines (69edd3a) with ct/scalar-fn-factory-ext (8f10bf2)

Open in CodSpeed

Footnotes

  1. 1 benchmark was skipped, so the baseline result was used instead. If it was deleted from the codebase, click here and archive it to remove it from the performance reports.

@joseph-isaacs

Copy link
Copy Markdown
Contributor

This benchmark has a too long runtime

Adds stable CodSpeed benchmark names for `byte_length`, tensor scalar functions, and the geo binary predicates. The benchmarks use public construction and execution APIs, which lets later implementation changes compare directly against this baseline.

CodSpeed runs tensor and geo in a dedicated shard. Each benchmark binary uses vendored `mimalloc` because scalar function execution allocates its output inside the timed trace.

`L2Denorm` keeps its checked inherent constructor because `ScalarFnFactoryExt` cannot represent its normalized-row invariant.

Signed-off-by: "Connor Tsui" <connor.tsui20@gmail.com>

Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
@connortsui20
connortsui20 force-pushed the ct/scalar-fn-baselines branch from 880ddac to 69edd3a Compare August 3, 2026 15:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants