From 4fdf68a2cb1fa9f196e2d6d340dbb9f3e46efdb4 Mon Sep 17 00:00:00 2001 From: Joe Isaacs Date: Tue, 4 Aug 2026 15:24:21 +0000 Subject: [PATCH] docs: make the 1 ms benchmark iteration budget an explicit maximum The guide asked for "less than 1ms" without saying whether that was a target or a limit, and without saying which number to measure. Benchmarks have landed at 23.8 ms and 123.4 ms per iteration, and the review conversation on those has no documented rule to point at. State that 1 ms is the maximum, say what to do when a benchmark does not fit, and point at CodSpeed's per-iteration number under HEAD as the value to check -- not the runtime of the whole benchmark binary, which is what "too long" usually gets confused with. Signed-off-by: Joe Isaacs Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01FZJAp4wVgwTVhLyErCWFZn --- docs/developer-guide/benchmarking.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/docs/developer-guide/benchmarking.md b/docs/developer-guide/benchmarking.md index 1655d7f8a08..9a35437b764 100644 --- a/docs/developer-guide/benchmarking.md +++ b/docs/developer-guide/benchmarking.md @@ -119,10 +119,20 @@ const VECTOR_SIZE: &[usize] = &[16, 256, 2048, 8192]; fn my_bench(bencher: Bencher, num_indices: usize) { ... } ``` -### Keep per-iteration execution time under ~1 ms +### Keep per-iteration execution time under 1 ms -Each individual iteration of the benchmarked closure should complete in -**less than 1ms**. This is to keep benchmarks snappy, locally and on CI. +**1 ms is the maximum, not a soft target.** Each individual iteration of the benchmarked +closure must complete in less than 1 ms. This is to keep benchmarks snappy, locally and on +CI. + +A benchmark that needs longer than that is measuring too much work at once. Shrink the +input size until a single iteration fits, split it into smaller parameterized cases, or +gate it with `#[cfg(not(codspeed))]` if it genuinely cannot be made to fit. + +The number to check against the budget is the per-iteration time, not the time the whole +benchmark binary takes. CodSpeed reports exactly that: its performance report on a pull +request lists the per-iteration time under `HEAD` for every benchmark the pull request adds +or changes, so check any new benchmark there before merging. ### Gate CodSpeed-incompatible benchmarks