Repository navigation
bench: model remote object storage faithfully for --simulate-latency - #24093
Merged
adriangb merged 3 commits intoOct 14, 2026
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24093 +/- ##
========================================
Coverage 82.80% 82.80%
========================================
Files 1149 1149
Lines 454146 454344 +198
Branches 454146 454344 +198
========================================
+ Hits 376035 376227 +192
- Misses 55260 55264 +4
- Partials 22851 22853 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
Good idea @adriangb let me know when it's reviewable. |
|
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days. |
adriangb
force-pushed
the
claude/storage-latency-simulation-c76ce0
branch
from
October 13, 2026 04:06
af1b50b to
03e6f35
Compare
adriangb
marked this pull request as ready for review
October 13, 2026 04:21
adriangb
force-pushed
the
claude/storage-latency-simulation-c76ce0
branch
from
October 13, 2026 04:48
03e6f35 to
1756fe4
Compare
Contributor
Author
|
@Dandandan I've revived this, rescoped it to be smaller and given it a review pass. Could you ptal? |
Dandandan
approved these changes
Oct 13, 2026
Dandandan
left a comment
Dandandan
left a comment
Contributor
There was a problem hiding this comment.
Looks like a net improvement.
I wonder if we can do some calibration of the latencies against gcs / s3 calls / latencies to increase the realism a bit further.
| /// GET time to first byte, inspired by S3 latencies. | ||
| /// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200 | ||
| /// P50≈32ms, P90≈110ms, P99≈200ms | ||
| const GET_TTFB_MS: &[u64] = &[ |
Contributor
There was a problem hiding this comment.
Would be good to measure/callibrat3 it against s3 / gcs to confirm this is realistic. Did you try that?
Contributor
Author
There was a problem hiding this comment.
No, but I agree we should.
Contributor
There was a problem hiding this comment.
Interesting numbers!
I feel like S3 has a bit better P50/P90 numbers in my experience (I think around 30ms and around 100ms on some rough graphs).
Contributor
There was a problem hiding this comment.
or P99 is even below 100ms in most cases
Contributor
Author
There was a problem hiding this comment.
I re-measured against S3 and GCS and added both. So it will be a mix. I think that's reasonable? I don't think this has to be perfect, there are many object store implemetnations out there. The point is to differentiate from a local SSD.
Dandandan
approved these changes
Oct 13, 2026
| /// GET time to first byte, inspired by S3 latencies. | ||
| /// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200 | ||
| /// P50≈32ms, P90≈110ms, P99≈200ms | ||
| const GET_TTFB_MS: &[u64] = &[ |
Contributor
There was a problem hiding this comment.
Interesting numbers!
I feel like S3 has a bit better P50/P90 numbers in my experience (I think around 30ms and around 100ms on some rough graphs).
Dandandan
approved these changes
Oct 13, 2026
| /// GET time to first byte, inspired by S3 latencies. | ||
| /// Sorted: 25,25,28,28,30,30,30,30,32,32,35, 70,85,100,100,110, 130,150, 180,200 | ||
| /// P50≈32ms, P90≈110ms, P99≈200ms | ||
| const GET_TTFB_MS: &[u64] = &[ |
Contributor
There was a problem hiding this comment.
or P99 is even below 100ms in most cases
adriangb
force-pushed
the
claude/storage-latency-simulation-c76ce0
branch
from
October 13, 2026 15:53
42af6d9 to
228f025
Compare
Contributor
Author
|
run benchmarks env:
SIMULATE_LATENCY: "true" |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing claude/storage-latency-simulation-c76ce0 (228f025) to 45de64c (merge-base) diff Run configurationrun benchmark tpch
env:
SIMULATE_LATENCY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing claude/storage-latency-simulation-c76ce0 (228f025) to 45de64c (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
SIMULATE_LATENCY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing claude/storage-latency-simulation-c76ce0 (228f025) to 45de64c (merge-base) diff Run configurationrun benchmark tpcds
env:
SIMULATE_LATENCY: "true"Results will be posted here when complete File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing claude/storage-latency-simulation-c76ce0 (228f025) to 45de64c (merge-base) diff Run configurationrun benchmark tpch
env:
SIMULATE_LATENCY: "true"CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
adriangb
enabled auto-merge
October 13, 2026 16:41
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing claude/storage-latency-simulation-c76ce0 (228f025) to 45de64c (merge-base) diff Run configurationrun benchmark tpcds
env:
SIMULATE_LATENCY: "true"CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: Comparing claude/storage-latency-simulation-c76ce0 (228f025) to 45de64c (merge-base) diff Run configurationrun benchmark clickbench_partitioned
env:
SIMULATE_LATENCY: "true"CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
github-merge-queue
Bot
removed this pull request from the merge queue due to no response for status checks
Oct 14, 2026
…iles Each request waits for one of 128 connections, then for a random time to first byte, then for its body at 100MiB/s per connection. LIST costs one request per 1000 keys. The store does not override get_ranges, as AmazonS3 and GoogleCloudStorage do not, so the trait default decides how many requests a vectored read costs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Measure each store from a VM in the same region as the bucket, with the object_store clients: S3 from a c7i.2xlarge in us-east-2, GCS from an e2-standard-8 in europe-west1. S3 is approximately 3x faster to the first byte than GCS, so one table cannot fit both. Each latency table is now an equal mix: 10 values at the S3 quantiles and 10 at the GCS quantiles. HEAD is much faster than GET on both, so it gets its own table. One connection gives approximately 100MiB/s on S3 and 80MiB/s on GCS; use 90MiB/s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
claude/storage-latency-simulation-c76ce0
branch
from
October 14, 2026 17:07
228f025 to
259d8fb
Compare
adriangb
enabled auto-merge
October 14, 2026 17:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
--simulate-latencymust make benchmarks pay what S3 or GCS would charge. The currentLatencyObjectStoredoes not, for three reasons:get_rangesand sleeps once.AmazonS3andGoogleCloudStoragedo not overrideget_ranges. The trait default merges ranges that are less than 1MiB apart, and sends each merged range as a separate GET. Thus a Parquet scan that reads 2 distant column chunks costs 2 requests on S3, but 1 request in the simulation. An IO change that alters the request count is invisible to the benchmark.What changes are included in this PR?
SimulatedObjectStorereplacesLatencyObjectStore. Each GET costs:A HEAD costs one connection and one HEAD time to first byte. Each LIST page costs one connection and one LIST time to first byte.
The store does not override
get_ranges, so arrow-rs merges and fans out the ranges exactly as it does against S3. The store overrideslist_with_offset, because S3 and GCS start that listing on the server. Writes, copies and deletes go directly to the local filesystem without a cost, because benchmarks do not measure them.These are not modelled: request failures, retries, and a shared network limit across connections.
Calibration
I measured S3 and GCS on 2026-10-13 with the
object_storeclients, from a VM in the same region as the bucket:S3 is approximately 3x faster to the first byte than GCS, thus one distribution cannot fit both. Each latency table is an equal mix: 10 values at the 5%, 15%, ..., 95% quantiles of S3, and 10 values at the same quantiles of GCS. Thus each request is S3-like or GCS-like with equal probability. The throughput of one connection is 90MiB/s, the mean of the two. The total limit is the network limit of the VM, not of the store, thus it is not modelled.
Are these changes tested?
Yes. 7 tests use tokio's paused clock, so the costs are exact and not flaky:
distant_ranges_are_separate_requestsnear_ranges_are_one_requestdistant_column_chunks_cost_more_requestsget_costs_ttfb_plus_transferhead_pays_only_the_head_ttfblist_costs_one_request_per_pagelist_with_offset_skips_earlier_pagesBenchmark results
The benchmark CI (
c4a-highmem-16), withSIMULATE_LATENCY=trueon both sides. The baseline isLatencyObjectStoreon main (45de64c). Minimum time per query:SELECT *withLIKE) reads almost all columns and is 3.3x slower. Q36-Q42 (selective on oneCounterID) send many small, scattered reads and are 1.6-3.3x slower, because each merged range is a separate request.Results of
--simulate-latencybefore this change are not comparable with results after it.Are there any user-facing changes?
No. This change is in the benchmarks only.
--simulate-latencyandSIMULATE_LATENCYkeep their names.🤖 Generated with Claude Code