WebClient performance collapse at high concurrency with 300KB octet-stream body

[bigstar119]

Environment

• Spring Boot: 3.5.14

• Reactor Netty: (version from Spring Boot BOM)

• JDK: 17

• OS: Linux (CentOS/RHEL)

• Network: 10GbE, cross-machine deployment

Problem Description

We are using WebClient (Reactor Netty) to send 300KB binary data (application/octet-stream) to a remote HTTP service. Under high concurrency (200+ virtual users), the QPS collapses dramatically while latency spikes. The same scenario with RestTemplate (Apache HttpComponents, blocking I/O) maintains stable throughput with no degradation.

Client Configuration

@Configurationpublic class BenchmarkWebClientConfiguration {​    @Bean    public ReactorClientHttpConnector reactorClientHttpConnector() {        ConnectionProvider connectionProvider = ConnectionProvider.builder("bench-pool")                .maxConnections(800)                .maxIdleTime(Duration.ofSeconds(30))                .maxLifeTime(Duration.ofMinutes(5))                .pendingAcquireMaxCount(2000)                .pendingAcquireTimeout(Duration.ofSeconds(10))                .fifo()                .build();​        LoopResources loopResources = LoopResources.create("bench-io", 8, true);​        HttpResources.set(connectionProvider);        HttpResources.set(loopResources);​        HttpClient httpClient = HttpClient.newConnection()                .option(ChannelOption.WRITE_BUFFER_WATER_MARK,                        new WriteBufferWaterMark(64 * 1024, 256 * 1024));​        return new ReactorClientHttpConnector(httpClient);    }​    @Bean(name = "loadBalanceWebClient")    @ConditionalOnBean(ReactorLoadBalancerExchangeFilterFunction.class)    public WebClient loadBalanceWebClient(ReactorClientHttpConnector connector,                                          ReactorLoadBalancerExchangeFilterFunction loadBalancerFilter) {        return WebClient.builder()                .clientConnector(connector)                .filter(loadBalancerFilter)                .build();    }​    @Bean(name = "benchmarkScheduler")    public Scheduler benchmarkScheduler() {        return Schedulers.newBoundedElastic(                Schedulers.DEFAULT_BOUNDED_ELASTIC_SIZE,                Integer.MAX_VALUE,                "bench-cpu",                60,                true);    }}

Client Code

@Componentpublic class OctetStreamBodyRushTestCase extends RushCase {​    @Resource(name = "loadBalanceWebClient")    private WebClient webClient;​    @Resource(name = "benchmarkScheduler")    private Scheduler benchmarkScheduler;​    private static final byte[] BODY_BYTES = RandomUtil.randomBytes(300 * 1000);​    @Override    public CompletableFuture<?> execute() {        String uri = "http://atomic-service/api/v1/benchmark";        CompletableFuture<InternalMonitorRsp> future = new CompletableFuture<>();​        Mono<String> response = webClient.post().uri(uri)                .contentType(MediaType.APPLICATION_OCTET_STREAM)                .bodyValue(BODY_BYTES)                .retrieve()                .onStatus(HttpStatusCode::isError,                        res -> Mono.error(new RuntimeException(                                res.statusCode().value() + ":" + res.statusCode())))                .bodyToMono(String.class)                .publishOn(benchmarkScheduler)                .timeout(Duration.ofSeconds(30))                .doOnSuccess(s -> {                    SystemResponse<InternalMonitorRsp> systemResponse = JSON.parseObject(                            s, new TypeReference<SystemResponse<InternalMonitorRsp>>() {});                    if (SystemCodeEnum.SYS_200.getCode().equals(systemResponse.getCode())) {                        future.complete(systemResponse.getData());                    } else {                        future.complete(NO_HIT_INTERNAL_MONITOR_RSP);                    }                })                .doOnError(ex -> {                    log.error("http request error, fallback", ex);                    future.complete(NO_HIT_INTERNAL_MONITOR_RSP);                });        response.subscribe();​        CompletableFuture[] completableFutures = {                future.thenAccept(resp -> Assert.notNull(resp, "response is null"))        };        return CompletableFuture.allOf(completableFutures);    }}

WebClient is configured with ReactorClientHttpConnector and ReactorLoadBalancerExchangeFilterFunction for service discovery (Nacos).

Server Code

The server endpoint simply receives the 300KB body and returns a JSON response:

@PostMapping(value = "/api/v1/benchmark", consumes = MediaType.APPLICATION_OCTET_STREAM_VALUE)public SystemResponse<MonitorRsp> benchmark(@RequestBody byte[] body) {    return new SystemResponse<>(processBody(body));}

Server runs on Undertow container. Same-machine testing confirms the server has no performance bottleneck (QPS 4132 at 200 VU, same-machine).

Performance Data

WebClient (Reactor Netty) - Cross-machine, 10GbE

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	1,049,421	100.00%	3,497.75	28.88ms	33.72ms	36.37ms	47.15ms	28.51ms
200	588,049	100.00%	1,959.73	73.47ms	191.76ms	219.94ms	410.26ms	101.96ms
300	230,381	100.00%	767.68	341.31ms	665.32ms	687.34ms	715.65ms	390.62ms

QPS drops 44% at 200 VU and 78% at 300 VU. P99 latency increases 15x (47ms -> 715ms).

RestTemplate (Apache HttpComponents, blocking I/O) - Same scenario

VU	Total Requests	Success Rate	QPS	P50	P90	P95	P99	AVG
100	1,107,616	100.00%	3,691.72	27.34ms	28.39ms	29.00ms	31.49ms	27.03ms
200	1,093,366	100.00%	3,643.90	55.35ms	57.90ms	59.57ms	61.73ms	54.82ms
300	1,077,186	100.00%	3,589.64	83.62ms	88.15ms	89.78ms	91.82ms	83.50ms

QPS stays stable at ~3,600 across all VU levels. Latency grows linearly (expected).

webclient-issue-draft.md

[bigstar119]

because issue body can not too big. details please see the webclient-issue-draft.md

[bclozel]

Thanks for the complete report, but I would suggest sharing this with the reactor-netty team. All of your questions and configuration changes are low level HTTP/concurrency related and none of those belong in this repository. There's nothing much we can do here, please reach out to the reactor team.