Fix HttpServiceEndpointResolver leak: share a single resolver instead of one per HTTP handler - #7671
Conversation
|
@dotnet-policy-service agree company="Polly Insurance, LLC" |
2142499 to
5db5c8d
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes an uptime-correlated leak/CPU climb in Microsoft.Extensions.ServiceDiscovery by ensuring HttpServiceEndpointResolver is shared (singleton) instead of being reallocated on every HttpClientFactory handler rotation. This aligns the HTTP resolver’s lifetime with the container so its timers and configuration subscriptions are disposed once with the service provider.
Changes:
- Register
HttpServiceEndpointResolveras a singleton inAddServiceDiscoveryCore. - Update both HTTP handler construction paths to resolve and reuse the singleton resolver.
- Add tests validating singleton registration and that
ResolvingHttpDelegatingHandleruses the shared resolver instance.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| test/Libraries/Microsoft.Extensions.ServiceDiscovery.Tests/HttpServiceEndpointResolverSharingTests.cs | Adds tests asserting the resolver is registered as a singleton and reused by the HTTP handler. |
| src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceDiscoveryServiceCollectionExtensions.cs | Registers HttpServiceEndpointResolver as a singleton to prevent per-rotation allocations/leaks. |
| src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceDiscoveryHttpClientBuilderExtensions.cs | Updates AddServiceDiscovery(IHttpClientBuilder) to use the singleton resolver instead of instantiating a new one per handler build. |
| src/Libraries/Microsoft.Extensions.ServiceDiscovery/Http/ServiceDiscoveryHttpMessageHandlerFactory.cs | Updates factory to inject and reuse the singleton resolver when creating handlers. |
jeffhandley
left a comment
There was a problem hiding this comment.
I think there might be a regression here for existing disposal scenarios. Registering HttpServiceEndpointResolver as a singleton appears to make the root provider track it once a service-discovery handler is created. Since the resolver implements only IAsyncDisposable, synchronous ServiceProvider.Dispose() may then throw, whereas previously the resolver was not container-owned.
Please add test coverage for existing synchronous provider disposal, along the lines of the code below. Ideally, establish the passing baseline before this change—rebasing and adding the test in an earlier commit is one option—so we can confirm whether this PR changes the behavior.
[Fact]
public void AddServiceDiscovery_HttpClient_SynchronousProviderDisposalDoesNotThrow()
{
var services = new ServiceCollection();
services.AddHttpClient("test").AddServiceDiscovery();
using var provider = services.BuildServiceProvider();
using var handler = provider
.GetRequiredService<IHttpMessageHandlerFactory>()
.CreateHandler("test");
}If those tests do show a change in synchronous disposal behavior, please preserve synchronous provider disposal—such as by adding a safe synchronous disposal path for the resolver.
…covery Establishes that a synchronous ServiceProvider.Dispose() does not throw when service discovery is added to an HttpClient, before any change to how the HTTP resolver is registered. This baseline lets the following commit demonstrate whether the singleton registration changes that behavior.
ResolvingHttpDelegatingHandler owns an IAsyncDisposable HttpServiceEndpointResolver but never disposes it, and one is created per handler build (per handler-lifetime rotation) in both AddServiceDiscovery(IHttpClientBuilder) and ServiceDiscoveryHttpMessageHandlerFactory. Each orphaned resolver keeps its refresh timers and configuration change-token subscriptions rooted for the process lifetime, so they accumulate over uptime (observed 0->199 in a two-day production heap diff). Register HttpServiceEndpointResolver as a singleton (mirroring the existing ServiceEndpointResolver registration) and resolve it at both sites. It has no per-client state and caches watchers per service name internally, so a shared instance is correct and is disposed with the container. This changes synchronous ServiceProvider.Dispose() behavior: the resolver is now container-tracked and implements only IAsyncDisposable, so the baseline synchronous-disposal test added in the previous commit now fails. The following commit restores synchronous disposal.
b8986c0 to
8b142d1
Compare
|
Restructured the PR baseline-first, as @jeffhandley suggested. He was correct, I introduced a regression. Now 3 commits:
|
Now that HttpServiceEndpointResolver is a container-tracked singleton, a synchronous ServiceProvider.Dispose() threw because the resolver implemented only IAsyncDisposable. Implement IDisposable on HttpServiceEndpointResolver (and its ResolverEntry and ServiceEndpointWatcher) so synchronous disposal releases the timers and configuration change-token registrations without throwing. DisposeAsync is unchanged in behavior and additionally awaits in-flight refreshes, the background cleanup task, and endpoint-provider disposal; the synchronous path releases the resources whose per-rotation accumulation this change exists to avoid. Making ServiceEndpointWatcher implement IDisposable makes CA2000 flag the watcher creation in ServiceEndpointResolver.CreateResolver, where ownership transfers to the returned ResolverEntry; suppress it there with the repo's standard pragma. Restores the baseline established two commits earlier: synchronous provider disposal with service discovery no longer throws.
8b142d1 to
31fd917
Compare
Fixes #7670.
Summary
AddServiceDiscoverycreates a newHttpServiceEndpointResolverfor every HTTP message handler that is built. Handlers are rebuilt on the normalHttpClientFactoryhandler-lifetime rotation, so a fresh resolver is allocated periodically for the life of the process. Both call sites are affected:AddHttpMessageHandlerdelegate inAddServiceDiscovery(IHttpClientBuilder), andCreateHandlerinServiceDiscoveryHttpMessageHandlerFactory.HttpServiceEndpointResolverisIAsyncDisposableand owns endpoint-refresh timers plus configuration change-token subscriptions, butResolvingHttpDelegatingHandlerwraps it and never disposes it. Each orphaned resolver stays rooted (by the runtime timer queue and the configuration root) and never becomes eligible for collection; its live timers keep firing, which shows up as a slow, uptime-correlated CPU climb. My two-day production heap diff for a minimal service (Azure Container Apps) showed the instance count grow from 0 to 199.Fix
Register
HttpServiceEndpointResolveras a singleton. Notably this is how it's done with theServiceEndpointResolverregistration inAddServiceDiscoveryCore, and resolve it at both sites. It has no per-client state and caches watchers per service name internally, so a single shared instance is correct and is disposed once with the container.Tests
Adds
HttpServiceEndpointResolverSharingTests:AddServiceDiscoveryCore_RegistersResolverAsSingleton: the resolver is registered and resolves to a single shared instance.AddServiceDiscovery_HttpClient_HandlerUsesSharedResolverSingleton: the resolver captured by the builtResolvingHttpDelegatingHandleris the same container-owned singleton.Both tests fail against the current code and pass with this change.