Skip to content

Fix HttpServiceEndpointResolver leak: share a single resolver instead of one per HTTP handler - #7671

Open
epricer-polly wants to merge 3 commits into
dotnet:mainfrom
epricer-polly:fix/service-discovery-resolver-leak
Open

Fix HttpServiceEndpointResolver leak: share a single resolver instead of one per HTTP handler#7671
epricer-polly wants to merge 3 commits into
dotnet:mainfrom
epricer-polly:fix/service-discovery-resolver-leak

Conversation

@epricer-polly

@epricer-polly epricer-polly commented Aug 1, 2026

Copy link
Copy Markdown

Fixes #7670.

Summary

AddServiceDiscovery creates a new HttpServiceEndpointResolver for every HTTP message handler that is built. Handlers are rebuilt on the normal HttpClientFactory handler-lifetime rotation, so a fresh resolver is allocated periodically for the life of the process. Both call sites are affected:

  • the AddHttpMessageHandler delegate in AddServiceDiscovery(IHttpClientBuilder), and
  • CreateHandler in ServiceDiscoveryHttpMessageHandlerFactory.

HttpServiceEndpointResolver is IAsyncDisposable and owns endpoint-refresh timers plus configuration change-token subscriptions, but ResolvingHttpDelegatingHandler wraps it and never disposes it. Each orphaned resolver stays rooted (by the runtime timer queue and the configuration root) and never becomes eligible for collection; its live timers keep firing, which shows up as a slow, uptime-correlated CPU climb. My two-day production heap diff for a minimal service (Azure Container Apps) showed the instance count grow from 0 to 199.

Fix

Register HttpServiceEndpointResolver as a singleton. Notably this is how it's done with the ServiceEndpointResolver registration in AddServiceDiscoveryCore, and resolve it at both sites. It has no per-client state and caches watchers per service name internally, so a single shared instance is correct and is disposed once with the container.

Tests

Adds HttpServiceEndpointResolverSharingTests:

  • AddServiceDiscoveryCore_RegistersResolverAsSingleton: the resolver is registered and resolves to a single shared instance.
  • AddServiceDiscovery_HttpClient_HandlerUsesSharedResolverSingleton: the resolver captured by the built ResolvingHttpDelegatingHandler is the same container-owned singleton.

Both tests fail against the current code and pass with this change.

@epricer-polly

Copy link
Copy Markdown
Author

@dotnet-policy-service agree company="Polly Insurance, LLC"

@epricer-polly
epricer-polly force-pushed the fix/service-discovery-resolver-leak branch from 2142499 to 5db5c8d Compare August 1, 2026 02:46
@epricer-polly
epricer-polly marked this pull request as ready for review August 1, 2026 03:37
Copilot AI review requested due to automatic review settings August 1, 2026 03:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes an uptime-correlated leak/CPU climb in Microsoft.Extensions.ServiceDiscovery by ensuring HttpServiceEndpointResolver is shared (singleton) instead of being reallocated on every HttpClientFactory handler rotation. This aligns the HTTP resolver’s lifetime with the container so its timers and configuration subscriptions are disposed once with the service provider.

Changes:

  • Register HttpServiceEndpointResolver as a singleton in AddServiceDiscoveryCore.
  • Update both HTTP handler construction paths to resolve and reuse the singleton resolver.
  • Add tests validating singleton registration and that ResolvingHttpDelegatingHandler uses the shared resolver instance.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
test/Libraries/Microsoft.Extensions.ServiceDiscovery.Tests/HttpServiceEndpointResolverSharingTests.cs Adds tests asserting the resolver is registered as a singleton and reused by the HTTP handler.
src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceDiscoveryServiceCollectionExtensions.cs Registers HttpServiceEndpointResolver as a singleton to prevent per-rotation allocations/leaks.
src/Libraries/Microsoft.Extensions.ServiceDiscovery/ServiceDiscoveryHttpClientBuilderExtensions.cs Updates AddServiceDiscovery(IHttpClientBuilder) to use the singleton resolver instead of instantiating a new one per handler build.
src/Libraries/Microsoft.Extensions.ServiceDiscovery/Http/ServiceDiscoveryHttpMessageHandlerFactory.cs Updates factory to inject and reuse the singleton resolver when creating handlers.

@jeffhandley jeffhandley left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there might be a regression here for existing disposal scenarios. Registering HttpServiceEndpointResolver as a singleton appears to make the root provider track it once a service-discovery handler is created. Since the resolver implements only IAsyncDisposable, synchronous ServiceProvider.Dispose() may then throw, whereas previously the resolver was not container-owned.

Please add test coverage for existing synchronous provider disposal, along the lines of the code below. Ideally, establish the passing baseline before this change—rebasing and adding the test in an earlier commit is one option—so we can confirm whether this PR changes the behavior.

[Fact]
public void AddServiceDiscovery_HttpClient_SynchronousProviderDisposalDoesNotThrow()
{
    var services = new ServiceCollection();
    services.AddHttpClient("test").AddServiceDiscovery();

    using var provider = services.BuildServiceProvider();
    using var handler = provider
        .GetRequiredService<IHttpMessageHandlerFactory>()
        .CreateHandler("test");
}

If those tests do show a change in synchronous disposal behavior, please preserve synchronous provider disposal—such as by adding a safe synchronous disposal path for the resolver.

@dotnet-policy-service dotnet-policy-service Bot added the waiting-author-feedback 📭 The author of this issue needs to respond in order for us to continue investigating this issue. label Aug 11, 2026
…covery

Establishes that a synchronous ServiceProvider.Dispose() does not throw when
service discovery is added to an HttpClient, before any change to how the HTTP
resolver is registered. This baseline lets the following commit demonstrate
whether the singleton registration changes that behavior.
ResolvingHttpDelegatingHandler owns an IAsyncDisposable HttpServiceEndpointResolver
but never disposes it, and one is created per handler build (per handler-lifetime
rotation) in both AddServiceDiscovery(IHttpClientBuilder) and
ServiceDiscoveryHttpMessageHandlerFactory. Each orphaned resolver keeps its refresh
timers and configuration change-token subscriptions rooted for the process lifetime,
so they accumulate over uptime (observed 0->199 in a two-day production heap diff).

Register HttpServiceEndpointResolver as a singleton (mirroring the existing
ServiceEndpointResolver registration) and resolve it at both sites. It has no
per-client state and caches watchers per service name internally, so a shared
instance is correct and is disposed with the container.

This changes synchronous ServiceProvider.Dispose() behavior: the resolver is now
container-tracked and implements only IAsyncDisposable, so the baseline
synchronous-disposal test added in the previous commit now fails. The following
commit restores synchronous disposal.
@epricer-polly
epricer-polly force-pushed the fix/service-discovery-resolver-leak branch from b8986c0 to 8b142d1 Compare August 15, 2026 17:03
@dotnet-policy-service dotnet-policy-service Bot removed the waiting-author-feedback 📭 The author of this issue needs to respond in order for us to continue investigating this issue. label Aug 15, 2026
@epricer-polly

Copy link
Copy Markdown
Author

Restructured the PR baseline-first, as @jeffhandley suggested. He was correct, I introduced a regression.

Now 3 commits:

  1. test: add baseline for synchronous provider disposal. AddServiceDiscovery_HttpClient_SynchronousProviderDisposalDoesNotThrow (your snippet) against unmodified main. Passes.
  2. fix the leak, add the regression: share HttpServiceEndpointResolver as a singleton - the singleton registration. Fails.
  3. fix the regression: preserve synchronous provider disposal for the shared resolver. HttpServiceEndpointResolver (and its ResolverEntry and ServiceEndpointWatcher) now implement IDisposable alongside IAsyncDisposable. the synchronous path releases the timers and configuration. Passes again.

Now that HttpServiceEndpointResolver is a container-tracked singleton, a
synchronous ServiceProvider.Dispose() threw because the resolver implemented
only IAsyncDisposable. Implement IDisposable on HttpServiceEndpointResolver
(and its ResolverEntry and ServiceEndpointWatcher) so synchronous disposal
releases the timers and configuration change-token registrations without
throwing. DisposeAsync is unchanged in behavior and additionally awaits
in-flight refreshes, the background cleanup task, and endpoint-provider
disposal; the synchronous path releases the resources whose per-rotation
accumulation this change exists to avoid.

Making ServiceEndpointWatcher implement IDisposable makes CA2000 flag the
watcher creation in ServiceEndpointResolver.CreateResolver, where ownership
transfers to the returned ResolverEntry; suppress it there with the repo's
standard pragma.

Restores the baseline established two commits earlier: synchronous provider
disposal with service discovery no longer throws.
@epricer-polly
epricer-polly force-pushed the fix/service-discovery-resolver-leak branch from 8b142d1 to 31fd917 Compare August 15, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HttpServiceEndpointResolver leaked per HTTP handler build in Microsoft.Extensions.ServiceDiscovery

3 participants