Skip to content

Document the internal APIs that the OpenTelemetry agent instruments #1241

Description

@pjfanning

This is informational rather than a bug report. The OpenTelemetry Java agent instruments pekko-http to create HTTP server and client spans and to propagate context, and it does so by attaching bytecode advice to specific methods. Most of those methods are internal to pekko-http, so a refactor that is invisible to users can silently stop the instrumentation from working. Filing this so the list is written down somewhere the pekko-http project can see it.

Why "silently": the agent's muzzle checks verify the classes and methods its advice code calls, and disable the instrumentation when they no longer exist. They do not verify the method matchers. If a matched method is renamed, inlined, or restructured, the advice simply never applies — no error, no log, just missing telemetry.

What is used

Public API, no concern, listed for completeness:

Symbol Used for
HttpExt.bindAndHandle, HttpExt.singleRequest server and client entry points
Http.IncomingConnection.handleWith server bindings created from a connection source
Directive.tapply, PathMatcher.apply, PathMatcher$.apply deriving http.route

Internal API:

Symbol Declared Used for
HttpServerBluePrint.requestPreparation @InternalApi private[http] object wraps the http/1.1 request/response bidi to start and end server spans
Http2Ext.bindAndHandleAsync @InternalApi private[http] final class same for http/2, which does not go through bindAndHandle
Http2Ext.system same obtaining a dispatcher for the response future
Http2.streamId @InternalApi private[http] object recognising a request that was replayed through the http/2 stack after an h2c upgrade
PoolMasterActor.startPoolInterface private def in @InternalApi private[http] final class clearing the context when a connection pool actor starts, so pool setup is not attributed to the request that triggered it

The last one is the fragile case. Being a private method it has no stable name, and the agent matches the mangled org$apache$pekko$http$impl$engine$client$PoolMasterActor$$startPoolInterface. That name exists only because a closure in startPoolInterface captures it. If that closure is refactored away the method stops being mangled, or is inlined and disappears from the bytecode entirely.

That is not hypothetical: while instrumenting pekko remoting we found EndpointReader.dispatchMessage in the sources but not in the bytecode, because scala had inlined it.

What would help

Nothing urgent, and no API change is being requested. Two things would reduce the risk if the project is willing:

  1. Awareness that these five are load-bearing for an out-of-tree consumer, so a refactor can mention it in release notes.
  2. If any of them are considered stable in practice, @InternalStableApi would say so explicitly. Http2Ext.bindAndHandleAsync and Http2Blueprint.handleWithStreamIdHeader already carry @noinline with comments about instrumentation, so the project clearly anticipates this kind of consumer.

Related OpenTelemetry work: open-telemetry/opentelemetry-java-instrumentation#19817 (http/2 server support).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions