A121: RPC Delay Observability by mbissa · Pull Request #556 · grpc/proposal

mbissa · 2026-06-23T03:51:20Z

No description provided.

…eep simple

markdroth

The design looks good overall. I have one or two minor questions about design details, but they don't affect the overall shape of the design. Most of my comments are about the organization of the document -- the content is good but is not organized in a way that makes it easy to understand.

Please let me know if you have any questions. Thanks!

This PR implements **Attempt-Level RPC Delay Observability** across the core channel transport, built-in load balancers, xDS policies, and the OpenTelemetry telemetry plugin, aligned with [gRPC Proposal A121](grpc/proposal#556).

jggj-c

...

markdroth

This looks much better! Remaining comments are mostly minor.

Please let me know if you have any questions. Thanks!

markdroth

This is getting closer!

Please let me know if you have any questions. Thanks!

markdroth · 2026-09-15T17:40:58Z

 ### Metric Schema

-The following client-side per-call metrics are registered, extending the instrumentation framework defined in [gRPC A66][A66].
+All metrics added in this proposal will start as experimental and therefore off by default. The long term goal will be to de-experimentalize them and have them be on by default, but the exact criteria for that change are TBD.


You added a section on this below, so I don't think it needs to be stated here as well. I would just remove this line.

markdroth · 2026-09-15T17:44:00Z

-| `recordAttemptDelayEnd()` | attempt | the attempt-level delay resolves |
+| Method | Called when |
+|---|---|
+| `RecordDelayStart(delay_type, reason)` | a delay begins, or the `delay_type` changes (which ends the previous delay and starts a new one) |


I don't think the "(which ends the previous delay and starts a new one)" part makes sense. If the channel is the only thing storing the current delay type, then the call tracer cannot automatically end the previous delay, because it doesn't know what the delay type was previously.

I think we need to require that if the delay type changes, the channel must call RecordDelayEnd() with the old delay type before calling RecordDelayStart() with the new delay type.

markdroth · 2026-09-15T17:44:42Z

+| `RecordDelayEnd(delay_type)` | the delay resolves |

-**Caller (channel) responsibilities.** The channel detects when a delay begins and ends, chooses the `delay_type` and `delay_reason`, and calls `Start` when a `delay_type` first appears or changes, `ReasonChanged` when only the reason changes, and `End` when the wait resolves. The scope — and therefore which histogram the delay is recorded to (see [Metric Schema](#metric-schema)) — is chosen by whether the channel calls the method on the call-scoped or the attempt-scoped tracer.
+**Caller (channel) responsibilities.** The channel is the single owner of the current `delay_type`: it stores it, chooses the `delay_type` and `delay_reason`, and passes the `delay_type` on every call so the call tracer never has to store it. It calls `RecordDelayStart` when a `delay_type` first appears or changes, `RecordDelayReasonChanged` when only the reason changes, and `RecordDelayEnd` when the delay resolves. The scope — and therefore which histogram the delay is recorded to (see [Metric Schema](#metric-schema)) — is chosen by whether the channel calls the method on the call-scoped or the attempt-scoped tracer.


As per my comment above, please change "first appears or changes" to just "first appears".

markdroth · 2026-09-15T17:46:08Z

+**Caller (channel) responsibilities.** The channel is the single owner of the current `delay_type`: it stores it, chooses the `delay_type` and `delay_reason`, and passes the `delay_type` on every call so the call tracer never has to store it. It calls `RecordDelayStart` when a `delay_type` first appears or changes, `RecordDelayReasonChanged` when only the reason changes, and `RecordDelayEnd` when the delay resolves. The scope — and therefore which histogram the delay is recorded to (see [Metric Schema](#metric-schema)) — is chosen by whether the channel calls the method on the call-scoped or the attempt-scoped tracer.

-**Call tracer (telemetry plugin) responsibilities.** On `Start`, the plugin opens the `Delay` span (see [Tracing Schema](#tracing-schema)) and begins timing. On `ReasonChanged`, it adds a `Delay state transition` event. On `End`, it closes the span and records the elapsed duration to the corresponding histogram. The plugin owns timing, so the `End` methods carry no duration argument.
+**Call tracer (telemetry plugin) responsibilities.** On `RecordDelayStart`, the plugin opens the `Delay` span (see [Tracing Schema](#tracing-schema)) and records the delay type and start time of the delay. On `RecordDelayReasonChanged`, it adds a `Delay triggered` event. On `RecordDelayEnd`, it closes the span and records the elapsed duration to the histogram identified by the supplied `delay_type`. Because `delay_type` is supplied on every call, the plugin does not need to store it, and the plugin owns timing (the methods carry no duration argument).


The call tracer will also add a Delay triggered event in RecordDelayStart().

markdroth · 2026-09-15T17:46:49Z

-**Call tracer (telemetry plugin) responsibilities.** On `Start`, the plugin opens the `Delay` span (see [Tracing Schema](#tracing-schema)) and begins timing. On `ReasonChanged`, it adds a `Delay state transition` event. On `End`, it closes the span and records the elapsed duration to the corresponding histogram. The plugin owns timing, so the `End` methods carry no duration argument.
+**Call tracer (telemetry plugin) responsibilities.** On `RecordDelayStart`, the plugin opens the `Delay` span (see [Tracing Schema](#tracing-schema)) and records the delay type and start time of the delay. On `RecordDelayReasonChanged`, it adds a `Delay triggered` event. On `RecordDelayEnd`, it closes the span and records the elapsed duration to the histogram identified by the supplied `delay_type`. Because `delay_type` is supplied on every call, the plugin does not need to store it, and the plugin owns timing (the methods carry no duration argument).

 **Cancellation and deadlines.** The channel already notifies the call tracer when an RPC is cancelled or reaches its deadline. If a delay is open at that point, the call tracer automatically terminates it — closing the span and recording the partial duration — so the channel does not need to explicitly end open delays on these paths.


This can no longer be the case if we're saying that only the channel will store the current delay type. We must require that the channel will call RecordDelayEnd() in this case.

markdroth · 2026-09-15T17:47:59Z

 ### Resolver Delays

-The channel records a call-level delay of type **`resolving`** while an RPC is blocked waiting for name resolution. When an RPC is sent while the channel has not yet received its first resolver result, the channel calls `recordCallDelayStart("resolving", reason)` on the call tracer; when the resolver delivers its first result and the RPC can proceed, the channel calls `recordCallDelayEnd()`. If resolution has already completed by the time the RPC is sent, no delay is recorded.
+The channel records a call-level delay of type **`resolving`** while an RPC is blocked waiting for name resolution. When an RPC is sent while the channel has not yet received its first resolver result, the channel calls `RecordDelayStart("resolving", reason)` on the call tracer; when the resolver delivers its first result and the RPC can proceed, the channel calls `RecordDelayEnd("resolving")`. If resolution has already completed by the time the RPC is sent, no delay is recorded.


s/delivers/delivers/

markdroth · 2026-09-15T17:52:44Z

 #### Channel Behavior for LB Pick Delays

-For picker-generated delays, the channel reads the deferred pick's `delay_type` and `delay_reason` and drives the call tracer per the [Call Tracer API](#call-tracer-api-changes): `Start` when the `delay_type` first appears or changes, `ReasonChanged` when only the reason changes, and `End` when a pick assigns a ready subchannel.
+For picker-generated delays, the channel reads the queued pick's `delay_type` and `delay_reason` and drives the call tracer per the [Call Tracer API](#call-tracer-api-changes): `RecordDelayStart` when the `delay_type` first appears or changes, `RecordDelayReasonChanged` when only the reason changes, and `RecordDelayEnd` when a pick assigns a ready subchannel.


As above, please change "first appears or changes" to just "first appears".

mbissa added 6 commits April 23, 2026 16:01

draft lb delay metric

75e1539

delay metric rough

631f5f9

newer changes after review

8ca5bbc

go v2 change and cleanup

5cbe0c0

A121 gRFC title and filename renaming

d39ed0f

fixing title and filename

386b1f8

mbissa self-assigned this Jun 23, 2026

v1 feature parity for go stats handler and detailed rationale

a7b6979

mbissa assigned easwars, ejona86, markdroth and dfawley Jun 23, 2026

AgraVator mentioned this pull request Jun 23, 2026

core, opentelemetry: Implement Attempt-Level RPC Delay Observability (Proposal A121) grpc/grpc-java#12807

Merged

AgraVator reviewed Jun 23, 2026

View reviewed changes

Comment thread A121-rpc-delay-observability.md Outdated

Comment thread A121-rpc-delay-observability.md Outdated

Comment thread A121-rpc-delay-observability.md Outdated

AgraVator reviewed Jun 23, 2026

View reviewed changes

Comment thread A121-rpc-delay-observability.md Outdated

mbissa requested review from dfawley, easwars, ejona86 and markdroth June 23, 2026 21:52

mbissa added 3 commits June 25, 2026 14:00

descope go v2 api verbose changes and cleanup api change section to k…

db81e29

…eep simple

add special section for attempt level delays handled by channel

6f6efbb

addressing review comment

5d8c3e1

mbissa removed their assignment Jun 25, 2026

mbissa added 2 commits June 25, 2026 17:50

fix AI slop

8265fe0

cleaning up some more

f6b5128

AgraVator reviewed Jun 26, 2026

View reviewed changes

Comment thread A121-rpc-delay-observability.md Outdated

markdroth reviewed Jul 14, 2026

View reviewed changes

markdroth mentioned this pull request Jul 27, 2026

A119: Auto-Sharding LB Policy #551

Open

review comments

0baf481

jggj-c approved these changes Aug 19, 2026

View reviewed changes

markdroth reviewed Sep 3, 2026

View reviewed changes

AgraVator reviewed Sep 9, 2026

View reviewed changes

Comment thread A121-rpc-delay-observability.md Outdated

AgraVator mentioned this pull request Sep 10, 2026

Name resolution delay grpc/grpc-java#12893

Open

mbissa added 2 commits September 15, 2026 11:43

review comments

67b990e

review comments

62d2ef1

markdroth reviewed Sep 15, 2026

View reviewed changes

Conversation

mbissa commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markdroth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jggj-c left a comment

Choose a reason for hiding this comment

Uh oh!

markdroth left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

markdroth left a comment

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

markdroth Sep 15, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants