Skip to content

feat(otel): Parent Workflow span onto shared execution trace - #647

Open
ayushiahjolia wants to merge 1 commit into
mainfrom
feat/otel-backend-parented-topology
Open

feat(otel): Parent Workflow span onto shared execution trace#647
ayushiahjolia wants to merge 1 commit into
mainfrom
feat/otel-backend-parented-topology

Conversation

@ayushiahjolia

@ayushiahjolia ayushiahjolia commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

#644

Description

After this change,

  • Trace ID is derived from the backend's _X_AMZN_TRACE_ID Root when valid, otherwise from the execution ARN - stable across reinvocations.

  • Workflow span parents onto the reconstructed remote backend parent (from _X_AMZN_TRACE_ID Parent), or the synthetic execution root when no valid Parent exists. Invocation span additionally prefers the current same-trace ambient span (Span.current()) when one exists, falling back to the same ancestor as the Workflow span.

  • Sampling is decided once per invocation with this precedence: explicit backend Sampled=0/1 → same-trace ambient span's decision → customer's configured sampler (evaluated once). The decision is applied to all durable spans; the sampler is not re-invoked per span.

  • Cross-invocation consistency comes from the inputs being stable (trace ID, upstream Sampled), so deterministic samplers reach the same decision every reinvocation without sharing state across processes.

  • Note that when auto-instrumentation (e.g. ADOT) is active, the Lambda runtime updates _X_AMZN_TRACE_ID Parent to point at the agent's handler span - which is the same span as the ambient Span.current(). So in practice both the Workflow and Invocation spans parent onto that same agent span; they simply reach it through different channels (the header vs. the live context API). The two paths diverge only when no ambient span is active (no auto-instrumentation): the Workflow span still uses the header's Parent, while the Invocation span falls back to the ancestor.

  • The Workflow span is created every invocation with a deterministic span ID but is exported only from the terminal invocation - earlier invocations' copies are left un-ended and discarded. Because each invocation parents its Workflow span onto that invocation's own ambient span (the _X_AMZN_TRACE_ID Parent), the single exported Workflow span carries the last invocation's ambient span as its parent.

Demo/Screenshots

N/A

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change

Testing

Unit Tests

Have unit tests been written for these changes? Updated.

Integration Tests

Have integration tests been written for these changes? Added/Updated

Examples

Has a new example been added for the change? (if applicable) N/A

@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 24, 2026 20:05 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 24, 2026 20:05 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the feat/otel-backend-parented-topology branch from 5c0f1a8 to 95fb7cd Compare August 24, 2026 22:15
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 24, 2026 22:15 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 24, 2026 22:15 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the feat/otel-backend-parented-topology branch from 95fb7cd to d97ef9c Compare August 24, 2026 23:02
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 24, 2026 23:03 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 24, 2026 23:03 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the feat/otel-backend-parented-topology branch from d97ef9c to 6896e3d Compare August 25, 2026 18:08
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 06:40 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the feat/otel-backend-parented-topology branch from 8a88e75 to 40ee121 Compare August 26, 2026 14:12
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 14:12 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 14:12 — with GitHub Actions Inactive
Comment thread otel-plugin/README.md Outdated
@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia force-pushed the feat/otel-backend-parented-topology branch from 40ee121 to f941656 Compare August 26, 2026 14:49
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 15:05 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 15:05 — with GitHub Actions Inactive
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@ayushiahjolia
ayushiahjolia marked this pull request as ready for review August 26, 2026 17:27
@ayushiahjolia
ayushiahjolia requested a review from a team August 26, 2026 17:27
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 17:30 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia temporarily deployed to ai-pr-review-runtime August 26, 2026 17:30 — with GitHub Actions Inactive
@ayushiahjolia
ayushiahjolia deployed to ai-pr-review-runtime August 26, 2026 17:30 — with GitHub Actions Active
Comment on lines +128 to +130
return deferredDecisions.computeIfAbsent(
intent.deferredTraceId(),
key -> delegate.shouldSample(Context.root(), key, name, spanKind, attributes, Collections.emptyList()));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codex AI review

[P2] Re-evaluate deferred sampling per invocation. This provider-lifetime cache is keyed only by trace ID, so later reinvocations never consult a stateful, rate-limited, or remotely updated agent sampler. The first decision persists until eviction, contrary to the documented once-per-invocation behavior. Carry a unique invocation key alongside the canonical trace ID and cache by that key; add a two-invocation test verifying the delegate is called once for each invocation.

@github-actions

Copy link
Copy Markdown
Contributor

Codex AI review

Found one deferred-sampling lifecycle bug. Tests do not cover sampler reevaluation across reinvocations.

Reviewed commit f9416561854224189713d26635026a8925e7c4a4. Workflow run

@github-actions

Copy link
Copy Markdown
Contributor

Claude AI review

No actionable findings. I reviewed the complete SHA-anchored diff (.ai-review-context/pr.diff, all 3,600 lines) plus the base revisions of both OTel plugins, DeterministicIdGenerator, OtelPluginSupport, ExtractedContext, XRayContextExtractor, and LocalDurableTestRunner.

The change reparents the Workflow span onto a shared execution ancestor (remote backend parent when a valid Root+Parent is propagated, else a synthetic ARN-derived root), resolves one sampling decision per invocation, and applies it to every durable span via a delegating DurableSampler (context-key carrier same-classloader, thread-scoped system-property bridge cross-classloader). Every candidate concern resolved to intended, documented behavior:

  • Trace-ID coherence: canonicalTraceId provably equals execCtx.executionAncestor().getTraceId() on both branches, so the Workflow span (forced span ID, null trace ID) always inherits a valid parent trace and never hits the random generateTraceId() path.
  • isSampled(null)==true (agent defer): pairs with an autoconfiguration-installed DurableSampler that makes the authoritative decision (preserving RECORD_ONLY/DROP); the default-sampled ancestor is a deliberate fallback and children are not force-sampled because DurableSampler overrides via the parent-context key.
  • Cross-classloader bridge: the lossy property encoding is read only when the ContextKey misses (agent path), where intents are always attribute-free built-in decisions; the scope is opened/closed synchronously on the same thread, so no leakage on thread reuse.
  • Public API: adding the sampling component to the public ExtractedContext record preserves source and binary compatibility via the retained 2-arg constructor; null-sampling normalization guards legacy Jackson deserialization.
  • Replay link ordering ([operation, Workflow]) and the LocalDurableTestRunner single-EXECUTION-op merge (stable ARN/op-ID, loud failure on >1 stored op) are correct and covered by new regression tests.

Test coverage is thorough and satisfies the AGENTS.md test requirement (sampling precedence, RECORD_ONLY preservation, explicit-sampled override, cross-classloader bridge, trace stability across reinvocations, replay link ordering, 7MB large-result replay).

Residual test risk (could not be verified under read-only, no-build inspection):

  1. OtelPluginSupport.resolveSamplingResult calls SdkTracerProvider.getSampler(); this depends on that method existing in the pinned OpenTelemetry SDK version. The PR's own app-owned-provider tests exercise this branch, so a missing method would surface as a compile failure in CI rather than at runtime.
  2. The ContextExtractor contract now requires extract() to be idempotent/side-effect-free and to return the execution's own context on every invocation; a custom extractor that returns stale or unrelated context could displace the execution trace. Only a documentation/behavioral contract change, not a signature break.
  3. The reflective sampler discovery in DurableSampler.configuredSampler (mirroring the existing ID-generator reflection) is coupled to the builder's private field layout; it throws a clear exception if the layout changes in a future SDK bump.

Reviewed commit f9416561854224189713d26635026a8925e7c4a4. Workflow run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants