Skip to content

otel: join Workflow to propagated trace and restore operation links #674

Description

@zhongkechen

Summary

Implement the revised shared OpenTelemetry topology tracked by
aws/aws-durable-execution-conformance-tests#99.

_X_AMZN_TRACE_ID.Root defines the canonical execution trace when valid.
However, _X_AMZN_TRACE_ID.Parent is only the immediate remote context exposed
to the Lambda runtime and is not guaranteed to be the durable backend attempt
span directly.

Create Workflow in the canonical trace and allow either of these parents:

  • the remote context reconstructed from _X_AMZN_TRACE_ID.Parent; or
  • the current valid ambient span when it has the same trace ID.

ADOT and community Python auto-instrumentation layers can create the ambient
handler span before the durable plugin runs. OpenTelemetry SpanContext has no
ancestor pointer, so the plugin cannot traverse from that local span to the
durable backend span. Same-trace membership is the available portable check.

Required changes

  • Preserve the remote X-Ray fields separately from ambient context and parse
    Root, Parent, and Sampled independently.
  • Create Workflow with its deterministic execution-scoped span ID in the
    canonical trace.
  • Allow Workflow to be parented to either the reconstructed remote context or
    the current valid same-trace ambient context.
  • Do not require or claim that the reconstructed Parent is directly the span
    named Durable Execution Attempt #1.
  • For Invocation, prefer ambient context only when its trace ID matches the
    canonical trace; otherwise use the reconstructed remote parent.
  • Do not attempt to traverse an ambient span's parent chain; the OTel Python API
    does not expose one.
  • When no valid remote parent can be constructed, ignore unrelated ambient
    context and parent Workflow and all Invocation spans to the deterministic
    synthetic execution root.
  • Restore continuation and replay links using the canonical trace ID and
    operation_id_to_span_id(execution_arn, operation_id).
  • Retain the Workflow and Invocation correlation links required by the shared
    contract.
  • Do not invent an authoritative sampled decision when Sampled is absent.

Valid parent shapes

Propagated remote parent
├── Workflow
└── Same-trace ambient Lambda span
    └── Invocation

Propagated remote parent
└── Same-trace ambient Lambda span
    ├── Workflow
    └── Invocation

The selected Workflow parent can be absent from a backend query. If it is
returned, it may be a Lambda/runtime/instrumentation span rather than the
durable backend attempt span.

Tests

  • Cover direct remote-parent and same-trace ambient-parent Workflow shapes.
  • Cover ambient context from another trace and verify it is rejected.
  • Cover ADOT and the OpenTelemetry community Python layer.
  • Cover application-owned and collector-only provider modes where no ambient
    handler span exists.
  • Cover every header/sampling state from the shared contract.
  • Cover deterministic synthetic fallback behavior.
  • Cover continuation and replay links, wait/resume, retries, and chained
    executions.
  • Do not require Workflow's execution-wide interval to be contained by a local
    ambient span covering only one Lambda invocation.

Update the package README and examples to describe both valid Workflow parent
shapes.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestpkg:otelPackage: aws-durable-execution-sdk-python-otel

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions