Skip to content

Define a production-ready default tracing policy - #711

Open
Krisztian F (krisztianfekete) wants to merge 1 commit into
agent-substrate:mainfrom
krisztianfekete:feat/set-sane-tracing-defaults
Open

Define a production-ready default tracing policy#711
Krisztian F (krisztianfekete) wants to merge 1 commit into
agent-substrate:mainfrom
krisztianfekete:feat/set-sane-tracing-defaults

Conversation

@krisztianfekete

Copy link
Copy Markdown
Contributor

Today everything outside ateapi hardcodes ParentBased(NeverSample), and setting OTEL_TRACES_SAMPLER does nothing because an explicit sampler silences the SDK's env handling. This makes troubleshooting production issues impossible via traces.

This PR gives every component a sane default and makes the standard OTel env vars work:

  • Control plane (ateapi, atelet, ateom) defaults to parentbased_traceidratio 0.1, the router (data plane root) to 0.01, per the discussion on Define a default tracing policy #584.
  • OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG override any of this without a rebuild. Invalid values keep the component default and log a warning instead of inheriting the SDK's fall-open-to-100% behavior.
  • Envoy's RandomSampling is derived from the router's resolved policy, so the two root decisions cannot drift.
  • kubectl-ate without --trace no longer installs a tracer provider at all: the old NeverSample provider injected sampled=0, which pinned every parent based sampler downstream and would have defeated the server side ratios. --trace still forces a full end to end trace.
  • ate-controller propagates the two env vars to the ateom worker pods it creates, same as the metric export vars.
  • kind pins ateapi to parentbased_always_on, so the local Jaeger flow keeps showing every API call.

Verified on a kind cluster e2e manually. It resolved samplers logged at startup, a --trace resume produced one trace across ateapi, atelet, and ateom, an unsampled CLI call got picked up server side, Envoy continued a sampled traceparent while sampling 0 of 30 parentless requests at the 1% default, and an invalid env value fell back to the component default.

Gating who may use --trace stays a separate follow-up (and a discussion), and the more sophisticaed tracing policies belongs in a collector, not in substrate.

Fixes #584
cc. Da Huang (@git286)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define a default tracing policy

1 participant