Define a production-ready default tracing policy - #711
Open
Krisztian F (krisztianfekete) wants to merge 1 commit into
Open
Define a production-ready default tracing policy#711Krisztian F (krisztianfekete) wants to merge 1 commit into
Krisztian F (krisztianfekete) wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Today everything outside ateapi hardcodes
ParentBased(NeverSample), and settingOTEL_TRACES_SAMPLERdoes nothing because an explicit sampler silences the SDK's env handling. This makes troubleshooting production issues impossible via traces.This PR gives every component a sane default and makes the standard OTel env vars work:
parentbased_traceidratio0.1, the router (data plane root) to 0.01, per the discussion on Define a default tracing policy #584.OTEL_TRACES_SAMPLER/OTEL_TRACES_SAMPLER_ARGoverride any of this without a rebuild. Invalid values keep the component default and log a warning instead of inheriting the SDK's fall-open-to-100% behavior.RandomSamplingis derived from the router's resolved policy, so the two root decisions cannot drift.--traceno longer installs a tracer provider at all: the oldNeverSampleprovider injectedsampled=0, which pinned every parent based sampler downstream and would have defeated the server side ratios.--tracestill forces a full end to end trace.parentbased_always_on, so the local Jaeger flow keeps showing every API call.Verified on a kind cluster e2e manually. It resolved samplers logged at startup, a
--traceresume produced one trace across ateapi, atelet, and ateom, an unsampled CLI call got picked up server side, Envoy continued a sampled traceparent while sampling 0 of 30 parentless requests at the 1% default, and an invalid env value fell back to the component default.Gating who may use
--tracestays a separate follow-up (and a discussion), and the more sophisticaed tracing policies belongs in a collector, not in substrate.Fixes #584
cc. Da Huang (@git286)