fix(ffe): fall back to direct EVP intake in agentless mode by leoromanovsky · Pull Request #4489 · DataDog/dd-trace-php

leoromanovsky · 2026-11-12T23:14:07Z

Description

Agentless flag evaluation currently configures event delivery as Agent-only. Exposures and evaluation-count events therefore have no direct delivery path.

Use libdatadog's shared local-proxy discovery and HTTPS fallback for both streams in agentless mode. Direct delivery uses DD_API_KEY and the intake derived from DD_SITE. A custom CDN URL never becomes the authenticated event destination. Missing credentials or an invalid site retain local-proxy discovery. Remote Config keeps its existing Agent-only route. Runtime test-session changes preserve the destination of batches already handed to the sidecar. PHP-buffered exposures and deferred userland observations use the session active at handoff, not necessarily the session active during evaluation.

The native transport fixture now reads concurrent HTTP requests without blocking on idle sockets. It retains the route, identity, credential, payload, and duplicate checks.

Validation

Tested commit	Validation	Result
`DataDog/dd-trace-php@a122cbc2ee47551e911de695ea2d45d0e1f4dc99`	System tests: direct and serverless-relay exposures and evaluation counts, explicit CDN configuration; Apache/PHP 8.2; `system-tests@7f153aa229a1ce67559bb47acf1fb3986c1be6af`	5 passed, 0 failed, 0 skipped, 1 deselected; development-version override
`DataDog/dd-trace-php@a122cbc2ee47551e911de695ea2d45d0e1f4dc99`	System tests: direct-EVP shutdown delivery	Not run; disabled in the PHP manifest. PHP-FPM was not tested.

Reviewer checklist

Test coverage seems ok.
Appropriate labels assigned.

datadog-datadog-prod-us1 · 2026-11-12T23:19:18Z

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 30 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | System Tests: [INTEGRATIONS] — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

❌ tests.integrations.test_db_integrations_sql.Test_MySql.test_span_kind[apache-mod-8.0] from system_tests_suite

KeyError: &#39;span.kind&#39;

self = &lt;tests.integrations.test_db_integrations_sql.Test_MySql object at 0x7f17d6b951c0&gt;
excluded_operations = ()

    def test_span_kind(self, excluded_operations: tuple[str, ...] = ()):
        &#34;&#34;&#34;Describes the relationship between the Span, its parents, and its children in a Trace.&#34;&#34;&#34;
    
        for _, span_meta in self.get_spans_meta(excluded_operations):
&gt;           assert span_meta[&#34;span.kind&#34;] == &#34;client&#34;
...

❌ tests.integrations.test_db_integrations_sql.Test_Postgres.test_span_kind[apache-mod-8.0] from system_tests_suite

KeyError: &#39;span.kind&#39;

self = &lt;tests.integrations.test_db_integrations_sql.Test_Postgres object at 0x7f17d6b815b0&gt;
excluded_operations = ()

    def test_span_kind(self, excluded_operations: tuple[str, ...] = ()):
        &#34;&#34;&#34;Describes the relationship between the Span, its parents, and its children in a Trace.&#34;&#34;&#34;
    
        for _, span_meta in self.get_spans_meta(excluded_operations):
&gt;           assert span_meta[&#34;span.kind&#34;] == &#34;client&#34;
...

DataDog/apm-reliability/dd-trace-php | System Tests: [parametric] 1/2 — ❌ 21 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

❌

tests.parametric.test_otel_api_interoperability.Test_Otel_API_Interoperability.test_distributed_headers_are_propagated_tracecontext[library_env0, parametric-...

from system_tests_suite

KeyError: &#39;_dd.p.dm&#39;

self = &lt;tests.parametric.test_otel_api_interoperability.Test_Otel_API_Interoperability object at 0x7f8a1550b770&gt;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f8a126c6330&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f8a072a5160&gt;

    def test_distributed_headers_are_propagated_tracecontext(self, test_agent: TestAgentAPI, test_library: APMLibrary):
        &#34;&#34;&#34;- Test that distributed tracecontext headers are propagated across APIs&#34;&#34;&#34;
        trace_id = &#34;0000000000000000000000000000002a&#34;  # 42
        parent_id = &#34;0000000000000003&#34;  # 3
...

❌

tests.parametric.test_otel_api_interoperability.Test_Otel_API_Interoperability.test_set_attribute_from_otel[library_env0, parametric-php]

from system_tests_suite

KeyError: &#39;bool&#39;

self = &lt;tests.parametric.test_otel_api_interoperability.Test_Otel_API_Interoperability object at 0x7f8a1550b560&gt;
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f8a126c6330&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f8a072a8cb0&gt;

    def test_set_attribute_from_otel(self, test_agent: TestAgentAPI, test_library: APMLibrary):
        &#34;&#34;&#34;- Test that attributes can be set on a Datadog span using the OTel API&#34;&#34;&#34;
        with test_library, test_library.dd_start_span(&#34;dd_span&#34;) as dd_span:
            otel_span = test_library.otel_current_span()
...

↳ and 19 more — View all

DataDog/apm-reliability/dd-trace-php | System Tests: [parametric] 2/2 — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

❌

tests.parametric.test_tracer.Test_TracerSCITagging.test_tracer_commit_sha_environment_variable[library_env0, parametric-php]

from system_tests_suite

AssertionError: assert 2 == 1
 &#43;  where 2 = len([{&#39;duration&#39;: 135243795, &#39;meta&#39;: {&#39;_dd.git.commit.sha&#39;: &#39;aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa&#39;, &#39;_dd.p.tid&#39;: &#39;6af8...aries,runtime.sapi:cli,svc.auto:server.php&#39;}, &#39;metrics&#39;: {&#39;_sampling_priority_v1&#39;: 1}, &#39;name&#39;: &#39;operation.child&#39;, ...}])

self = &lt;tests.parametric.test_tracer.Test_TracerSCITagging object at 0x7f6c2b73ffe0&gt;
library_env = {&#39;DD_GIT_COMMIT_SHA&#39;: &#39;aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa&#39;}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f6c28b33560&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f6c1db9ab40&gt;

    @parametrize(&#34;library_env&#34;, [{&#34;DD_GIT_COMMIT_SHA&#34;: &#34;aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa&#34;}])
    def test_tracer_commit_sha_environment_variable(
...

❌

tests.parametric.test_tracer.Test_TracerSCITagging.test_tracer_repository_url_environment_variable[library_env0, parametric-php]

from system_tests_suite

AssertionError: assert 2 == 1
 &#43;  where 2 = len([{&#39;duration&#39;: 135857858, &#39;meta&#39;: {&#39;_dd.git.repository_url&#39;: &#39;https://github.com/DataDog/dd-trace-go&#39;, &#39;_dd.p.tid&#39;: &#39;6a...aries,runtime.sapi:cli,svc.auto:server.php&#39;}, &#39;metrics&#39;: {&#39;_sampling_priority_v1&#39;: 1}, &#39;name&#39;: &#39;operation.child&#39;, ...}])

self = &lt;tests.parametric.test_tracer.Test_TracerSCITagging object at 0x7f6c2b73c530&gt;
library_env = {&#39;DD_GIT_REPOSITORY_URL&#39;: &#39;https://github.com/DataDog/dd-trace-go&#39;}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f6c28b33560&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f6c1db99400&gt;

    @parametrize(&#34;library_env&#34;, [{&#34;DD_GIT_REPOSITORY_URL&#34;: &#34;https://github.com/DataDog/dd-trace-go&#34;}])
    def test_tracer_repository_url_environment_variable(
...

View all 30 failed jobs.

⚠️ Warnings

❄️ 10 New flaky tests detected

FFE agentless discovers a shared local EVP route while Remote Config keeps EVP v2 from PHP.tmp.build_extension.tests.ext.ffe

007&#43; 
008&#43; Fatal error: Uncaught RuntimeException: remote-config: missing event stream; received exposures/transport-session; child output: delivered
007- remote-config/false: expected streams delivered with PHP identity, token and no API key
008- remote-config/true: expected streams delivered with PHP identity, token and no API key
009&#43;  in tmp/build_extension/tests/ext/ffe/stubs/agentless_evp_transport.inc:102
010&#43; Stack trace:
011&#43; #0 tmp/build_extension/tests/ext/ffe/agentless_evp_transport.php(3): require()
012&#43; #1 {main}
013&#43;   thrown in tmp/build_extension/tests/ext/ffe/stubs/agentless_evp_transport.inc on line 102

FFE keeps local EVP discovery when managed CDN setup fails from PHP.tmp.build_extension.tests.ext.ffe

001&#43; Fatal error: Uncaught RuntimeException: managed-no-key: missing event stream; received ; child output: delivered
002&#43;  in tmp/build_extension/tests/ext/ffe/stubs/agentless_evp_transport.inc:102
003&#43; Stack trace:
004&#43; #0 tmp/build_extension/tests/ext/ffe/agentless_evp_managed_failure.php(3): require()
001- managed-no-key/false: expected streams delivered with PHP identity, token and no API key
002- managed-no-key/true: expected streams delivered with PHP identity, token and no API key
003- managed-invalid-site/false: expected streams delivered with PHP identity, token and no API key
004- managed-invalid-site/true: expected streams delivered with PHP identity, token and no API key
005&#43; #1 {main}
006&#43;   thrown in tmp/build_extension/tests/ext/ffe/stubs/agentless_evp_transport.inc on line 102

View in Flaky Test Management

ℹ️ Info

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 80.12% (-0.01%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: a98560d | Docs | View more details | Give us feedback!}

pr-commenter · 2026-11-13T00:26:36Z

Benchmarks [ tracer ]

Benchmark execution time: 2026-11-15 00:05:24

Comparing candidate commit a98560d in PR branch leo/php-agentless-evp-fallback-20261112 with baseline commit 14d2bef in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 2 performance regressions! Performance is the same for 216 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:SamplingRuleMatchingBench/benchGlobMatching4-opcache

🟥 execution_time [+61.698ns; +212.902ns] or [+2.044%; +7.054%]

scenario:SamplingRuleMatchingBench/benchRegexMatching4

🟥 execution_time [+35.306ns; +115.494ns] or [+2.383%; +7.796%]

chatgpt-codex-connector · 2026-11-13T01:28:33Z

Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-13T01:29:46.694309Z	`ec428b1`	Draft marked ready
🔒 Security Review	✅ Completed 2026-11-13T01:33:49.969778Z	`ec428b1`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

datadog-datadog-prod-us1

More details

The authenticated fallback remains separate from custom CDN URLs, and static review raised no reportable concerns about routing or test-token propagation. Native delivery could not be independently validated because required runtime dependencies were unavailable.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit ec428b1}

vjfridge

Approach looks good. I posted some p2 comments for you to consider and looks like CI needs iteration, but I trust you'll take care of those!

…-agentless-evp-fallback-20261112 * 'master' of github.com:DataDog/dd-trace-php: feat(tracer): send traces as V1 payloads through the libdatadog builder FFI (#4453) Move profiling storage and export into SSI common (#4455)

…ndows Valgrind: the sidecar builds its rustls root store lazily inside the first timed /info probe. Under valgrind that alone takes several seconds, so the 5s DD_TRACE_AGENT_TIMEOUT expired, no EVP route was found and the flagevaluation batch was dropped. Raise the child's agent timeout when running under valgrind. Windows: every subprocess sidecar is spawned from the fixed trampoline %TEMP%\datadog-ipc-helper-<session>, with a single ".old" slot for a still-running predecessor. The test's own shared sidecar plus the previous child's lingering per-process sidecar occupy both slots, so the second child failed with ERROR_FILE_EXISTS (os error 80). Give each child a private TMP/TEMP directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fix(ffe): deliver agentless events directly when no Agent is available

7aecaff

leoromanovsky mentioned this pull request Nov 12, 2026

test(ffe): enable PHP agentless EVP egress coverage DataDog/system-tests#8476

Draft

leoromanovsky added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 12, 2026

leoromanovsky mentioned this pull request Nov 13, 2026

test(ffe): enable PHP CDN coverage in existing agentless scenario DataDog/system-tests#8475

Open

leoromanovsky removed the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 13, 2026

fix(ffe): preserve updated EVP tokens and wait for sidecar delivery

ec428b1

leoromanovsky marked this pull request as ready for review November 13, 2026 01:28

leoromanovsky requested review from a team as code owners November 13, 2026 01:28

leoromanovsky requested a review from bwoebi November 13, 2026 01:28

datadog-datadog-prod-us1 Bot reviewed Nov 13, 2026

View reviewed changes

leoromanovsky requested review from sameerank and vjfridge November 13, 2026 01:32

leoromanovsky mentioned this pull request Nov 13, 2026

[DOCS-16420] Make agentless the primary server Feature Flags setup DataDog/documentation#41720

Open

1 task

test(ffe): serve concurrent EVP requests without blocking

a122cbc

vjfridge reviewed Nov 13, 2026

View reviewed changes

Comment thread ext/sidecar.c Resolved

vjfridge reviewed Nov 13, 2026

View reviewed changes

Comment thread ext/sidecar.c Resolved

vjfridge reviewed Nov 13, 2026

View reviewed changes

Comment thread ext/sidecar.c Resolved

vjfridge approved these changes Nov 13, 2026

View reviewed changes

leoromanovsky added 2 commits November 13, 2026 18:09

Merge latest master into PHP agentless EVP fallback

cde25b7

Fix FFE local discovery across agentless setup and tracing modes

4984b2f

leoromanovsky added crucible: reviewing A Crucible review has started; this PR has reviewer attention. and removed crucible: reviewing A Crucible review has started; this PR has reviewer attention. labels Nov 13, 2026

leoromanovsky requested a review from vjfridge November 13, 2026 21:52

Merge branch 'master' into leo/php-agentless-evp-fallback-20261112

b85fc8d

leoromanovsky enabled auto-merge November 14, 2026 00:46

bwoebi and others added 2 commits November 14, 2026 21:49

bwoebi approved these changes Nov 15, 2026

View reviewed changes

bwoebi disabled auto-merge November 15, 2026 01:46

bwoebi merged commit b1c7b41 into master Nov 15, 2026
2326 of 2342 checks passed

bwoebi deleted the leo/php-agentless-evp-fallback-20261112 branch November 15, 2026 01:46

github-actions Bot added this to the 1.28.0 milestone Nov 15, 2026

Navigation Menu

Repository navigation

fix(ffe): fall back to direct EVP intake in agentless mode - #4489

fix(ffe): fall back to direct EVP intake in agentless mode#4489
bwoebi merged 8 commits into
masterfrom
leo/php-agentless-evp-fallback-20261112

leoromanovsky commented Nov 12, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Nov 12, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

pr-commenter Bot commented Nov 13, 2026 •

edited

Loading

Explanation

More details about the CI and significant changes

Uh oh!

chatgpt-codex-connector Bot commented Nov 13, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vjfridge left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

leoromanovsky commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation

Reviewer checklist

Uh oh!

datadog-datadog-prod-us1 Bot commented Nov 12, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

⚠️ Warnings

ℹ️ Info

Uh oh!

pr-commenter Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks [ tracer ]

Explanation

More details about the CI and significant changes

scenario:SamplingRuleMatchingBench/benchGlobMatching4-opcache

scenario:SamplingRuleMatchingBench/benchRegexMatching4

Uh oh!

chatgpt-codex-connector Bot commented Nov 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vjfridge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leoromanovsky commented Nov 12, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Nov 12, 2026 •

edited by datadog-official Bot

Loading

pr-commenter Bot commented Nov 13, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 13, 2026 •

edited

Loading