Fix AWS API Gateway endpoints correlation HTTP span tags - Inferred Proxy Spans#10561
Open
Fix AWS API Gateway endpoints correlation HTTP span tags - Inferred Proxy Spans#10561
Conversation
jandro996
added
tag: do not merge
jandro996
force-pushed
the
alejandro.gonzalez/RFC-1081
branch
from
7163ef9 to
85a368c
Compare
jandro996
force-pushed
the
alejandro.gonzalez/RFC-1081
branch
from
c681b6c to
c8a926f
Compare
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 62 metrics, 9 unstable metrics. Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.068 s) : 0, 1067592
Total [baseline] (10.913 s) : 0, 10912509
Agent [candidate] (1.068 s) : 0, 1067974
Total [candidate] (10.852 s) : 0, 10851653
section appsec
Agent [baseline] (1.263 s) : 0, 1262964
Total [baseline] (11.077 s) : 0, 11076732
Agent [candidate] (1.251 s) : 0, 1250825
Total [candidate] (11.239 s) : 0, 11239102
section iast
Agent [baseline] (1.23 s) : 0, 1230348
Total [baseline] (11.287 s) : 0, 11286765
Agent [candidate] (1.229 s) : 0, 1229304
Total [candidate] (11.213 s) : 0, 11213093
section profiling
Agent [baseline] (1.202 s) : 0, 1201947
Total [baseline] (11.036 s) : 0, 11035904
Agent [candidate] (1.201 s) : 0, 1200971
Total [candidate] (11.093 s) : 0, 11093482
gantt
title petclinic - break down per module: candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.206 ms) : 0, 1206
crashtracking [candidate] (1.194 ms) : 0, 1194
BytebuddyAgent [baseline] (628.086 ms) : 0, 628086
BytebuddyAgent [candidate] (628.749 ms) : 0, 628749
AgentMeter [baseline] (29.169 ms) : 0, 29169
AgentMeter [candidate] (29.116 ms) : 0, 29116
GlobalTracer [baseline] (258.178 ms) : 0, 258178
GlobalTracer [candidate] (257.814 ms) : 0, 257814
AppSec [baseline] (33.033 ms) : 0, 33033
AppSec [candidate] (33.018 ms) : 0, 33018
Debugger [baseline] (66.775 ms) : 0, 66775
Debugger [candidate] (67.559 ms) : 0, 67559
Remote Config [baseline] (622.342 µs) : 0, 622
Remote Config [candidate] (621.584 µs) : 0, 622
Telemetry [baseline] (10.625 ms) : 0, 10625
Telemetry [candidate] (9.941 ms) : 0, 9941
Flare Poller [baseline] (3.809 ms) : 0, 3809
Flare Poller [candidate] (3.771 ms) : 0, 3771
section appsec
crashtracking [baseline] (1.225 ms) : 0, 1225
crashtracking [candidate] (1.202 ms) : 0, 1202
BytebuddyAgent [baseline] (671.15 ms) : 0, 671150
BytebuddyAgent [candidate] (663.277 ms) : 0, 663277
AgentMeter [baseline] (12.149 ms) : 0, 12149
AgentMeter [candidate] (12.147 ms) : 0, 12147
GlobalTracer [baseline] (262.924 ms) : 0, 262924
GlobalTracer [candidate] (261.376 ms) : 0, 261376
IAST [baseline] (26.277 ms) : 0, 26277
IAST [candidate] (25.918 ms) : 0, 25918
AppSec [baseline] (169.981 ms) : 0, 169981
AppSec [candidate] (169.526 ms) : 0, 169526
Debugger [baseline] (68.284 ms) : 0, 68284
Debugger [candidate] (67.184 ms) : 0, 67184
Remote Config [baseline] (682.304 µs) : 0, 682
Remote Config [candidate] (653.739 µs) : 0, 654
Telemetry [baseline] (9.943 ms) : 0, 9943
Telemetry [candidate] (9.537 ms) : 0, 9537
Flare Poller [baseline] (3.882 ms) : 0, 3882
Flare Poller [candidate] (3.74 ms) : 0, 3740
section iast
crashtracking [baseline] (1.183 ms) : 0, 1183
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (794.878 ms) : 0, 794878
BytebuddyAgent [candidate] (793.929 ms) : 0, 793929
AgentMeter [baseline] (11.269 ms) : 0, 11269
AgentMeter [candidate] (11.239 ms) : 0, 11239
GlobalTracer [baseline] (247.27 ms) : 0, 247270
GlobalTracer [candidate] (247.136 ms) : 0, 247136
IAST [baseline] (27.076 ms) : 0, 27076
IAST [candidate] (26.847 ms) : 0, 26847
AppSec [baseline] (34.661 ms) : 0, 34661
AppSec [candidate] (32.291 ms) : 0, 32291
Debugger [baseline] (65.383 ms) : 0, 65383
Debugger [candidate] (68.104 ms) : 0, 68104
Remote Config [baseline] (532.754 µs) : 0, 533
Remote Config [candidate] (547.167 µs) : 0, 547
Telemetry [baseline] (8.689 ms) : 0, 8689
Telemetry [candidate] (8.574 ms) : 0, 8574
Flare Poller [baseline] (3.454 ms) : 0, 3454
Flare Poller [candidate] (3.436 ms) : 0, 3436
section profiling
crashtracking [baseline] (1.226 ms) : 0, 1226
crashtracking [candidate] (1.182 ms) : 0, 1182
BytebuddyAgent [baseline] (688.17 ms) : 0, 688170
BytebuddyAgent [candidate] (688.611 ms) : 0, 688611
AgentMeter [baseline] (8.642 ms) : 0, 8642
AgentMeter [candidate] (8.609 ms) : 0, 8609
GlobalTracer [baseline] (217.745 ms) : 0, 217745
GlobalTracer [candidate] (217.464 ms) : 0, 217464
AppSec [baseline] (32.73 ms) : 0, 32730
AppSec [candidate] (32.833 ms) : 0, 32833
Debugger [baseline] (68.173 ms) : 0, 68173
Debugger [candidate] (67.61 ms) : 0, 67610
Remote Config [baseline] (643.144 µs) : 0, 643
Remote Config [candidate] (643.163 µs) : 0, 643
Telemetry [baseline] (9.253 ms) : 0, 9253
Telemetry [candidate] (9.22 ms) : 0, 9220
Flare Poller [baseline] (3.808 ms) : 0, 3808
Flare Poller [candidate] (3.781 ms) : 0, 3781
ProfilingAgent [baseline] (100.472 ms) : 0, 100472
ProfilingAgent [candidate] (99.994 ms) : 0, 99994
Profiling [baseline] (101.045 ms) : 0, 101045
Profiling [candidate] (100.574 ms) : 0, 100574
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.079 s) : 0, 1079371
Total [baseline] (8.771 s) : 0, 8770757
Agent [candidate] (1.063 s) : 0, 1062846
Total [candidate] (8.74 s) : 0, 8740033
section iast
Agent [baseline] (1.232 s) : 0, 1232188
Total [baseline] (9.431 s) : 0, 9430660
Agent [candidate] (1.229 s) : 0, 1228914
Total [candidate] (9.418 s) : 0, 9417686
gantt
title insecure-bank - break down per module: candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.208 ms) : 0, 1208
crashtracking [candidate] (1.184 ms) : 0, 1184
BytebuddyAgent [baseline] (636.137 ms) : 0, 636137
BytebuddyAgent [candidate] (627.034 ms) : 0, 627034
AgentMeter [baseline] (29.475 ms) : 0, 29475
AgentMeter [candidate] (29.009 ms) : 0, 29009
GlobalTracer [baseline] (260.288 ms) : 0, 260288
GlobalTracer [candidate] (257.311 ms) : 0, 257311
AppSec [baseline] (33.66 ms) : 0, 33660
AppSec [candidate] (32.924 ms) : 0, 32924
Debugger [baseline] (64.706 ms) : 0, 64706
Debugger [candidate] (62.834 ms) : 0, 62834
Remote Config [baseline] (634.691 µs) : 0, 635
Remote Config [candidate] (633.744 µs) : 0, 634
Telemetry [baseline] (12.252 ms) : 0, 12252
Telemetry [candidate] (11.445 ms) : 0, 11445
Flare Poller [baseline] (4.657 ms) : 0, 4657
Flare Poller [candidate] (4.39 ms) : 0, 4390
section iast
crashtracking [baseline] (1.195 ms) : 0, 1195
crashtracking [candidate] (1.195 ms) : 0, 1195
BytebuddyAgent [baseline] (795.504 ms) : 0, 795504
BytebuddyAgent [candidate] (794.189 ms) : 0, 794189
AgentMeter [baseline] (11.349 ms) : 0, 11349
AgentMeter [candidate] (11.29 ms) : 0, 11290
GlobalTracer [baseline] (248.542 ms) : 0, 248542
GlobalTracer [candidate] (247.763 ms) : 0, 247763
IAST [baseline] (27.069 ms) : 0, 27069
IAST [candidate] (27.13 ms) : 0, 27130
AppSec [baseline] (33.314 ms) : 0, 33314
AppSec [candidate] (33.753 ms) : 0, 33753
Debugger [baseline] (66.487 ms) : 0, 66487
Debugger [candidate] (65.158 ms) : 0, 65158
Remote Config [baseline] (535.085 µs) : 0, 535
Remote Config [candidate] (531.764 µs) : 0, 532
Telemetry [baseline] (8.651 ms) : 0, 8651
Telemetry [candidate] (8.59 ms) : 0, 8590
Flare Poller [baseline] (3.455 ms) : 0, 3455
Flare Poller [candidate] (3.389 ms) : 0, 3389
LoadParameters
See matching parameters
SummaryFound 1 performance improvements and 1 performance regressions! Performance is the same for 18 metrics, 16 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (1.255 ms) : 1242, 1268
. : milestone, 1255,
iast (3.18 ms) : 3136, 3224
. : milestone, 3180,
iast_FULL (5.809 ms) : 5751, 5866
. : milestone, 5809,
iast_GLOBAL (3.582 ms) : 3530, 3634
. : milestone, 3582,
profiling (2.105 ms) : 2086, 2124
. : milestone, 2105,
tracing (1.78 ms) : 1765, 1795
. : milestone, 1780,
section candidate
no_agent (1.18 ms) : 1168, 1192
. : milestone, 1180,
iast (3.106 ms) : 3068, 3143
. : milestone, 3106,
iast_FULL (5.967 ms) : 5906, 6027
. : milestone, 5967,
iast_GLOBAL (3.672 ms) : 3603, 3741
. : milestone, 3672,
profiling (2.138 ms) : 2118, 2157
. : milestone, 2138,
tracing (1.783 ms) : 1768, 1798
. : milestone, 1783,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (19.109 ms) : 18915, 19303
. : milestone, 19109,
appsec (18.535 ms) : 18347, 18723
. : milestone, 18535,
code_origins (17.607 ms) : 17433, 17782
. : milestone, 17607,
iast (17.416 ms) : 17244, 17589
. : milestone, 17416,
profiling (19.263 ms) : 19073, 19454
. : milestone, 19263,
tracing (17.701 ms) : 17528, 17874
. : milestone, 17701,
section candidate
no_agent (18.166 ms) : 17973, 18359
. : milestone, 18166,
appsec (18.314 ms) : 18126, 18503
. : milestone, 18314,
code_origins (18.307 ms) : 18123, 18490
. : milestone, 18307,
iast (17.742 ms) : 17565, 17920
. : milestone, 17742,
profiling (18.681 ms) : 18494, 18868
. : milestone, 18681,
tracing (17.489 ms) : 17316, 17663
. : milestone, 17489,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (1.478 ms) : 1467, 1490
. : milestone, 1478,
appsec (3.82 ms) : 3597, 4043
. : milestone, 3820,
iast (2.25 ms) : 2181, 2318
. : milestone, 2250,
iast_GLOBAL (2.305 ms) : 2235, 2375
. : milestone, 2305,
profiling (2.088 ms) : 2033, 2142
. : milestone, 2088,
tracing (2.074 ms) : 2021, 2128
. : milestone, 2074,
section candidate
no_agent (1.48 ms) : 1469, 1492
. : milestone, 1480,
appsec (3.789 ms) : 3567, 4011
. : milestone, 3789,
iast (2.26 ms) : 2191, 2329
. : milestone, 2260,
iast_GLOBAL (2.305 ms) : 2235, 2374
. : milestone, 2305,
profiling (2.117 ms) : 2060, 2174
. : milestone, 2117,
tracing (2.062 ms) : 2008, 2115
. : milestone, 2062,
Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.60.0-SNAPSHOT~a49044a437, baseline=1.60.0-SNAPSHOT~5418feb23a
dateFormat X
axisFormat %s
section baseline
no_agent (14.964 s) : 14964000, 14964000
. : milestone, 14964000,
appsec (14.694 s) : 14694000, 14694000
. : milestone, 14694000,
iast (18.205 s) : 18205000, 18205000
. : milestone, 18205000,
iast_GLOBAL (17.968 s) : 17968000, 17968000
. : milestone, 17968000,
profiling (14.691 s) : 14691000, 14691000
. : milestone, 14691000,
tracing (14.715 s) : 14715000, 14715000
. : milestone, 14715000,
section candidate
no_agent (14.876 s) : 14876000, 14876000
. : milestone, 14876000,
appsec (14.66 s) : 14660000, 14660000
. : milestone, 14660000,
iast (18.471 s) : 18471000, 18471000
. : milestone, 18471000,
iast_GLOBAL (17.978 s) : 17978000, 17978000
. : milestone, 17978000,
profiling (15.386 s) : 15386000, 15386000
. : milestone, 15386000,
tracing (14.647 s) : 14647000, 14647000
. : milestone, 14647000,
|
jandro996
force-pushed
the
alejandro.gonzalez/rfc-1076
branch
from
35abdf1 to
9e4cfe6
Compare
jandro996
requested review from
daniel-romano-DD,
manuel-alvarez-alvarez and
smola
Contributor
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 06ba1cf584
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
internal-api/src/main/java/datadog/trace/api/gateway/InferredProxySpan.java
Outdated
Show resolved
Hide resolved
internal-api/src/main/java/datadog/trace/api/gateway/InferredProxySpan.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 883f1cbd9c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
internal-api/src/main/java/datadog/trace/api/gateway/InferredProxySpan.java
Show resolved
Hide resolved
internal-api/src/main/java/datadog/trace/api/gateway/InferredProxySpan.java
Outdated
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR implements standardized tags for inferred proxy spans produced by the Java tracer when instrumenting AWS API Gateway (v1 REST and v2 HTTP APIs). The changes align proxy spans with the cross-platform contract defined in RFC-1081 for endpoint discovery and correlation.
Mandatory tags implemented:
aws.httpapifor v2 HTTP API,aws.apigatewayfor v1 REST (previously only v1 was supported)span.kind: set toserverhttp.url: prefixed withhttps://scheme (was missing, causing backend parsing issues)http.route: populated from newx-dd-proxy-resource-pathheader (resource template, e.g./users/{id})resource.name: uses<Method> <Route>whenhttp.routeis available, falls back to<Method> <Path>Optional tags (set only when the corresponding header is present):
account_idfromx-dd-proxy-account-idapiidfromx-dd-proxy-api-idregionfromx-dd-proxy-regiondd_resource_key: computed ARN (arn:aws:apigateway:{region}::/restapis/{api-id}or.../apis/{api-id})AppSec Tag Propagation
RFC-1081 requires
_dd.appsec.enabledand_dd.appsec.jsonon the inferred proxy span so security activity can be correlated with the API Gateway endpoint.GatewayBridge.onRequestEndedsets AppSec tags on the service-entry span (where detection occurs).InferredProxySpan.copyAppSecTagsFromServiceEntry(), called fromfinish(), then copies only those two RFC-required tags to the inferred proxy span — keeping the propagation logic encapsulated inInferredProxySpan.traceSeg.setTagTop()is still used for_dd.asm.keepand_dd.propagated_trace_source, as sampling decisions require these on the root span immediately.Motivation
This implementation is required by RFC-1081: Endpoint Discovery & Correlation from Inferred Spans
This PR covers the Inferred Proxy Spans portion of the RFC. The Inferred Lambda Spans portion will be addressed in a separate PR #10576.
Additional Notes
ST checked locally
aws_userexcluded: intentionally omitted per RFC guidance due to PII concerns (assumed-role session names may contain user identifiers). Implementation requires explicit approval.AppSec Bug Fix
When an inferred proxy span is present, it becomes the local root of the trace.
HttpServerDecorator.onRequestEndForInstrumentationGateway()was guarded byif (span.getLocalRootSpan() != span) return, which caused it to silently skip the service-entry span — meaning no AppSec tags were set on any span in this scenario.The fix relaxes the guard to also allow execution when the local root is an inferred proxy span (detected via the
_dd.inferred_spantag).Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: APPSEC-61198
Note: Once your PR is ready to merge, add it to the merge queue by commenting
/merge./merge -ccancels the queue request./merge -f --reason "reason"skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see this doc.