Add telemetry for the RUM injector #9267

sarahchen6 · 2025-07-28T21:10:33Z

What Does This Do

Add telemetry for the RUM injector. Gather injection metrics and tags, and send the data via StatsDClient. The common metrics are based on: https://github.com/DataDog/dd-go/blob/prod/trace/apps/tracer-telemetry-intake/telemetry-metrics/static/common_metrics.json

Motivation

Complete the RUM Injector requirements (doc)

Additional Notes

This telemetry implementation does not include the following metrics:

injection.initialization.failed: Initialization failures can be due to a multitude of reasons and are already covered by SSI telemetry
injection.installation: Installation metrics are related to web server installers and not captured by this RUM injector implementation
injection.installation.duration: Installation metrics are related to web server installers and not captured by this RUM injector implementation

It also does not include the injector_version tag for the injection.failed metric. This tag refers to the Rust injector and we do not implement that.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: https://datadoghq.atlassian.net/browse/LANGPLAT-679

pr-commenter · 2025-07-28T21:48:07Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/implement-rum-injector-telemetry
git_commit_date	1755014538	1755113222
git_commit_sha	`ee43e5f`	`b20d6c4`
release_version	1.53.0-SNAPSHOT~ee43e5f19c	1.53.0-SNAPSHOT~b20d6c46fd

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1755115080	1755115080
ci_job_id	1079407590	1079407590
ci_pipeline_id	73607980	73607980
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-993xm0b5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-993xm0b5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 11 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.055 s) : 0, 1054650
Total [baseline] (10.687 s) : 0, 10687413
Agent [candidate] (1.045 s) : 0, 1045383
Total [candidate] (10.742 s) : 0, 10742399
section appsec
Agent [baseline] (1.217 s) : 0, 1216792
Total [baseline] (10.718 s) : 0, 10717964
Agent [candidate] (1.232 s) : 0, 1231966
Total [candidate] (10.764 s) : 0, 10763844
section iast
Agent [baseline] (1.174 s) : 0, 1173743
Total [baseline] (10.849 s) : 0, 10849107
Agent [candidate] (1.185 s) : 0, 1184506
Total [candidate] (10.937 s) : 0, 10936520
section profiling
Agent [baseline] (1.186 s) : 0, 1186306
Total [baseline] (10.787 s) : 0, 10786629
Agent [candidate] (1.192 s) : 0, 1191734
Total [candidate] (10.953 s) : 0, 10953049

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.055 s	-
Agent	appsec	1.217 s	162.142 ms (15.4%)
Agent	iast	1.174 s	119.092 ms (11.3%)
Agent	profiling	1.186 s	131.656 ms (12.5%)
Total	tracing	10.687 s	-
Total	appsec	10.718 s	30.551 ms (0.3%)
Total	iast	10.849 s	161.694 ms (1.5%)
Total	profiling	10.787 s	99.217 ms (0.9%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	appsec	1.232 s	186.582 ms (17.8%)
Agent	iast	1.185 s	139.122 ms (13.3%)
Agent	profiling	1.192 s	146.351 ms (14.0%)
Total	tracing	10.742 s	-
Total	appsec	10.764 s	21.445 ms (0.2%)
Total	iast	10.937 s	194.121 ms (1.8%)
Total	profiling	10.953 s	210.65 ms (2.0%)

gantt
    title petclinic - break down per module: candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.44 ms) : 0, 1440
crashtracking [candidate] (1.433 ms) : 0, 1433
BytebuddyAgent [baseline] (737.692 ms) : 0, 737692
BytebuddyAgent [candidate] (730.588 ms) : 0, 730588
GlobalTracer [baseline] (243.637 ms) : 0, 243637
GlobalTracer [candidate] (241.585 ms) : 0, 241585
AppSec [baseline] (30.394 ms) : 0, 30394
AppSec [candidate] (30.002 ms) : 0, 30002
Debugger [baseline] (6.046 ms) : 0, 6046
Debugger [candidate] (5.985 ms) : 0, 5985
Remote Config [baseline] (658.345 µs) : 0, 658
Remote Config [candidate] (643.924 µs) : 0, 644
Telemetry [baseline] (13.664 ms) : 0, 13664
Telemetry [candidate] (14.235 ms) : 0, 14235
section appsec
crashtracking [baseline] (1.432 ms) : 0, 1432
crashtracking [candidate] (1.444 ms) : 0, 1444
BytebuddyAgent [baseline] (751.64 ms) : 0, 751640
BytebuddyAgent [candidate] (760.952 ms) : 0, 760952
GlobalTracer [baseline] (233.924 ms) : 0, 233924
GlobalTracer [candidate] (237.68 ms) : 0, 237680
IAST [baseline] (23.458 ms) : 0, 23458
IAST [candidate] (23.979 ms) : 0, 23979
AppSec [baseline] (168.398 ms) : 0, 168398
AppSec [candidate] (169.553 ms) : 0, 169553
Debugger [baseline] (7.164 ms) : 0, 7164
Debugger [candidate] (7.958 ms) : 0, 7958
Remote Config [baseline] (610.426 µs) : 0, 610
Remote Config [candidate] (616.165 µs) : 0, 616
Telemetry [baseline] (9.102 ms) : 0, 9102
Telemetry [candidate] (8.42 ms) : 0, 8420
section iast
crashtracking [baseline] (1.422 ms) : 0, 1422
crashtracking [candidate] (1.436 ms) : 0, 1436
BytebuddyAgent [baseline] (847.038 ms) : 0, 847038
BytebuddyAgent [candidate] (855.691 ms) : 0, 855691
GlobalTracer [baseline] (232.145 ms) : 0, 232145
GlobalTracer [candidate] (233.646 ms) : 0, 233646
IAST [baseline] (29.247 ms) : 0, 29247
IAST [candidate] (31.038 ms) : 0, 31038
AppSec [baseline] (27.456 ms) : 0, 27456
AppSec [candidate] (25.965 ms) : 0, 25965
Debugger [baseline] (6.569 ms) : 0, 6569
Debugger [candidate] (6.62 ms) : 0, 6620
Remote Config [baseline] (582.626 µs) : 0, 583
Remote Config [candidate] (586.427 µs) : 0, 586
Telemetry [baseline] (8.303 ms) : 0, 8303
Telemetry [candidate] (8.416 ms) : 0, 8416
section profiling
crashtracking [baseline] (1.392 ms) : 0, 1392
crashtracking [candidate] (1.402 ms) : 0, 1402
BytebuddyAgent [baseline] (755.532 ms) : 0, 755532
BytebuddyAgent [candidate] (758.655 ms) : 0, 758655
GlobalTracer [baseline] (220.61 ms) : 0, 220610
GlobalTracer [candidate] (221.366 ms) : 0, 221366
AppSec [baseline] (29.721 ms) : 0, 29721
AppSec [candidate] (29.849 ms) : 0, 29849
Debugger [baseline] (6.224 ms) : 0, 6224
Debugger [candidate] (6.274 ms) : 0, 6274
Remote Config [baseline] (673.184 µs) : 0, 673
Remote Config [candidate] (678.145 µs) : 0, 678
Telemetry [baseline] (15.937 ms) : 0, 15937
Telemetry [candidate] (16.057 ms) : 0, 16057
ProfilingAgent [baseline] (106.989 ms) : 0, 106989
ProfilingAgent [candidate] (108.072 ms) : 0, 108072
Profiling [baseline] (107.6 ms) : 0, 107600
Profiling [candidate] (108.75 ms) : 0, 108750

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.052 s) : 0, 1051900
Total [baseline] (8.585 s) : 0, 8584945
Agent [candidate] (1.047 s) : 0, 1047334
Total [candidate] (8.562 s) : 0, 8562292
section iast
Agent [baseline] (1.173 s) : 0, 1173129
Total [baseline] (9.305 s) : 0, 9304556
Agent [candidate] (1.191 s) : 0, 1191152
Total [candidate] (9.315 s) : 0, 9314554

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.052 s	-
Agent	iast	1.173 s	121.228 ms (11.5%)
Total	tracing	8.585 s	-
Total	iast	9.305 s	719.61 ms (8.4%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.047 s	-
Agent	iast	1.191 s	143.818 ms (13.7%)
Total	tracing	8.562 s	-
Total	iast	9.315 s	752.262 ms (8.8%)

gantt
    title insecure-bank - break down per module: candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.439 ms) : 0, 1439
crashtracking [candidate] (1.419 ms) : 0, 1419
BytebuddyAgent [baseline] (735.882 ms) : 0, 735882
BytebuddyAgent [candidate] (734.157 ms) : 0, 734157
GlobalTracer [baseline] (243.13 ms) : 0, 243130
GlobalTracer [candidate] (242.36 ms) : 0, 242360
AppSec [baseline] (30.332 ms) : 0, 30332
AppSec [candidate] (30.308 ms) : 0, 30308
Debugger [baseline] (6.041 ms) : 0, 6041
Debugger [candidate] (6.041 ms) : 0, 6041
Remote Config [baseline] (646.409 µs) : 0, 646
Remote Config [candidate] (655.073 µs) : 0, 655
Telemetry [baseline] (13.428 ms) : 0, 13428
Telemetry [candidate] (11.48 ms) : 0, 11480
section iast
crashtracking [baseline] (1.421 ms) : 0, 1421
crashtracking [candidate] (1.438 ms) : 0, 1438
BytebuddyAgent [baseline] (846.977 ms) : 0, 846977
BytebuddyAgent [candidate] (861.739 ms) : 0, 861739
GlobalTracer [baseline] (231.932 ms) : 0, 231932
GlobalTracer [candidate] (233.969 ms) : 0, 233969
IAST [baseline] (27.564 ms) : 0, 27564
IAST [candidate] (30.42 ms) : 0, 30420
AppSec [baseline] (29.057 ms) : 0, 29057
AppSec [candidate] (27.749 ms) : 0, 27749
Debugger [baseline] (6.519 ms) : 0, 6519
Debugger [candidate] (5.768 ms) : 0, 5768
Remote Config [baseline] (581.206 µs) : 0, 581
Remote Config [candidate] (597.654 µs) : 0, 598
Telemetry [baseline] (8.244 ms) : 0, 8244
Telemetry [candidate] (8.376 ms) : 0, 8376

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/implement-rum-injector-telemetry
git_commit_date	1755014538	1755113222
git_commit_sha	`ee43e5f`	`b20d6c4`
release_version	1.53.0-SNAPSHOT~ee43e5f19c	1.53.0-SNAPSHOT~b20d6c46fd

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1755114836	1755114836
ci_job_id	1079407591	1079407591
ci_pipeline_id	73607980	73607980
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-76zhsek2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-76zhsek2 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 3 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling:high_load	worse [+4.021ms; +4.971ms] or [+8.842%; +10.929%]	unstable [-16.306op/s; -2.069op/s] or [-15.856%; -2.012%]	49.978ms	93.650op/s	45.482ms	102.838op/s
scenario:load:petclinic:tracing:high_load	worse [+1.603ms; +2.419ms] or [+3.724%; +5.620%]	unstable [-12.532op/s; +2.732op/s] or [-11.522%; +2.512%]	45.043ms	103.862op/s	43.032ms	108.763op/s
scenario:load:petclinic:appsec:high_load	worse [+2.944ms; +3.871ms] or [+6.338%; +8.332%]	unstable [-14.054op/s; +0.304op/s] or [-13.949%; +0.302%]	49.862ms	93.875op/s	46.455ms	100.750op/s

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.445 ms) : 4389, 4500
.   : milestone, 4445,
iast (9.394 ms) : 9240, 9547
.   : milestone, 9394,
iast_FULL (13.728 ms) : 13456, 13999
.   : milestone, 13728,
iast_GLOBAL (10.447 ms) : 10260, 10635
.   : milestone, 10447,
profiling (8.85 ms) : 8717, 8983
.   : milestone, 8850,
tracing (7.792 ms) : 7666, 7918
.   : milestone, 7792,
section candidate
no_agent (4.36 ms) : 4309, 4411
.   : milestone, 4360,
iast (9.383 ms) : 9232, 9535
.   : milestone, 9383,
iast_FULL (14.138 ms) : 13855, 14422
.   : milestone, 14138,
iast_GLOBAL (10.493 ms) : 10301, 10685
.   : milestone, 10493,
profiling (9.087 ms) : 8938, 9236
.   : milestone, 9087,
tracing (7.624 ms) : 7509, 7740
.   : milestone, 7624,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.445 ms [4.389 ms, 4.5 ms]	-
iast	9.394 ms [9.24 ms, 9.547 ms]	4.949 ms (111.3%)
iast_FULL	13.728 ms [13.456 ms, 13.999 ms]	9.283 ms (208.8%)
iast_GLOBAL	10.447 ms [10.26 ms, 10.635 ms]	6.003 ms (135.0%)
profiling	8.85 ms [8.717 ms, 8.983 ms]	4.405 ms (99.1%)
tracing	7.792 ms [7.666 ms, 7.918 ms]	3.347 ms (75.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.36 ms [4.309 ms, 4.411 ms]	-
iast	9.383 ms [9.232 ms, 9.535 ms]	5.023 ms (115.2%)
iast_FULL	14.138 ms [13.855 ms, 14.422 ms]	9.778 ms (224.3%)
iast_GLOBAL	10.493 ms [10.301 ms, 10.685 ms]	6.133 ms (140.7%)
profiling	9.087 ms [8.938 ms, 9.236 ms]	4.727 ms (108.4%)
tracing	7.624 ms [7.509 ms, 7.74 ms]	3.264 ms (74.9%)

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c
    dateFormat X
    axisFormat %s
section baseline
no_agent (36.59 ms) : 36295, 36886
.   : milestone, 36590,
appsec (46.455 ms) : 46048, 46861
.   : milestone, 46455,
code_origins (44.618 ms) : 44251, 44985
.   : milestone, 44618,
iast (45.27 ms) : 44887, 45653
.   : milestone, 45270,
profiling (45.482 ms) : 45079, 45885
.   : milestone, 45482,
tracing (43.032 ms) : 42648, 43417
.   : milestone, 43032,
section candidate
no_agent (37.247 ms) : 36947, 37548
.   : milestone, 37247,
appsec (49.862 ms) : 49409, 50315
.   : milestone, 49862,
code_origins (45.438 ms) : 45018, 45858
.   : milestone, 45438,
iast (45.303 ms) : 44907, 45699
.   : milestone, 45303,
profiling (49.978 ms) : 49502, 50455
.   : milestone, 49978,
tracing (45.043 ms) : 44669, 45417
.   : milestone, 45043,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	36.59 ms [36.295 ms, 36.886 ms]	-
appsec	46.455 ms [46.048 ms, 46.861 ms]	9.864 ms (27.0%)
code_origins	44.618 ms [44.251 ms, 44.985 ms]	8.028 ms (21.9%)
iast	45.27 ms [44.887 ms, 45.653 ms]	8.68 ms (23.7%)
profiling	45.482 ms [45.079 ms, 45.885 ms]	8.892 ms (24.3%)
tracing	43.032 ms [42.648 ms, 43.417 ms]	6.442 ms (17.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.247 ms [36.947 ms, 37.548 ms]	-
appsec	49.862 ms [49.409 ms, 50.315 ms]	12.614 ms (33.9%)
code_origins	45.438 ms [45.018 ms, 45.858 ms]	8.19 ms (22.0%)
iast	45.303 ms [44.907 ms, 45.699 ms]	8.056 ms (21.6%)
profiling	49.978 ms [49.502 ms, 50.455 ms]	12.731 ms (34.2%)
tracing	45.043 ms [44.669 ms, 45.417 ms]	7.795 ms (20.9%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	sarahchen6/implement-rum-injector-telemetry
git_commit_date	1755014538	1755113222
git_commit_sha	`ee43e5f`	`b20d6c4`
release_version	1.53.0-SNAPSHOT~ee43e5f19c	1.53.0-SNAPSHOT~b20d6c46fd

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1755115303	1755115303
ci_job_id	1079407592	1079407592
ci_pipeline_id	73607980	73607980
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-8mzf5rwp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-8mzf5rwp 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (3.57 ms) : 3357, 3783
.   : milestone, 3570,
iast (2.192 ms) : 2129, 2254
.   : milestone, 2192,
iast_GLOBAL (2.242 ms) : 2179, 2304
.   : milestone, 2242,
profiling (2.056 ms) : 2004, 2108
.   : milestone, 2056,
tracing (2.001 ms) : 1952, 2049
.   : milestone, 2001,
section candidate
no_agent (1.472 ms) : 1461, 1484
.   : milestone, 1472,
appsec (3.592 ms) : 3381, 3804
.   : milestone, 3592,
iast (2.188 ms) : 2126, 2250
.   : milestone, 2188,
iast_GLOBAL (2.233 ms) : 2170, 2295
.   : milestone, 2233,
profiling (2.058 ms) : 2006, 2110
.   : milestone, 2058,
tracing (2.007 ms) : 1958, 2056
.   : milestone, 2007,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.459 ms, 1.482 ms]	-
appsec	3.57 ms [3.357 ms, 3.783 ms]	2.1 ms (142.9%)
iast	2.192 ms [2.129 ms, 2.254 ms]	721.815 µs (49.1%)
iast_GLOBAL	2.242 ms [2.179 ms, 2.304 ms]	771.519 µs (52.5%)
profiling	2.056 ms [2.004 ms, 2.108 ms]	585.845 µs (39.9%)
tracing	2.001 ms [1.952 ms, 2.049 ms]	530.825 µs (36.1%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.472 ms [1.461 ms, 1.484 ms]	-
appsec	3.592 ms [3.381 ms, 3.804 ms]	2.12 ms (144.0%)
iast	2.188 ms [2.126 ms, 2.25 ms]	715.899 µs (48.6%)
iast_GLOBAL	2.233 ms [2.17 ms, 2.295 ms]	760.392 µs (51.6%)
profiling	2.058 ms [2.006 ms, 2.11 ms]	585.391 µs (39.8%)
tracing	2.007 ms [1.958 ms, 2.056 ms]	534.936 µs (36.3%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.53.0-SNAPSHOT~b20d6c46fd, baseline=1.53.0-SNAPSHOT~ee43e5f19c
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.947 s) : 14947000, 14947000
.   : milestone, 14947000,
appsec (14.871 s) : 14871000, 14871000
.   : milestone, 14871000,
iast (18.269 s) : 18269000, 18269000
.   : milestone, 18269000,
iast_GLOBAL (18.172 s) : 18172000, 18172000
.   : milestone, 18172000,
profiling (15.306 s) : 15306000, 15306000
.   : milestone, 15306000,
tracing (14.844 s) : 14844000, 14844000
.   : milestone, 14844000,
section candidate
no_agent (14.984 s) : 14984000, 14984000
.   : milestone, 14984000,
appsec (14.838 s) : 14838000, 14838000
.   : milestone, 14838000,
iast (18.69 s) : 18690000, 18690000
.   : milestone, 18690000,
iast_GLOBAL (17.72 s) : 17720000, 17720000
.   : milestone, 17720000,
profiling (15.774 s) : 15774000, 15774000
.   : milestone, 15774000,
tracing (14.884 s) : 14884000, 14884000
.   : milestone, 14884000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.947 s [14.947 s, 14.947 s]	-
appsec	14.871 s [14.871 s, 14.871 s]	-76.0 ms (-0.5%)
iast	18.269 s [18.269 s, 18.269 s]	3.322 s (22.2%)
iast_GLOBAL	18.172 s [18.172 s, 18.172 s]	3.225 s (21.6%)
profiling	15.306 s [15.306 s, 15.306 s]	359.0 ms (2.4%)
tracing	14.844 s [14.844 s, 14.844 s]	-103.0 ms (-0.7%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.984 s [14.984 s, 14.984 s]	-
appsec	14.838 s [14.838 s, 14.838 s]	-146.0 ms (-1.0%)
iast	18.69 s [18.69 s, 18.69 s]	3.706 s (24.7%)
iast_GLOBAL	17.72 s [17.72 s, 17.72 s]	2.736 s (18.3%)
profiling	15.774 s [15.774 s, 15.774 s]	790.0 ms (5.3%)
tracing	14.884 s [14.884 s, 14.884 s]	-100.0 ms (-0.7%)

datadog-datadog-prod-us1 · 2025-07-31T15:08:33Z

Code coverage: total 57.43%, base diff 0.10%, patch 94.07% (view details)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: b20d6c4 | Docs | Was this helpful? Give us feedback!}

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorMetrics.java

sarahchen6 · 2025-08-09T20:10:15Z

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorMetrics.java

+        + "\ncontentSecurityPolicyDetected="
+        + contentSecurityPolicyDetected.get();
+  }
+}


Unlike TracerHealthMetrics, the RumInjectorMetrics are not sent via periodic flushes. I thought this was okay because RUM injection events are much less frequent than span events. There is also more context-specific tagging for RumInjectorMetrics, so flushing these events would introduce a lot of complexity for little benefit. It seems like the StatsDClient already provides aggregation and batching as well (ref).

PerfectSlayer

I only did a preliminary review but looking good so far.
I will have a look at testing tomorrow and check how this run.
Sharing the first comments now :)

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

internal-api/src/main/java/datadog/trace/api/rum/RumInjector.java

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorMetrics.java

internal-api/src/main/java/datadog/trace/api/rum/RumTelemetryCollector.java

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java

PerfectSlayer

Very nice testing! 👏
I only few comments about minor improvements and readability.

...rc/test/groovy/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeWriterTest.groovy

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

...t/groovy/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStreamTest.groovy

...t/instrumentation/servlet/request-3/src/test/groovy/RumHttpServletResponseWrapperTest.groovy

internal-api/src/test/groovy/datadog/trace/api/rum/RumInjectorMetricsTest.groovy

internal-api/src/test/groovy/datadog/trace/api/rum/RumInjectorTest.groovy

PerfectSlayer

Thanks for all the follow up changes!
Nice work, especially around testing 👏

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

sarahchen6 added type: feature request comp: metrics Metrics labels Jul 28, 2025

sarahchen6 force-pushed the sarahchen6/implement-rum-injector-telemetry branch 3 times, most recently from 1e2a4ed to 0687a05 Compare July 31, 2025 14:45

sarahchen6 force-pushed the sarahchen6/implement-rum-injector-telemetry branch from dbf1bee to 87c7127 Compare August 2, 2025 03:25

PerfectSlayer added comp: rum Realtime User Monitoring and removed comp: metrics Metrics labels Aug 8, 2025

sarahchen6 added 16 commits August 8, 2025 10:47

Add RumInjectorHealthMetrics

d19d27e

Add telemetry collector and methods to RumInjector

e01d1a9

Initialize health metrics and telemetry collector

8c71ef6

Get injectionsucceed count

6c01e10

Add comments

ceb3e11

Reorganize classes

92f0c54

Connect rum injector, telemetry collector, and statsdclient

c5c860f

Add tests

3d4ac53

Get and test metrics for injection failures and skips

3fd498d

Add Content Security Policy and HTTP response size telemetry

14c338b

Add injection duration telemetry

c5b5389

Fix some things

feb40be

Fix content-length retrieval and add test for injection timing

23e0455

Add injection initialization success telemetry

07db174

Fix CoreTracer compilation with InstrumenterConfig

a5352ff

Add tags to all metrics

2a1f676

sarahchen6 force-pushed the sarahchen6/implement-rum-injector-telemetry branch from 7bd7a48 to 2a1f676 Compare August 8, 2025 19:21

sarahchen6 added 3 commits August 8, 2025 16:42

Update InjectingPipeOutputStreamTest

b32874c

Tweaks

7a5cd68

Address jacoco coverage and injectingpipeoutstream interface updates

412a3f8

sarahchen6 commented Aug 9, 2025

View reviewed changes

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java Show resolved Hide resolved

sarahchen6 commented Aug 9, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorMetrics.java Show resolved Hide resolved

sarahchen6 commented Aug 9, 2025

View reviewed changes

Add content-length detection for InjectingPipeWriter and improve tests

b3fcde4

PerfectSlayer reviewed Aug 11, 2025

View reviewed changes

Address review comments

4b2f6b3

PerfectSlayer reviewed Aug 12, 2025

View reviewed changes

sarahchen6 added 2 commits August 12, 2025 14:35

Fix header retrieval

3e462d4

Add lots of improvements from review comments

b1ab871

PerfectSlayer approved these changes Aug 13, 2025

View reviewed changes

sarahchen6 changed the title ~~[WIP] Add telemetry for the RUM injector~~ Add telemetry for the RUM injector Aug 13, 2025

sarahchen6 marked this pull request as ready for review August 13, 2025 16:27

sarahchen6 requested review from a team as code owners August 13, 2025 16:27

sarahchen6 requested a review from amarziali August 13, 2025 16:27

sarahchen6 added 2 commits August 13, 2025 12:55

Fix constructors and address review comment

93bd0a4

Merge branch 'master' into sarahchen6/implement-rum-injector-telemetry

5a24d7d

AlexeyKuznetsov-DD reviewed Aug 13, 2025

View reviewed changes

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java Outdated Show resolved Hide resolved

Clarify bytes written and address review comment

b20d6c4

AlexeyKuznetsov-DD approved these changes Aug 13, 2025

View reviewed changes

Add telemetry for the RUM injector #9267

Are you sure you want to change the base?

Add telemetry for the RUM injector #9267

Conversation

sarahchen6 commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

pr-commenter bot commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

datadog-datadog-prod-us1 bot commented Jul 31, 2025 • edited by datadog-official bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sarahchen6 Aug 9, 2025

Choose a reason for hiding this comment

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

PerfectSlayer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sarahchen6 commented Jul 28, 2025 •

edited

Loading

pr-commenter bot commented Jul 28, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Jul 31, 2025 •

edited by datadog-official bot

Loading