Commit 6d68aad
authored
Fix for sending duplicate logs when using Agentless Logging in Azure Function host (#7383)
## Summary of changes
Disables agentless logging (for the Azure Functions Host Process) if we
detect that we are instrumenting an Azure Functions host process.
Previously this would cause us to duplicate every log sent from the
worker process.
This bug fix can be reverted by setting the new configuration key
`DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED` to `true`.
## Reason for change
If customers are running with the Isolated Azure Function model we will
be instrumenting two applications, one is the function host process the
other is the worker process. If they have direct log submission enabled
the function host ends up duplicating the logs from the function process
which results in us shipping two nearly identical logs.
This behavior isn't ideal as the duplicate log is likely not valuable so
we've disabled agentless logging in the azure functions host process.
## Implementation details
Added `IsRunningInAzureFunctionsHost()` to `EnvironmentHelpers.cs` which
allows for a rough detection of whether we are running on the function
host using the following logic:
- Is `FUNCTIONS_WORKER_RUNTIME` present AND set to `dotnet-isolated`?
- Are both `--functions-worker-id` or `--workerId` NOT present in the
command line?
If both are true we treat that scenario as being running in the function
host - otherwise we are likely the worker process
I wasn't able to find a more robust way of checking, but when looking at
the various log output that I had the `--functions-worker-id` or
`--workerId` seemed to always be called by the function host.
```
[2025-10-03T16:01:42.901Z] Reading functions metadata (Worker)
[2025-10-03T16:01:47.176Z] {
[2025-10-03T16:01:47.177Z] "ProcessId": 71080,
[2025-10-03T16:01:47.178Z] "RuntimeIdentifier": "win-x64",
[2025-10-03T16:01:47.179Z] "WorkerVersion": "2.0.0.0",
[2025-10-03T16:01:47.180Z] "ProductVersion": "2.0.0\u002Bd8b5fe998a8c92819b8ee41d2569d2525413e9c5",
[2025-10-03T16:01:47.181Z] "FrameworkDescription": ".NET 9.0.9",
[2025-10-03T16:01:47.182Z] "OSDescription": "Microsoft Windows 10.0.26100",
[2025-10-03T16:01:47.183Z] "OSArchitecture": "X64",
[2025-10-03T16:01:47.184Z] "CommandLine": "C:\\Users\\steven.bouwkamp\\source\\repos\\dd-trace-dotnet\\artifacts\\bin\\Samples.AzureFunctions.V4Isolated.AspNetCore\\debug_net9.0\\Samples.AzureFunctions.V4Isolated.AspNetCore.dll --host 127.0.0.1 --port 65401 --workerId e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --requestId 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --grpcMaxMessageLength 2147483647 --functions-uri http://127.0.0.1:65401/ --functions-worker-id e94d23fd-cd3c-4780-a3e3-4980d7b0f644 --functions-request-id 6dba68ac-1954-466a-aeb4-9570cc9b12c2 --functions-grpc-max-message-length 2147483647"
[2025-10-03T16:01:47.185Z] }
```
- Added `DD_LOGS_DIRECT_SUBMISSION_AZURE_FUNCTIONS_HOST_ENABLED` that
defaults to `false` to disable the duplicate logs from being sent.
## Test coverage
- Added a new test project and test for having host logs enabled /
disabled.
- The reason why I added a new test project instead of re-using an
existing one was because when I re-ran the Function application in our
tests multiple times the `func.exe` would fail to obtain a lock and
would need to wait some period of time to for recovery after each
subsequent run of the same application. I think this is because we end
the `func.exe` process with a `Kill()`. Making a new project wasn't
ideal but was a quick and simple workaround.
## Other details
Fixes SLES-2364
<!-- 1 parent 232217e commit 6d68aad
File tree
24 files changed
+1092
-34
lines changed- tracer
- src/Datadog.Trace
- Configuration
- Logging/DirectSubmission
- Util
- test
- Datadog.Trace.ClrProfiler.IntegrationTests
- Datadog.Trace.Tests/Telemetry
- snapshots
- test-applications/azure-functions
- Samples.AzureFunctions.V4Isolated.HostLogsDisabled
- Properties
- Samples.AzureFunctions.V4Isolated
- Properties
24 files changed
+1092
-34
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
439 | 439 | | |
440 | 440 | | |
441 | 441 | | |
| 442 | + | |
| 443 | + | |
442 | 444 | | |
443 | 445 | | |
444 | 446 | | |
| |||
1057 | 1059 | | |
1058 | 1060 | | |
1059 | 1061 | | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
1060 | 1066 | | |
1061 | 1067 | | |
1062 | 1068 | | |
| |||
1232 | 1238 | | |
1233 | 1239 | | |
1234 | 1240 | | |
| 1241 | + | |
1235 | 1242 | | |
1236 | 1243 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
617 | 617 | | |
618 | 618 | | |
619 | 619 | | |
| 620 | + | |
| 621 | + | |
620 | 622 | | |
621 | 623 | | |
622 | 624 | | |
| |||
1489 | 1491 | | |
1490 | 1492 | | |
1491 | 1493 | | |
| 1494 | + | |
| 1495 | + | |
| 1496 | + | |
| 1497 | + | |
1492 | 1498 | | |
1493 | 1499 | | |
1494 | 1500 | | |
| |||
1731 | 1737 | | |
1732 | 1738 | | |
1733 | 1739 | | |
| 1740 | + | |
1734 | 1741 | | |
1735 | 1742 | | |
1736 | 1743 | | |
| |||
1857 | 1864 | | |
1858 | 1865 | | |
1859 | 1866 | | |
| 1867 | + | |
1860 | 1868 | | |
1861 | 1869 | | |
1862 | 1870 | | |
| |||
Lines changed: 8 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| |||
79 | 79 | | |
80 | 80 | | |
81 | 81 | | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
82 | 89 | | |
83 | 90 | | |
84 | 91 | | |
Lines changed: 17 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
98 | 99 | | |
99 | 100 | | |
100 | 101 | | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
101 | 105 | | |
102 | 106 | | |
103 | 107 | | |
| |||
139 | 143 | | |
140 | 144 | | |
141 | 145 | | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
142 | 153 | | |
143 | 154 | | |
144 | 155 | | |
| |||
230 | 241 | | |
231 | 242 | | |
232 | 243 | | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
233 | 250 | | |
234 | 251 | | |
235 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
133 | 159 | | |
134 | 160 | | |
135 | 161 | | |
| |||
Lines changed: 65 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
278 | 278 | | |
279 | 279 | | |
280 | 280 | | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
281 | 289 | | |
282 | 290 | | |
283 | 291 | | |
| |||
286 | 294 | | |
287 | 295 | | |
288 | 296 | | |
289 | | - | |
290 | 297 | | |
291 | 298 | | |
292 | 299 | | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
293 | 355 | | |
294 | 356 | | |
295 | 357 | | |
| |||
325 | 387 | | |
326 | 388 | | |
327 | 389 | | |
| 390 | + | |
| 391 | + | |
328 | 392 | | |
329 | 393 | | |
330 | 394 | | |
| |||
Lines changed: 1 addition & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
528 | 528 | | |
529 | 529 | | |
530 | 530 | | |
| 531 | + | |
531 | 532 | | |
532 | 533 | | |
533 | 534 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
317 | 317 | | |
318 | 318 | | |
319 | 319 | | |
320 | | - | |
| 320 | + | |
321 | 321 | | |
322 | 322 | | |
323 | 323 | | |
| |||
470 | 470 | | |
471 | 471 | | |
472 | 472 | | |
473 | | - | |
| 473 | + | |
474 | 474 | | |
475 | 475 | | |
476 | 476 | | |
| |||
Lines changed: 2 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
317 | 317 | | |
318 | 318 | | |
319 | 319 | | |
320 | | - | |
| 320 | + | |
321 | 321 | | |
322 | 322 | | |
323 | 323 | | |
| |||
469 | 469 | | |
470 | 470 | | |
471 | 471 | | |
472 | | - | |
| 472 | + | |
473 | 473 | | |
474 | 474 | | |
475 | 475 | | |
| |||
0 commit comments