Commit cf23e14
authored
feat: Increase trace payload limit from 2 MiB to 50 MiB (#902)
# This PR
Change the trace request limit from 2 MiB to 50 MiB.
# Motivation
When the Node.js tracer layer sends a request to Lambda extension that's
between 2 MiB and 50 MiB, the extension closes the HTTP connection, the
tracer gets an `EPIPE` error and breaks. (Maybe the tracer should handle
the error better, but that's out of scope of this PR.)
According to @rochdev:
> the agent is supposed to have a limit of 50mb
So let's change the limit on agent side to match the expectation.
# Testing
Tested with Node.js 22 Lambda with this handler:
```
import tracer from 'dd-trace';
import crypto from 'crypto';
tracer.init();
function randomGarbage(len) {
// low-compressibility payload (random bytes -> base64)
return crypto.randomBytes(len).toString('base64');
}
export const handler = async (event) => {
const SPANS = 3000;
const TAG_BYTES_PER_SPAN = 20_000; // ~20 KB per span tag (base64 expands a bit)
const root = tracer.startSpan('repro.root');
root.setTag('dd.repro', 'true');
for (let i = 0; i < SPANS; i++) {
console.log(`Sending the ${i}-th span`);
const span = tracer.startSpan('repro.child', { childOf: root });
span.setTag('blob', randomGarbage(TAG_BYTES_PER_SPAN));
span.finish();
}
root.finish();
const response = {
statusCode: 200,
body: JSON.stringify('Hello from Lambda!'),
};
return response;
};
```
### Before:
There are errors like:
```
Error: write EPIPE
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:95:16)
at WriteWrap.callbackTrampoline (node:internal/async_hooks:130:17)
```
```
LAMBDA_RUNTIME Failed to post handler success response. Http response code: 403. {"errorMessage":"State transition from Ready to InvocationErrorResponse failed for runtime. Error: State transition is not allowed","errorType":"InvalidStateTransition"}
```
### After
When Lambda's memory is 1024 MB, the error no longer happens.
When Lambda's memory is 512 MB, the invocation can fail due to OOM. But
I think that's a legit error. We can ask customers to increase memory
limit for high-volume workload like this.
# Notes
cc @astuyve who set a `MAX_CONTENT_LENGTH` of 10 MiB in
#294. This PR
increases it to 50 MiB as well.
Thanks @dougqh @duncanista @lucaspimentel @rochdev for discussion.
#899
Jira: https://datadoghq.atlassian.net/browse/SVLS-77771 parent c5ee4fc commit cf23e14
3 files changed
+13
-2
lines changedSome generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| 52 | + | |
52 | 53 | | |
53 | 54 | | |
54 | 55 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3 | 3 | | |
4 | 4 | | |
5 | 5 | | |
6 | | - | |
| 6 | + | |
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
72 | 73 | | |
73 | 74 | | |
74 | 75 | | |
75 | | - | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
76 | 79 | | |
77 | 80 | | |
78 | 81 | | |
| |||
231 | 234 | | |
232 | 235 | | |
233 | 236 | | |
| 237 | + | |
234 | 238 | | |
235 | 239 | | |
236 | 240 | | |
237 | 241 | | |
| 242 | + | |
238 | 243 | | |
239 | 244 | | |
240 | 245 | | |
| |||
254 | 259 | | |
255 | 260 | | |
256 | 261 | | |
| 262 | + | |
257 | 263 | | |
258 | 264 | | |
259 | 265 | | |
| |||
264 | 270 | | |
265 | 271 | | |
266 | 272 | | |
| 273 | + | |
| 274 | + | |
267 | 275 | | |
268 | 276 | | |
269 | 277 | | |
| |||
0 commit comments