aws-serverless + SST: LocalVariables integration / its usage of Node Inspector kills lambda functions and floods Sentry issues

[steffen-mueller]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/aws-serverless

SDK Version

8.41.0

Framework Version

No response

Link to Sentry event

https://addcraft.sentry.io/issues/5987124956

Reproduction Example/SDK Setup

We've set up Sentry according to the NPM Package - ESM method in the docs. We are using SST v3.3.30.

Sentry is setup in a sentry.ts file:

/**
 * This file initializes Sentry. It has to be an own file so it is loaded before any other module.
 * The loading happens via NODE_OPTIONS in the infra/Helpers.ts file.
 * 
 * For more information, see https://docs.sentry.io/platforms/javascript/guides/aws-lambda/install/esm-npm/
 * This file is compiled into config/sentry-loader.js by running `npm run build-sentry-loader`.
 * This is done automatically on npm postinstall event.
 */

import * as Sentry from "@sentry/aws-serverless";

console.log("Initializing Sentry for DSN "+process.env.SENTRY_DSN);

Sentry.init({
    enabled: !!process.env.SENTRY_DSN,
    dsn: process.env.SENTRY_DSN,
    release: process.env.PLUGIN_VERSION || "0.1.0-SNAPSHOT",
    environment: process.env.PRESENTER_STAGE || "local",

    enableTracing: true,
    
    // Keep it on 1.0 on prod for now
    tracesSampleRate: 1.0,

    // Profiling Sample Rate 1 for now
    profilesSampleRate: 1.0,

    includeLocalVariables: true
});

As described in the code comment above, we compile this .ts file to .js on build time:

const esbuild = require('esbuild');

esbuild.build({
    entryPoints: ['src/sentry.ts'],
    bundle: true,
    platform: 'node',
    target: 'node18',
    outfile: 'config/sentry-loader.js',
    format: 'cjs'
});

The created sentry-loader.js is referenced in the lambda definition of the SST endpoint:

export const backendApi = new sst.aws.ApiGatewayV2("BackendApi", {
    domain: "api.myapp.com"
});

backendApi.route("$default", {
    handler: "src/index.handler",
    runtime: "nodejs20.x", // This was "nodejs18.x" before, see actual result for details
    timeout: "30 seconds",
    copyFiles: [
        { from: "./config" },
        { from: "./src/views" }
    ],
    environment: {
        DEPLOYED_REGION: aws.getRegionOutput().name,
        NODE_ENV: process.env.NODE_ENV || "development",
        PLUGIN_VERSION: process.env.PLUGIN_VERSION || "0.0.0",

        // Sentry configuration
        SENTRY_DSN: process.env.SENTRY_DSN || "",
        NODE_OPTIONS: "--import ./config/sentry-loader.js",
        SENTRY_TRACES_SAMPLE_RATE: "1.0"
    }
    link: [
        // Some boring dynamodb tables etc.
    ]
});

Steps to Reproduce

Setup a SST project with a sentry ESM loader as explained above and deploy to production.

Expected Result

Sentry logs errors and causes no problems.

Actual Result

The results differ between runtime versions (I assume - we've upgraded the runtime version along with a big bunch of NPM packages, including Sentry, going from 8.33.1 to 8.41.0, and I don't want to use the production environment to isolate the problem :)).

With runtime "nodejs20.x":

The lambda function does not start up at all, resulting in HTTP 503s after 30 seconds. The CloudWatch logs say:

Initializing Sentry for DSN https://[redacted]@[redacted].ingest.us.sentry.io/[redacted]
Debugger listening on ws://127.0.0.1:39651/3c7e36e4-ec77-4846-a745-7f5f5759e85f
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:37035/481827cc-e29d-420a-9d45-8bb57a07511b
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:36793/4cb1457e-a94c-406f-8006-80bddf296d28
For help, see: https://nodejs.org/en/docs/inspector
[... some dozens more ...]
Debugger listening on ws://127.0.0.1:38873/d5df3f49-ee27-41b7-b8dd-09ef21cb3c42
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:38959/5191a315-bc8e-493e-b196-e2cdafded6d1
For help, see: https://nodejs.org/en/docs/inspector
Debugger listening on ws://127.0.0.1:45543/100f3533-56d2-476c-babc-b19a2a203d3a
For help, see: https://nodejs.org/en/docs/inspector
INIT_REPORT Init Duration: 30000.64 ms	Phase: invoke	Status: timeout

With runtime "nodejs18.x"

The application works as expected, but the Sentry issues are flooded with thousands (!) of issues affecting 0 users looking like that:

{
  "event_id": "d3234489202f43dca677175e8bf3fd5d",
  "project": [redacted],
  "release": "[redacted]",
  "dist": null,
  "platform": "node",
  "message": "",
  "datetime": "2024-11-27T16:38:44+00:00",
  "tags": [
    [
      "environment",
      "production"
    ],
    [
      "handled",
      "no"
    ],
    [
      "level",
      "error"
    ],
    [
      "mechanism",
      "onunhandledrejection"
    ],
    [
      "os.name",
      "Linux"
    ],
    [
      "runtime",
      "node v20.18.0"
    ],
    [
      "runtime.name",
      "node"
    ],
    [
      "release",
      "[redacted]"
    ],
    [
      "server_name",
      "[redacted]"
    ]
  ],
  "_dsc": {
    "environment": "[redacted]",
    "public_key": "[redacted]",
    "release": "[redacted]",
    "replay_id": null,
    "trace_id": "[redacted]",
    "transaction": null
  },
  "_metrics": {
    "bytes.ingested.event": 2792,
    "bytes.stored.event": 5297
  },
  "contexts": {
    "app": {
      "app_start_time": "2024-11-27T16:38:28.640Z",
      "app_memory": 582651904,
      "free_memory": 525627392,
      "type": "app"
    },
    "cloud_resource": {
      "cloud.platform": "AWS_Lambda_nodejs20.x",
      "cloud.provider": "aws",
      "cloud.region": "eu-central-1",
      "type": "cloud_resource"
    },
    "culture": {
      "locale": "en-US",
      "timezone": "UTC",
      "type": "culture"
    },
    "device": {
      "arch": "x64",
      "memory_size": 1249615872,
      "free_memory": 525627392,
      "boot_time": "2024-11-27T16:36:44.205Z",
      "processor_count": 2,
      "cpu_description": "Intel(R) Xeon(R) Processor @ 2.50GHz",
      "processor_frequency": 0,
      "type": "device"
    },
    "os": {
      "name": "Linux",
      "kernel_version": "5.10.227-239.884.amzn2.x86_64",
      "type": "os"
    },
    "runtime": {
      "runtime": "node v20.18.0",
      "name": "node",
      "version": "v20.18.0",
      "type": "runtime"
    },
    "trace": {
      "trace_id": "[redacted]",
      "span_id": "b194994bab6ec9c4",
      "status": "unknown",
      "type": "trace"
    }
  },
  "culprit": "#onMessage(node:inspector)",
  "environment": "production-de",
  "exception": {
    "values": [
      {
        "type": "Error",
        "value": "Inspector error -32000: Could not find object with given id",
        "stacktrace": {
          "frames": [
            {
              "function": "AsyncResource.runMicrotask",
              "module": "task_queues",
              "filename": "node:internal/process/task_queues",
              "abs_path": "node:internal/process/task_queues",
              "lineno": 137,
              "colno": 8,
              "in_app": false
            },
            {
              "function": "AsyncResource.runInAsyncScope",
              "module": "node:async_hooks",
              "filename": "node:async_hooks",
              "abs_path": "node:async_hooks",
              "lineno": 206,
              "colno": 9,
              "in_app": false
            },
            {
              "module": "task_queues",
              "filename": "node:internal/process/task_queues",
              "abs_path": "node:internal/process/task_queues",
              "lineno": 140,
              "colno": 7,
              "in_app": false
            },
            {
              "module": "node:inspector",
              "filename": "node:inspector",
              "abs_path": "node:inspector",
              "lineno": 75,
              "colno": 58,
              "in_app": false
            },
            {
              "function": "#onMessage",
              "module": "node:inspector",
              "filename": "node:inspector",
              "abs_path": "node:inspector",
              "lineno": 87,
              "colno": 15,
              "in_app": false
            }
          ]
        },
        "mechanism": {
          "type": "onunhandledrejection",
          "handled": false
        }
      }
    ]
  },
  "extra": {
    "unhandledPromiseRejection": true
  },
  "fingerprint": [
    "{{ default }}"
  ],
  "grouping_config": {
    "enhancements": "KLUv_SAYwQAAkwKRs25ld3N0eWxlOjIwMjMtMDEtMTGQ",
    "id": "newstyle:2023-01-11"
  },
  "hashes": [
    "[redacted]"
  ],
  "ingest_path": [
    {
      "version": "24.11.1",
      "public_key": "[redacted]"
    }
  ],
  "key_id": "3799414",
  "level": "error",
  "location": "node:inspector",
  "logger": "",
  "metadata": {
    "filename": "node:inspector",
    "function": "#onMessage",
    "in_app_frame_mix": "system-only",
    "type": "Error",
    "value": "Inspector error -32000: Could not find object with given id"
  },
  "nodestore_insert": 1732725526.58189,
  "received": 1732725524.25593,
  "sdk": {
    "name": "sentry.javascript.aws-serverless",
    "version": "8.33.1",
    "integrations": [
      "InboundFilters",
      "FunctionToString",
      "LinkedErrors",
      "RequestData",
      "Console",
      "Http",
      "NodeFetch",
      "OnUncaughtException",
      "OnUnhandledRejection",
      "ContextLines",
      "LocalVariablesAsync",
      "Context",
      "Modules",
      "Aws",
      "AwsLambda",
      "AWSLambda"
    ],
    "packages": [
      {
        "name": "npm:@sentry/aws-serverless",
        "version": "8.33.1"
      }
    ]
  },
  "timestamp": 1732725524.164,
  "title": "Error: Inspector error -32000: Could not find object with given id",
  "type": "error",
  "version": "7"
}

Workaround

Disabling includeLocalVariables ends the pain - the lambda loads as expected again.

[Lms24]

Hi, this looks indeed like a problem with our localVariables integration or probably with Node's inspector API. For the moment it's probably easiest to leave includeLocalVariables disabled.

I will tag @timfish - did you by any chance test local variables in lambda environments? I think this might very well be unexplored territory otherwise.

[timfish]

I don't know why this doesn't work with lambda but I suspect that it's because there are differences to plain Node.

Can you attach a debugger remotely to aws lambda? It would be very tricky/laborious to debug this if you can't!

[jonmast]

I spent some time debugging, and I believe this is because the NODE_OPTIONS environment variable is passed down into the worker, leading to an infinite spawn loop.

We use a nearly identical setup to @steffen-mueller, and I added an isMainThread check in our loader and it resolved the issue.

import { isMainThread } from "node:worker_threads";

if (isMainThread) {
  Sentry.init({
    includeLocalVariables: true,
    // ...
  });
} else {
  // skip setup in worker so we don't infinitely recurse
  console.warn("Sentry init skipped in worker thread");
}

[timfish]

I believe this is because the NODE_OPTIONS environment variable is passed down into the worker

Oh nice find! We specifically pass an empty execArgv: [] when we start workers so that --import doesn't get passed through and it looks like we need to exclude NODE_OPTIONS too.