High CPU usage when deployed onto EC2 instance (even after restricting 90%+ referers the errors can be originated from)

[SwatejReddy]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

8.48.0

Framework Version

Node v18.20.4

Link to Sentry event

No response

Reproduction Example/SDK Setup

const Sentry = require("@sentry/node");
const { nodeProfilingIntegration } = require("@sentry/profiling-node");
Sentry.init({
  dsn: "our-dsn-string",
  integrations: [
    nodeProfilingIntegration(),
  ],
  tracesSampler: (samplingContext) => {
    try {
      // Ensure headers and referer are defined before accessing them
      const headers = samplingContext?.normalizedRequest?.headers;
      const referer = headers?.referer;
  
      if (referer) {
        if (referer.includes('frontend-url')) {
          return 1.0; // Capture 100% of transactions for this referer
        } else {
          return 0.0; // Ignore transactions for other referers
        }
      }
      // Default to 1.0 if referer is undefined
      return 1.0;
    } catch (error) {
      console.error('Error in tracesSampler:', error);
      return 0.0; // Safely ignore errors by defaulting to 0.0
    }
  },
  // Tracing
  tracesSampleRate: 1.0, //  Capture 100% of the transactions

  beforeSend(event){
    // console.log(JSON.stringify(event.request.headers, null, 2));
    if (event.request?.headers?.["referer"]?.includes("frontend-url")) {
      console.log(`SENDING THIS REQUEST`)
      return event;
    }
    return null;
  }
});

// Manually call startProfiler and stopProfiler
// to profile the code in between
Sentry.profiler.startProfiler();
// Starts a transaction that will also be profiled
Sentry.startSpan({
  name: "My First Transaction",
}, () => {
  // the code executing inside the transaction will be wrapped in a span and profiled
});
// Calls to stopProfiling are optional - if you don't stop the profiler, it will keep profiling
// your application until the process exits or stopProfiling is called.
Sentry.profiler.stopProfiler();

Steps to Reproduce

1. Hit an error from frontend ("frontend-url").
2. Hit an error from another frontend to the same backend.
3. Only the error originated from "frontend-url" gets logged onto Sentry.
4. The problem is with CPU usage. Earlier, when we first integrated Sentry into our backend, our EC2 instances CPU usage shot through the roof and we had to disable it.
5. We then implemented the above checks to make sure that the errors that are not being originated from "frontend-url" don't even get sampled, thus reducing the CPU usage.
6. But tracesSampler alone doesn't stop errors from going to Sentry, so I implemented beforeSend() with a check to not send the errors and it worked.
7. But the problem with this is the CPU usage hasn't reduced and when we deployed it again, the CPU usage shot through the roof again.
8. I am guessing that beforeSend() is not stopping the errors that are not originated from the "frontend-url" from getting sampled but it is filtering them out after sampling them which explains why the CPU usage is still high even after this.

FYI: The incoming requests from the "frontend-url" are 10 out of 1000 total requests so, if we are only sampling those then CPU usage should have been reduced.

This is the only thing stopping us from deploying Sentry onto production which is very important and urgent for us.

Expected Result

• We only need the errors originated from "frontend-url" to be sampled, reducing the CPU usage.

Actual Result

[s1gr1d]

Hello, we already wrote about this on Discord where I was suggesting you to use tracesSampler and beforeSend to only monitor the errors of requests that originated from your website (and not other frontends).

Now looking at it again, I see that you are also using the profiling integration which actually has some CPU overhead. Do you still get the same CPU overhead when not including the profiling integration?

[SwatejReddy]

I haven't tried it without profiling integration on production. Will confirm once I test it out on prod.
And in case if that solves the issue, how do we get the performance metrics which are dependent on profiling integration?

[s1gr1d]

The sample rate of the nodeProfilingIntegration is dependent on and relative to the sample rate of tracing (e.g. the tracesSampleRate). And only when a trace is sampled, the nodeProfilingIntegration will decide wether to sample as well or not. Does this answer your question?

And please report back on what you found out when removing the nodeProfilingIntegration on production :)

[s1gr1d]

It could be related to this issue: #14388

The person was able to reduce performance issues on Google Cloud Run by pre-bundling the code.

[SwatejReddy]

Since we removed nodeProfilingIntegration and deployed it onto the production, there isn't a significant surge in CPU but what about all the things that we would be missing out by not including this feature?

As performance monitoring is one of our main reasons behind the decision to integrate Sentry into our project in the first place.