Possible Memory Leak?

[igorescobar-moonpay]

Hi,

Is there any known memory leak issue with grpc-js + google-gax?

We have an application running on Nodejs 20 and we are trying to pinpoint what could be the root cause of this forever increasing memory consumption:

We compared two memory heap dumps from this application, just a few hours appart:

And you can see that all the hot paths are pointing to google grpc-js and google-gax. Zooming in a bit you can see it with more details:

This NestJS application runs on Node20, the lib versions are:

  ├─┬ @google-cloud/pubsub@4.10.0
   │ └─┬ google-gax@4.4.1
   │   └── @grpc/grpc-js@1.12.6

This applications uses google-gax from the google-cloud/pubsub library. It has just one server with a flow control of 50 messages. Two listeners (subscriptions), two different topics that that's pretty much it.

Is there any guidance here that could help us to fix this issue?

[karliky]

you should take heap snapshots to understand where memory is growing

[igorescobar-moonpay]

you should take heap snapshots to understand where memory is growing

@karliky the screenshots above were taken from the snapshot, anything in particular you are looking for? The red bars in the screenshot is indicating where the memory has grown over time so it's all inside grpc-js it seems. So I was wondering if there is any issue around how nodejs-pubsub uses grpc-js internally.

[karliky]

@igorescobar-moonpay that's not a heapsnapshot file from devtools but a custom tool, right? you need to look for Memory -> All Objects like this:

[igorescobar-moonpay]

@igorescobar-moonpay that's not a heapsnapshot file from devtools but a custom tool, right? you need to look for Memory -> All Objects like this:

It's not from devtools but it's the same thing. This is what Datadog generates but its the same thing. For example, here:

You can see that the heap size from this method grew considerably after a few hours.

[igorescobar-moonpay]

Also considerable growth here:

The growth is all coming from pubsub -> google-gax -> grpc-js.

[karliky]

DataDog doesn't capture full heap snapshots but heap profiles.

They only capture objects still alive at each 65-second collection and exclude those already garbage-collected or allocated natively (like strings, compiled code, etc.). These profiles come from the V8 HeapProfiler, which can generate both snapshots and allocation profiles.

I think Datadog uses allocation profiles for efficiency and privacy, so they’re similar to DevTools but not the same. You are missing a lot of information from those heap profiles.

[igorescobar-moonpay]

DataDog doesn't capture full heap snapshots but heap profiles.

They only capture objects still alive at each 65-second collection and exclude those already garbage-collected or allocated natively (like strings, compiled code, etc.). These profiles come from the V8 HeapProfiler, which can generate both snapshots and allocation profiles.

I think Datadog uses allocation profiles for efficiency and privacy, so they’re similar to DevTools but not the same. You are missing a lot of information from those heap profiles.

If you think it would be useful, I can ask our team to grab a devtools type of dump and take a screenshot of it so I can attach it here. I can't have access to the actual dump tho. We did compared snapshots using actual dump files and devtools too but in this case it was very similar to what we had on datadog already so we stopped and started using datadog.

So what are you saying is that, even tho the heap profile snapshot is given strong indication that pubsub libraries are leaking it could still be a false positive?

[karliky]

Man I'm not part of nodejs-pubsub nor grpc-js, I'm just a regular user trying to help you, if you take one heapsnapshot, wait 2 hours or so and then another one you'll pinpoint exactly where the issue is coming and you'll be able to patch it, wether it comes from a third party library or not.

[karliky]

I'm not saying anything about a false positive, I'm saying go to those pods, servers, processes or whatever, take heapsnapshots from Node.js processes and do a deep analysis of how memory evolves over time

[igorescobar-moonpay]

Man I'm not part of nodejs-pubsub nor grpc-js, I'm just a regular user trying to help you, if you take one heapsnapshot, wait 2 hours or so and then another one you'll pinpoint exactly where the issue is coming and you'll be able to patch it, wether it comes from a third party library or not.
I'm not saying anything about a false positive, I'm saying go to those pods, servers, processes or whatever, take heapsnapshots from Node.js processes and do a deep analysis of how memory evolves over time

Sorry @karliky, as I mentioned earlier, we already did that and the result is the same. Thank you for taking the time to reply to me.

[igorescobar-moonpay]

On grpc/grpc-node#2896, @murgatroid99 took the time to have a look at the screenshots and gave us a strong indication that the leak is coming from the clients so probably something is off with how we are instantiating the PubSub client somewhere. I'm are already looking into it.

[igorescobar-moonpay]

Can confirm the issue was indeed our NestJS application creating multiple instances of the PubSub client.

So if you find yourself in a similar situation, try to wrap PubSub into a module such as:

import { PubSub } from '@google-cloud/pubsub';
import { PubSubClient } from '@internal/pubsub-client';
import { Global, Module, Scope } from '@nestjs/common';

@Global()
@Module({
  providers: [
    {
      provide: PubSub,
      useFactory: () => new PubSub(),
      scope: Scope.DEFAULT,
    },
    PubSubClient,
  ],
  exports: [PubSub, PubSubClient],
})
export class PubSubModule {}

Make your PubSubClient @Injectable

@Injectable()
export class PubSubClient {
// ...

Then make sure you load the PubSubModule you have in your AppModule or similar. This will make sure you have a singleton reference for your client so it can be re-used across your app without having this issue.