Memory leak

[Joakim-Palm]

Introduction

2 month ago we started seeing problems with the memory increasing constantly until reaching a point where the app restarted.

Specifications

Next.js: v.10.2.3
React: v.16.14.0
Node local: v.15.14.0

Hosting: AWS ECS Fargate
Docker Node: v.15.12.0

Application Background

https://etonshirts.com/
E-commerce store selling globally. around 10.000 users daily

The problem

Steady increase of memory usage. When the memory reaches 50% it restarts and then the memory immediately starts increasing.

First arrow (from left) was a deploy that failed due to logging added to express. We removed this logging and made a new deploy that was able to finish without any problem (second arrow) 12-14 hours after this deploy the memory leak started.
We have reverted or changed more or less all changes in this second deploy but the memory leak still exist

We have tried this locally and on staging environments but we are not able to reproduce it there.
Also found an article about adding: experimental: { isrMemoryCacheSize: 0 }, but sadly it did not help.

A bit lost here so any help/ideas is appreciated 😀

[icyJoseph]

Hi,

I don't know if this would help at all, but I could try and guess a few things.

Let's, see it's been a while since Next 10.2.9, did you also upgrade to Node 15 around the same time? Perhaps something is going on with unhandled promises? Although those should break the entire application.

Do you use getStaticPaths with getStaticProps with fallback true to generate pages on demand, at runtime?

Are you using next/image? Maybe it is accumulating optimised images in memory?

Do you have any globals? Maps, Sets, arrays, plain JS objects that could potentially prevent garbage collection, or just keep on increasing memory consumption. I know this is a vague one, and hard to track, but still possible.

Are you properly closing handlers to databases? outgoing requests, etc? In general, is asynchronous work properly finished, cleaned up and deleted?

I think you can check this out through process._getActiveHandlers and process._getActiveRequests. I don't want to straight up recommend packages, but there's npm packages that could help out debugging this section. There's also packages that can help getting heap dumps.

[Joakim-Palm]

Hey @icyJoseph !

Any help is appreciated 😃

Yeah its been a while, we plan to update to newer next version but we want to get this sorted before looking into updating.
Node was updated at the same time as Next han has been running fine for some time now but that could be good to look into

Not using any getStatic so that one thing to check off

Currently not using any Next/image components so that should not be a problem.

We have som globals and we are looking into this but as you said a bit hard to track

99% sure we don't have any outgoing request and so on but will have a look into this just to be sure

[icyJoseph]

I'd also imagine that upgrading next versions would be a way to mitigate this issue. Since you are not necessarily using all APIs you could just upgrade all the way to 12, and see what happens.

Is there any correlation between amount of users and these peaks on memory? I am trying to understand if the memory is leaking through actual usage or if there's some runaway process that when activated just won't stop.

Are you using a custom server for Next.js?

[Joakim-Palm]

Im guessing it has to do with the amount of users seeing that we cant recreate the problem on staging or locally.

We have a custom set up on AWS ECS Fargate for the Next app

Did some testing with the dev tools Memory tab and found that Heap Snapshots goes up every time i open the mobile menu or any other pop ups. Seems to incresse with around 0.3-0.6mb every time i open it. Havent used these tools before but for me it feels strange that it keeps going up or is this standard?

[icyJoseph]

Did some testing with the dev tools Memory tab and found that Heap Snapshots goes up every time i open the mobile menu or any other pop ups. Seems to incresse with around 0.3-0.6mb every time i open it.

Sorry, was this in the client side code? that shouldn't be a problem right? or was this with node --inspect-brk?

Regarding load, maybe you could use k6 and perform load tests against your stage environment, https://github.com/grafana/k6.

Alternatively, I'd consider tagging timneutkens, I do remember seeing some discussion where they were able to help out through an actual talk.

I guess it is possible in next 10, to create an api/process-health endpoint, and say for example, every 10 minutes make an automated call to the endpoint. That endpoint should log to your CloudWatch logs a few metrics about the ongoing process:

// pages/api/process-health
export default async function handler(req: NextApiRequest, res: NextApiResponse) { 
// maybe reject non GET requests

    loggingSystem({
        rssMemoryUsage: process.memoryUsage.rss(),
        cpuUsage: process.cpuUsage(),
        architecture: process.arch,
    }); // check this results in CloudWatch

    return res.send('ok');
}

There are more variables in process that you could use to take measurements from the server, while it is alive. This might give you another perspective into the resource consumption of the server.

[fmmoret]

are you using next-connect? Bug in versions <0.12.1 hoangvvo/next-connect#178

[thomb]

I ran into this problem as well - pretty much the exact memory curve and behaviour - and took me ages to track down.

Not sure this would be the same cause for everyone, but in my situation this stemmed from a combination an async call on the server side - to fetch auth data - and a health check configured in AWS.

The health check just gets the header response code, but doesn't load the site so the promise never settles.

Changing the health-check away from a page performing that async request - ie. the index page of the site - to a specific static health check page immediately resolved this problem.

I could be wrong about the cause described above, but as this is a no-code change it might be worth trying out to see if it alleviates the problem.

[wshen001]

We also run into the same critical issue as well on production environment... does anyone have any suggestion?

[edward1127]

I face the same issue...

[eppix-martin]

We faced the same issue with AWS health check and the page we were using. We invested months to track the issue 🙏