Issues with npm datasource updates hitting memory limits on Mend-hosted infrastructure

[jamietanna]

How are you running Renovate?

A Mend.io-hosted app

Which platform you running Renovate on?

GitHub.com

Which version of Renovate are you using?

42.95.2

Please tell us more about your question or problem

We've recently seen an increase in projects using pnpm and yarn hitting memory issues when installing dependencies on a project

• pnpm Renovate stopped working on the open-source software fast-check #40541
• yarn Job fails with kernel-out-of-memory #40352
• yarn: Frequent intermittent kernel out of memory errors (with `npm` datasource) #38880
• pnpm: Renovate started failing with "kernel-out-of-memory" #40940
• pnpm: Increase memory - job fails with kernel-out-of-memory #40935
• pnpm: Renovate-job started to fail with a kernel-out-of-memory Exception #40943
• pnpm: Renovate job fails with kernel-out-of-memory #41020
• pnpm: Increased resources for OSS organization gemini-hlsw #41685

We're in the process of investigating this further, and will share updates in this Discussion.

Logs (if relevant)

No response

[jamietanna]

One attempt at fixing using pnpm configuration: #40541 (reply in thread)

[jamietanna]

#40957 is a suggested change (which we can then use on Mend-hosted)

[ssbarnea]

Is this the magic conversion trick for funnelling more free users into paying customers? -- clearly would not be able to squeeze much from unfunded open source project using it. /sarcasm

Now here is an funny aspect that might have contributed to the OOM issue: the recent new dependency added to the project was renovate npm package itself, which:

• is the only way to install renovate-config-validate (linting config)
• has a huge list of dependencies
• has a huge list of outdated and compromised dependencies
• is updated extremely often (not a problem by itself but if you only need linting config... it becomes one)

I wonder if we can spot a pattern among those affected by this bug.

Considering the number of users reporting this issue very recently, is almost for sure related to increase of memory needs from newer releases. Still, the increase might prove to not be critical and an increase of node memory of 10-20% might address it but sadly I did not see any signs of this being tried on mend.io services.

I seen the need to increase this in the past with tools like webpack.

[jraoult]

Hi @jamietanna, we've been hitting this consistently in our repo for 4 days in a row (therefore blocking any dependency upgrades). Is there an official workaround at the moment?

Happy for you to use our repo if you need a reproduction case.

[jamietanna]

Short answer, no, sorry! We're working on the CLI changes that would allow specifying these environment variables (but a few other things have also popped up needing work on them)

[jamietanna]

A couple of tests I've done so far with (against a few repos who've reported issues):

{
    "toolSettings": {
      "nodeMaxMemory": 1024
    }
  }

seem to be going positively - I'll do a little bit more testing, then we'll have this set across all repos

[faradaytrs]

We still face the issue even with 1024 limit @jamietanna

[faradaytrs]

I tried to limit with 700, then we faced the following issue:

[faradaytrs]

1024 and more still kernel out of memory, 700 is javascript heap out of memory ……
https://developer.mend.io/github/globuslimited/globus/-/job/511acee6-0137-4a26-879c-0dcd4554532e

[jamietanna]

How high have you gone, memory wise? 1.5GB? Have you been able to see if higher values work?

[faradaytrs]

How high have you gone, memory wise? 1.5GB? Have you been able to see if higher values work?

i went up to 1.5, if there are many dependencies to update, it fails with kernel out of memory, if i update them manually, then run can success, but it will fail again as soon as there are many deps to update.