Cypress crashes with "We detected that the Chromium Renderer process just crashed."

[Lasidar]

Test code to reproduce

I cannot provide the specific test code I am using to produce this, since it contains internal company data. However, what I can tell you is these tests run for quite a long time (5-10 minutes), as they contain a lot of steps through our E2E flow in our application.

Cypress Mode

cypress run

Cypress Version

12.13.0

Browser Version

Chrome 110.0.5481.96

Node version

v16.18.1

Operating System

Debian GNU/Linux 11 (Docker image cypress/browsers:node-16.18.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1)

Memory Debug Logs

Only the end of the failing test results from the memory dump have been included due to limitations of this input form.

[
  {
    "checkMemoryPressureDuration": 7764.029118001461,
    "testTitle": "Crashing test (example #1)",
    "testOrder": 18.2,
    "garbageCollected": true,
    "timestamp": 1690553650738
  },
  ...
  {
    "getRendererMemoryUsageDuration": 2.619260013103485,
    "totalMemoryWorkingSetUsed": 6995816448,
    "getAvailableMemoryDuration": 58.69076597690582,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 5469155328,
    "rendererUsagePercentage": 127.34646813770371,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372029858955000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": true,
    "timestamp": 1690553801030,
    "calculateMemoryStatsDuration": 58.72436600923538
  },
  {
    "getRendererMemoryUsageDuration": 2.208419978618622,
    "totalMemoryWorkingSetUsed": 5089853440,
    "getAvailableMemoryDuration": 61.31387501955032,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372031764918000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553802092,
    "calculateMemoryStatsDuration": 61.33369600772858
  },
  {
    "getRendererMemoryUsageDuration": 2.69021999835968,
    "totalMemoryWorkingSetUsed": 1682976768,
    "getAvailableMemoryDuration": 50.05962598323822,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372035171795000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553803143,
    "calculateMemoryStatsDuration": 50.07922601699829
  },
  {
    "getRendererMemoryUsageDuration": 2.889739990234375,
    "totalMemoryWorkingSetUsed": 1682792448,
    "getAvailableMemoryDuration": 60.31445497274399,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372035171979000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553804204,
    "calculateMemoryStatsDuration": 60.33361500501633
  },
  {
    "getRendererMemoryUsageDuration": 2.6974300146102905,
    "totalMemoryWorkingSetUsed": 1682558976,
    "getAvailableMemoryDuration": 225.94400304555893,
    "jsHeapSizeLimit": 4294705152,
    "totalMemoryLimit": 9223372036854772000,
    "rendererProcessMemRss": 0,
    "rendererUsagePercentage": 0,
    "rendererMemoryThreshold": 2147352576,
    "currentAvailableMemory": 9223372035172213000,
    "maxAvailableRendererMemory": 4294705152,
    "shouldCollectGarbage": false,
    "timestamp": 1690553805431,
    "calculateMemoryStatsDuration": 225.9711429476738
  }
]

Other

Our test specs that contain multiple long running tests are prone to crashing mid-run in CI. This seems to be more likely when there are test retries in the run. We are running with both experimentalMemoryManagement set to true and numTestsKeptInMemory set to 0. We also have the memory and CPU allocation in our GitLab runners set quite high (see below). Despite this, we still get the crashes. Example:

Some top level test description
    (Attempt 1 of 4) A test scenario containing a scenario outline template (example #1)
    (Attempt 2 of 4) A test scenario containing a scenario outline template (example #1)
    ✓ A test scenario containing a scenario outline template (example #1) (849857ms)
    ✓ A test scenario containing a scenario outline template (example #2) (360954ms)
    ✓ A test scenario containing a scenario outline template (example #3) (556574ms)
    (Attempt 1 of 4) A test scenario containing a scenario outline template (example #4)

We detected that the Chromium Renderer process just crashed.

This can happen for a number of different reasons.

If you're running lots of tests on a memory intense application.
  - Try increasing the CPU/memory on the machine you're running on.
  - Try enabling experimentalMemoryManagement in your config file.
  - Try lowering numTestsKeptInMemory in your config file during 'cypress open'.

You can learn more here:

https://on.cypress.io/renderer-process-crashed

Here are the memory allocations we are providing in Gitlab CI:

  KUBERNETES_CPU_REQUEST: "3"
  KUBERNETES_CPU_LIMIT: "4"
  KUBERNETES_MEMORY_REQUEST: "12Gi"
  KUBERNETES_MEMORY_LIMIT: "32Gi"

It should be noted these tests run within docker in CI, and are running in the cypress/browsers:node-16.18.1-chrome-110.0.5481.96-1-ff-109.0-edge-110.0.1587.41-1 version of the cypress image.

We are utilizing the cypress-cucumber-preprocessor library, but I do not believe that is having any impact on this issue.

[Lasidar]

I have a speculation as to what is causing this which is based on a few logical leaps, but I think they are reasonable. I think the issue is mainly caused by the open issues involving cypress and K8s (see jlandure/alpine-chrome#109).

Looking at the logs in question, I noticed the jsHeapSizeLimit limit is being exceeded right before the crash.

    "jsHeapSizeLimit": 4294705152,
    "rendererProcessMemRss": 5469155328,
    "rendererUsagePercentage": 127.34646813770371,

This is strange, since we have 12GB allocated to the Gitlab runner, with the ability to scale up to 32GB if the process calls for it. But looking at the issue linked above, I believe this prevents the Chrome renderer from being able to scale up its heap memory usage, even if the system has capacity.

From looking at the Cypress source code, it seems the renderer process memory is pulled from the chrome jsHeap, and if we exceed this, we are likely smashing the stack which probably leads to the crash.

  // the max available memory is the minimum of the js heap size limit and
  // the current available memory plus the renderer process memory usage
  const maxAvailableRendererMemory = Math.min(jsHeapSizeLimit, currentAvailableMemory + rendererProcessMemRss)

So my workaround, at least temporarily, is to increase the chrome jsHeapSizeLimit value. I was able to achieve this through the following additions to my cypress/plugins/index.js file:

module.exports = (on, config) => {
  on("before:browser:launch", (browser = {}, launchOptions) => {
    if (browser.family === "chromium") {
      launchOptions.args.push('--js-flags="--max_old_space_size=1024 --max_semi_space_size=1024"');
    }
    return launchOptions;
  });
});

This seems to have made my memory crashing issues go away for the time being. I believe the correct fix for this is for jlandure/alpine-chrome#109 to be resolved, since as I mentioned above, I suspect this issue prevents the process from being able to properly scale up its heap size.

[Pardoniano]

I started to have the same problem this week. Already tried your solution on cypress.config.ts, with --max_old_space_size=4096. And still crash on a specific element. He crashes in Chrome and Edge, but work's fine in electron.

[uladzimirdev]

increasing max_old_space_size helps locally, but I can clearly see memory usage increase on chrome 115 - it can be 3.8Gb in my case

[Pardoniano]

I tested it in the homolog environment, I encountered this error. Interestingly, the button who causes the crash it works perfectly outside of Cypress, and I have tested it in various scenarios with cypress. The version of Cypress used is 12.7.0.

Here are the details of the environments where the issue occurred:

1. Environment: Windows 22621.1992
   Browser: Chrome - Version 115.0.5790.110 (Official Build) (64-bit)
   Result: Didn't work

2. Environment: macOS Ventura version 13.4.1 (c) (22F770820d)
   Browser: Chrome - Version 115.0.5790.114 (Official Build) (arm64)
   Result: Didn't work

3. Environment: macOS Ventura version 13.1 (22C65)
   Browser: Chrome - Version 115.0.5790.114 (Versão oficial) (arm64)
   Result: Didn't work

4. Environment: Ubuntu
   Browser: Chrome - Version 113.0.5672.126
   Result: Worked fine

5. Also tested it in our pipeline, and it worked without any issues. The Docker image used was cypress/browsers:node16.14.2-slim-chrome100-ff99-edge.

I suspect that the Chrome version 115 might be causing the problem, possibly due to memory leaks or other issues, idk, just guessing so far. Additionally, in the Electrum application, after clicking the specific button he shows in the cypress console one of the following error messages:

1. ResizeObserver loop limit exceeded;
2. ResizeObserver loop completed with undelivered notifications.

Wich i already fixed way back then with Cypress.on("uncaught:exception")

Unfortunately, this is all the information I currently have, as I haven't had enough time to investigate further. And i tried the workaround, sadly didn't work. So far my workaround is gonna be work on electron.

[nagash77]

Hi everyone on this thread. Thank you for submitting this ticket and all the details and investigation you all have done. We do continually revisit performance and resource utilization within Cypress so these are great data points. Unfortunately due to the nature of these problems they can be extremely hard to reproduce and even harder to track down so I cannot give any timelines about a resolution. If anyone seeing this ticket has ideas on culprits, I encourage you to open a PR and we will work with you to get that merged if it shows that it can help this problem.

[Adrian-Sana]

I am having the same issue, it is crashing with Chrome 115 but if I download the stable 114 from here: https://chromium.cypress.io/mac/ and then run it with --browser [path to the downloaded chrome] it works.

[Lasidar]

@nagash77 I understand that actually finding the root cause here will be extremely difficult. One thing I did find interesting when I was analyzing the memory usage leading up to the crash is between tests, the memory usage never returns back down to a baseline level. It is continually increasing. This is even with experimentalMemoryManagement set to true and numTestsKeptInMemory set to 0. I find this strange since I am not sure what would be persisting in memory between tests.

If a true root cause fix is not possible, I have a few possible suggestions:

• Give the users control over the browser heap size allocation via a parameter in cypress.config.js. That way, instead of setting it through a before:browser:launch trigger, it could be set more transparently via a test configuration option.
• In the event of a renderer crash, perhaps we could print more verbose memory statistics (i.e. max rendererProcessMemRss used, jsHeapSizeLimit, etc).
• If possible, try and solve the K8s / docker issue (i.e. ☸️ Can not run in Kubernetes? jlandure/alpine-chrome#109), which I believe may be contributing to this issue.

[ChristianKlima]

It seems as if Chrome has been overwhelmed with the layout of our application including Cypress since version 115. Our application runs unstably when command-log window is displayed. As soon as an action is performed in the application that writes a new entry in the command log, the browser crashes. This is usually not easy to reproduce, but we were able to reproduce it in a test at some point. Debugging the Chrome crashdumbs didn't help us either. Everything was stable up to version 114 of Chrome. Our solution is to set CYPRESS_NO_COMMAND_LOG: 1 now. Maybe it will help someone. We hope that at some point the problem will solve itself with a Chrome update or with a simpler cypress layout. Unfortunately, this is not that easy to reproduce in a small sample application.

[olahaida]

I ran in to the same issue. Set of test that I was running without any problems a month ago, now is crushing with

"We detected that the Chromium Renderer process just crashed.
This can happen for a number of different reasons.
If you're running lots of tests on a memory intense application.

• Try increasing the CPU/memory on the machine you're running on.
• Try enabling experimentalMemoryManagement in your config file.
• Try lowering numTestsKeptInMemory in your config file during 'cypress open'."

I've added experimentalMemoryManagement: true to my config file. Also reduced the numTestsKeptInMemory, but it's still crashing. Did not crash only when I reduced the number of tests to 0, but it's not solving my problem because I need to see those results (and a month ago it was working without any reduction).

I can see in the Task Manager that the process is crashing whenever Cypress is reaching over 4GB of RAM, I know it was always consuming a lot of resources but not that much

Windows 10
Cypress version: 12.17.4
Chrome v116
Electron v106

[AMEntwistle]

We also ran into this issue, particularly with chrome v116. We currently are locking to chrome version 115.0.5790.110-1 which is stable as far as we can see in regards to this memory issue. I don't know if this is more an issue with chrome or cypress, but being unable to test on latest browser versions is not ideal.