Slow test suite in Github Actions (long setup)

[nlarusstone]

Hi, I'm seeing really slow test runs on Github Actions. My test suite (~80 files, 500 tests) takes about 1.5min locally on my laptop, but over 12min on Github Actions:

   Duration  727.80s (transform 6.65s, setup 471.07s, collect 26.21s, tests 138.94s, environment 31.97s, prepare 5.56s)

I've tried threads, forks, vmThreads, and vmForks and nothing seems to make a significant difference.

I've noticed that the setup is strangely long, but I haven't been able to figure out where or why that is so slow.

Any advice on how I can profile this/speed it up?

[samiemad]

I also have a similar problem.

On github actions:

   Duration  938.28s (transform 9.79s, setup 97.10s, collect 431.30s, tests 34.35s, environment 256.93s, prepare 43.16s)

And locally on my laptop:

   Duration  187.06s (transform 35.37s, setup 250.92s, collect 855.88s, tests 90.76s, environment 562.52s, prepare 106.54s)

The weird thing is that for all sub timings, github is actually faster! But the total duration is still much larger than locally.
So it looks like on github, everything is executed on a single cpu or sequentailly. While locally it is using multiple cpu cores.

P.S. running the same test suite on google cloudbuild works well. I only see this problem when using github actions

   Duration  207.73s (transform 19.07s, setup 180.71s, collect 657.92s, tests 56.24s, environment 364.90s, prepare 62.48s)

@nlarusstone Did you find a solution for this?
Can you share the times for the local run as well?

[niklaas]

I'm interested in the comparison with the Google Cloud build. Is the environment comparable to the GitHub runner one?

We're investigating a similar issue and noticed that, apart from the environment, whether or not we enable DataDog tracing has a big impact on performance, see https://github.com/DataDog/dd-trace-js/discussions/5482.

[yaroslav-akhromeev]

We're really suffering from the same problem, any solution for it?

[vinothkumar-tw]

same problem

[SARAN-thala]

facing the same issue (237 test files, 2541 tests).

---

My local (Mac M1 pro, 10 core)
Duration 99.49s (transform 17.97s, setup 257.28s, collect 66.59s, tests 420.12s, environment 79.47s, prepare 14.81s)

In GHA (4core ubuntu machine)
Duration 625.28s (transform 22.17s, setup 593.92s, collect 63.45s, tests 931.14s, environment 143.87s, prepare 27.05s)

[sir-captainmorgan21]

Ya same here. It takes upwards of 5-7 times longer. I wonder if it has to do with the reporter?

[MarcosCosmos]

Not sure if it's related but I'm seeing vitest effectively hang in GHA when running vitest browser mode with chromium, only firefox seems to work for my team's codebase.

The weird thing is that I can't reproduce the issue at all outside of standard runners, even with https://github.com/nektos/act and the container image that is the most literal clone of official standard runners available, including setting memory constraints.

Note: It's a private repo, and there's a lot of complexity, e.g. storybook, so I don't have reproductions, but I noticed this thread is missing reproduction steps or any narrowing of relevant packages so I thought I should give that a nudge either way!

[dompuiu]

My test suite was also never completing the run on GHA until we disabled fileParallelism. We kept fileParallelism enabled on local machines only. You might give it a try.

[MarcosCosmos]

Ah thanks! For our repo, we weirdly managed to get it working if we ran browser mode using playwright with firefox; but not with any of the other playwright browsers!

We will also try file parallelism but I think the effect on during will be very steep

[dompuiu]

I remembered that we were seeing this behavior only when code coverage was enabled. So we can play with that too if you have it enabled.

[IanGraingerGMSL]

We have >300 tests in one of our repos, many of them using fake timers. I've migrated them from Jest to Vitest (v4.0.18) and was fine with ~2m running the tests locally - but in GH Actions where the Jest tests took 3m, the Vitest ones took 11m.

Duration  651.56s (transform 9.14s, setup 91.23s, import 90.81s, tests 206.13s, environment 184.21s)

I've tried disabling coverage and fileParallelism, as well as turning up the maxConcurrency to 10. None of that worked, so we're not able to migrate to Vitest at the moment. 😢

I've created a new post with this content - as our issue is slightly different (no long setup - just slow everything): #9683

[vinothwino]

I implemented Vitest sharding in my previous project, reducing test execution time from 20 minutes to 6 minutes. The GitHub Actions example is included in the Vitest documentation.
https://vitest.dev/guide/improving-performance.html#sharding