Payload performs extremely poor when handling multiple requests?

[christian-reichart]

We've recently taken our site live and had extreme issues with speed. Pages where taking up to 20s to load. The CPU on our server was maxed out with something like 10 simultaneous users. We quickly implemented a SQLite cache layer between our Frontend and Payload to make our site usable again.

We still haven't figured out in detail what causes the issue and wanted to ask if anybody else is experiencing such performance issues or can think of a cause? We want to figure out if it's something specific we are doing or if it's a general performance problem others experience aswell.
Are there performance optimizations in a newer version of Payload?

A few facts:

• v1.5.3
• 21 collections
• 7 globals
• Interconnected collections and globals with many relationships (using feat(fields): add select property to upload and relationship field #1288 as a patch)

Any suggestions are appreciated :)

[jmikrut]

Hey Christian!

Hmm, this is definitely something that we have not seen so far. We have production instances of Payload that have LOTS of concurrent users.

We have been consistently releasing performance enhancements, although we've been in production with Payload since pre-1.0 and have never seen a slowdown like this. We actually have spikes of significant traffic from a few of our apps (Quikplow in specific) that handles thousands of users concurrently without breaking a sweat. That site is running on something like a $25 droplet though if I remember correctly. But that app relies on just about every piece of Payload - authentication, media storage, versions, pretty complex access control, hooks, etc.

What is the CPU / RAM of your server in prod? Where are you hosting?

Do you have any afterRead hooks? Are you using the cloud storage plugin, or storing media locally?

I would love to dig into this further and will dedicate time immediately if you can help us come up with a reproduction.

Another thought would be to have you run your site locally and try to stress-test it with a bot or similar. See if you can reproduce / narrow it down. If you can, that would be awesome!

[jmikrut]

Update: an unrelated Discord thread just brought to my attention that there is a memory leak in his application, so the team and I dug into it for a bit. We found that the probe-image-size package may have an issue with memory leaks, so I just pushed a quick fix to see if that will resolve his use case. It may also resolve yours.

What does the memory usage look like for your app in production?

[christian-reichart]

Hey James,
thanks for your reply!

I've been suspecting that it has something to do with our setup, as such bad performance probably would have come up elsewhere as an issue. :) Your examples give me an ease of mind, as it shows that Payload is capable of scaling and we can fix our setup if we figure out what in detail is causing the problems.

We are using 8 VCPUs and 16 GB RAM at the moment from a local hosting service in Germany. Server-wise everything should be fine, since we are running other apps on that same instance already.

We do have a few afterRead hooks and store media locally.

I will try to narrow things done, disabling hooks, etc. and get back to you!

[jmikrut]

OK keep me posted. I am more than willing to help you figure out what's happening here and would love to learn from it as well.

[ssyberg]

@jmikrut is there a known performance issue with afterRead hooks?

[jmikrut]

No, not at all. But, it depends on what you're doing in your afterRead hooks.

If you're fetching lots of external data for example, that will obviously slow down the response in general.

But the hooks themselves are next to no overhead!

[ssyberg]

Ok that makes sense!