Can Payload 3 be used with load balancing?

[hdodov]

Once we switched to Payload 3, we started experiencing issues, likely due to the nature of React RSC and Next not playing well with our load-balanced AWS environment.

Here is what happens in DevTools after each character I type in a field of a payload global named "sitewide". One time the response is RSC, as expected:

Correct response

…but the other time it's a regular HTML document:

Incorrect response

In the second screenshot, there's Accept: text/x-component in the request, but the response has Content-Type: text/html. As a result, I end up with errors like this when I try to save the document:

Uncaught (in promise) TypeError: Cannot destructure property 'lockedState' of '(intermediate value)' as it is undefined.
    at d98dc95b-a70d9a2d40c18cff.js:1:577891
    at async d98dc95b-a70d9a2d40c18cff.js:1:377772

Error submitting form TypeError: Cannot destructure property 'state' of '(intermediate value)' as it is undefined.
    at d98dc95b-a70d9a2d40c18cff.js:1:577418
    at async d98dc95b-a70d9a2d40c18cff.js:1:373997

I guess this happens because one of the servers has the user state and the other one doesn't.

@jmikrut can you confirm that Payload 3 can't or shouldn't be used with load balancing?

---

P.S. I know that this has been previously discussed in #6584, but it's about Payload 2, which wasn't based on Next and RSC.

[HarleySalas]

Do you have a custom cache handler for multiple instances of NextJS?

I'm a big ignorant to a lot of what Payload does behind the scenes, as I haven't had as much time as I'd like to dig in to the source.
However, I can't imagine why it'd be different than any other Nextjs application. Despite the fact that what you're doing is dynamic and you do seem to have cache disabled on the headers side; the data cache layer of Nextjs operates independently of caching headers and will not respect them.

I imagine this is an issue related more to NextJS and/or how you're hosting, rather than something specific to Payload, but I could be completely wrong.

edit: to be clear, if you are hosting NextJS outside of Vercel, or something that handles everything for you, you should absolutely be using your own cache handler. Without a cache handler, you will face numerous issues, similar, but not limited to what you've described above.

[hdodov]

But I'm not having issues with the cache handler. I could set up Amazon S3 as a handler for the static ISR cache if I wanted to.

I think that's more of a problem with the usage of server actions by Payload, not related to caching. I think that server actions store state in-memory and expect that the requests get routed to the same server (which has the same state). In my case, I have multiple load-balanced servers, so there's a chance that the request hits a server that doesn't have the required state.

The cacheHandler option by Next is concerned with the ISR cache, which is completely unrelated, I think.

@jmikrut @denolfe @JarrodMFlesch could you shed some light? I'm wondering if React 19 allows Payload CMS to be load-balanced at all. We're currently experiencing a lot of strange behaviors.

[HarleySalas]

custom cache handlers are not only for ISR. The newest documentation fails to fully explain it. The url slug for it is even called "incrementalCacheHandlerPath" which is quite odd to me, as it is definitely not only for incremental caching. It used to include more information on the page for cacheHandler, but here, you can still see that they describe a custom cache handler as being suitable for the Data Cache layer.

@nescha/cache-handler is a great library for working with distributed caches when self-hosting and it used to be officially recommended by Vercel.

Perhaps this is not the root of your issue and I could have the wrong approach though. I just wanted to clarify my thoughts on the data cache and distributed systems. I would still consider setting a custom cache handler up, as the NextJS documentation recommends, for Kubernetes.

[hdodov]

@HarleySalas I'll look into that, then. Too bad that the docs are misleading…

But what could server actions have to deal with in regards to caching? Can the result of a server action get cached or something? I can see that successful requests to the action endpoint (ones which return actual JSON, rather than the app HTML shell) have this header:

X-Action-Revalidated: [[],0,0]

…which leads me to thing that there may be some caching going on.

[HarleySalas]

Perhaps it could be something due to Form State, or somewhere along those lines? I haven't had this specific issue before, so I don't really have specific knowledge for it. While I'm quite comfortable with NextJS and caching, I don't try digging in to it deeper than I need to, as I'm sure it's quite the rabbit hole.

Wish I could be more helpful. Perhaps someone else will be able to chime in though. I'd guesstimate the odds of this issue being fixed with a custom cache handler setup is around 75%. Don't ask how I came up with that totally accurate number. 😂

[hdodov]

I added a custom cache handler (as suggested in the Next.js docs) with logging on each get(), set(), and revalidateTag(), and in production mode it seems that get() and set() aren't being used in Payload at all. And when the server action in question is invoked, it appears that only revalidateTag() is called with [] as the argument (not sure what this is…).

When navigating the rest of the site, the custom cache handler works as expected and actually caches pages (ISR), so it's not like my cache handler is faulty.

Since nothing is getting cached when the server action is run, it appears that a custom cache handler wouldn't solve the issue? Looks like I'll have to somehow attach a reproduction repo to this issue. Not sure how I'll simulate load balancing, though…

[cbratschi]

@hdodov Next.js uses a single cache for everything. If you use multiple processes or servers you have to implement a custom cache. There are a few examples how to do this correctly.

[hdodov]

Read my comment in the other thread. I know about Next.js cache. I think my issue is related to server actions and the way they interact with the server, rather than caching. Every time I hit "Save" in the browser, there's a 50/50 chance that it works, depending on how the request gets load-balanced.