Background operations / pending documents / queues / jobs

[AlessioGr]

I have collection A and B. Each of them has a beforeChange hook.

Collection A - in its beforeChange hook - creates 5 documents in collection B and adds them to its relationship field.

The problem is, that collection B also has a beforeChange hook which does some expensive tasks - takes around 5 minutes to complete. Thus, when you create a document in collection A currently, it displays a loooooong ass loading screen.

Do you think it would be possible to instead return a job or something like that for that new document, and have it in a pending state, while also being able to display that pending document in the relationship of collection A already (and mark it as such)?

That would be one use-case/example for such a queue/background job system. There's probably tons more.

[DanRibbens]

Great feature idea!

We at Payload have implemented a job system using a collection to manage the jobs and https://www.npmjs.com/package/node-cron to run them. It works quite well.

The job script queries jobs for any that haven't been ran, there is a switch statement to invoke the correct job function to run passing along the JSON data from the job document as the args.

One piece that we could have improved was the job status/visibility. It would be nice to have a way for frontends to be able to subscribe (websocket) to know the status.

It likely should be a Payload plugin rather than core since it is not as common of a requirement and the extra dependencies could be lessened and some apps might want to separate worker instances.

Do you think it would be possible to instead return a job or something like that for that new document, and have it in a pending state, while also being able to display that pending document in the relationship of collection A already (and mark it as such)?

This is some deeper integration work that is very specific to your situation. I would want that to be possible with some custom UI components, but it doesn't seem like a standard requirement.

[AlessioGr]

Hah interesting! Haven't thought about using collections for those. I actually spun up a system using BullMQ + redis there. I guess the end result is similar.

However, it does seem to introduce quite a bit of additional complexity, particularly when it comes to setting up a status endpoint and displaying job status. This can be even more challenging when jobs are interdependent and affect multiple collections.

With Hooks alone, managing such chains is relatively simple, for instance, when collections create other collections through their BeforeChange hooks, which in turn also create further collections using their own BeforeChange hooks. It would be interesting to explore a more manageable solution that could potentially work as a plugin.

Do you not think some kind native solution would be helpful for people with very long-running BeforeChange hooks, though - which could itself be a more common problem? E.g. giving the user the ability to exit the page and still see the status of one of these hooks when revisiting the document - if it's still in progress, instead of looking at a very long loading screen.

[DanRibbens]

I think Payload's responsibility needs to cover pending operations and provide a way to lock further changes or provide enough info to make working with locked documents possible. I see there being overlap here on the requirements needed for having multiple users work on the same documents in the admin UI as well as being able to handle programmatic changes.

Anything beyond that would be the job of a queue system which would likely be a plugin. The reason I say to position it as a plugin is because there could be wildly different implementations with their own opinions and have different package dependencies. Like you said, you use BullMQ + redis, somebody else will want SQS and we chose to keep it simple using a standard Payload collection.

Thoughts?

[AlessioGr]

The reason I say to position it as a plugin is because there could be wildly different implementations with their own opinions and have different package dependencies. Like you said, you use BullMQ + redis, somebody else will want SQS and we chose to keep it simple using a standard Payload collection.

Completely makes sense! On the other hand, that could be said for a lot of things in payload, e.g. MongoDB as database, Slate as richtext editor. Wouldn't hurt to provide a "default" solution but allowing people to use their own.

In any case, I do think just having some kind of pending operation / locking mechanism would be great! That would make working with queues a lot easier, esp. when it comes to giving users feedback in the admin UI.

[DanRibbens]

Going forward we want to introduce adapters in areas where dev's tool preferences can come through: database, bundler, image resizer, richtext editor (close to your heart), etc. I could see an adapter pattern being used for job queues also.