General Testing Best Practices

[ssyberg]

So this isn't super specific to Payload as I've encountered various testing issues across multiple CMS solutions, but I'm interested what people are doing and if it might even make sense to incorporate some baseline testing into Payload/documentation.

With a relational based CMS, model changes (or collection config changes in Payload) that might result in broken data often get exposed while trying to generate migrations. With document/flatfile based persistence layers, we generally don't have this feature and it can be tricky to pinpoint what change led to your admin/API malfunctioning. I often stumble upon a broken admin list view or detail view from some previous PR.

Does anyone have a simple test suite that exposes this during CI? Hitting every API endpoint with a GET would probably be a decent start...

[denolfe]

This is a great topic. To me, this comes down to being mindful of changes to your payload config during code review. I've run into similar scenarios when working with DynamoDB (NoSQL) at previous places. We do have some work related to migrations coming out with the next major release that may assist in this aspect, though.

In the meantime, I have a few suggestions:

1. Run type generation automatically on commit/pr etc. The diff on the single generated file is an easy spot to see if any fundamental changes were made to your models.
2. Post-deploy testing - Because your concern mainly stems around data, it feels like this one may be necessary. The trick here would be whether you can get accurate data into a staging environment or how comfortable you are running them against production.
3. Open API plugins - You could diff the swagger.json output when using one of these plugins: payload-openapi or payload-swagger. Probably not as helpful as the ones above.

it might even make sense to incorporate some baseline testing into Payload/documentation

I'm curious what you had in mind here. Definitely open to ideas.

[ssyberg]

Well our client asked us to stick to js on this project, so don't thing type generation will help - but agreed that could be a good one. Post deploy testing is a must, but really looking for something automated.

I'm curious what you had in mind here. Definitely open to ideas.

I guess it's a bit of a partial solution, but we've found in other headless/flatfile projects we get a lot of utility out of just doing a GET on every valid API endpoint. Not sure if that makes sense to build into core, but it's super low lift and has exposed quite a few issues in our CI. Seems like the API plugins might just do the trick here.

Much larger lift but maintaining seed data and exposing management commands could also help with exposing breaking changes (I've implemented an entire payload management command system (similar to django-admin) + standardized seed format, but I haven't have time to suggest it to core).

[gregwhitworth]

Thanks @ssyberg for raising this issue. And while reviewing PRs and preview testing is nice we also can't test everything that may be indirectly impacted. I'm curious about some e2e solutions; possibly with Cypress or similar on Github actions. Unsure on whether this will be easy/possible with Payload but recommended approaches to an automated solution would be awesome.

On the site side of things which is Next.js there are numerous recommendations I can lean into.

[ssyberg]

Thanks @ssyberg for raising this issue. And while reviewing PRs and preview testing is nice we also can't test everything that may be indirectly impacted. I'm curious about some e2e solutions; possibly with Cypress or similar on Github actions. Unsure on whether this will be easy/possible with Payload but recommended approaches to an automated solution would be awesome.

On the site side of things which is Next.js there are numerous recommendations I can lean into.

For sure, I can tell you what we've settled on for now that feels pretty good overall:

1. For testing general stability of the CMS, we rely on the seeds I mentioned above. This is a big lift to set up as you need to build seed scripts for every collection type, and ideally you should be adding data to every possible field. I'd also recommend having a single collection (real or contrived) that hits all of your block types so that you are creating data for each block type as well. The seed script failure itself has caught the vast majority of our regressions/bugs during development in the last few months.

2. We save the seeded data to a json file and we use the data as "factory" data in our actual "e2e" testing suite. This suite effectively just runs a graphql query on every collection type in our system. This hits a ephemeral development server that's built during test runs (I know that will rub a lot of people the wrong way) - but using vitest it is blazing fast.

The main things we are testing here:
(1) Our GQL fragments/queries are correct and we can read from all collections.
(2) All of our save/update hooks are doing what they are supposed to do, and our "virtual" fields are returning the correct data. I think this is kind of the extent of what one should be testing in a CMS.

FWIW we have ~30 collections and this all runs in about 3-5 minutes. Leaving vitest up locally it runs continuously during development, the longest part of the process is reseeding data after a model change.