Add acceptance / smoke tests for the server

[ieugen]

I will make configuration changes and code changes to add extension points.
I would like to be able to check that system produced by the build is working properly.

For that I believe we need to have some form of testing.
We should test for stuff like:

• are all the ports UP ?
• can we send an email locally with SMTP (dry run)
• does IMAP work
• do we have the users / domains we configured in the system ?
• can we read the mailboxes via JMAP
• are the custom mailets registered and are they working?
• are the SMTP hooks in place and woring?
• are the web-admin extensions working?
• etc.

At this point, I expect to automate just the running part.
Checking for fail/pass can be done by going through the logs with a written checklist like the one above.
The logs should contain clues that each sub-system works as expected.
It would be great if some checks can be automated, but I believe some of them will require a lot of work to do and it will not be worth it.

In time we might include a smoke test API for these ?!

Looking for docs on how to work on each one.

[rouazana]

Did you have a look at https://github.com/apache/james-project/blob/master/dockerfiles/compilation/java-11/integration_tests.sh which provides some basic smoke tests for a built server?

[chibenwa]

Did you have a look at https://github.com/apache/james-project/blob/master/dockerfiles/compilation/java-11/integration_tests.sh which provides some basic smoke tests for a built server?

It misses some of the point mentioned above like JMAP, SMTP.

https://github.com/chibenwa/james-blacklist/tree/master/blacklist-integration could provide a way to write Java integration tests (and covers extension testing). However James would be started in the testing JVM, while the link pointed by @rouazana relies on a dockerized James server.

[ieugen]

Thanks for the links. I will check them out.

I would like to test the released artifact.
If possible I would like to interact only with the exposed API and not use any back-channels.

In this context the docker image or the files built with tar.gz is the relased artifact.