Setting up a mail server with OpenSMTPD, Dovecot and Rspamd

[poolpOrg]

[leo-unglaub]

Awesome article, thanks for all your work!

[mchack23]

neat, been waiting for this article for some days now since I read your "mail's not hard" article just when I was trying to set up an openbsd smtpd server myself.
Since I'm still flailing a bit (it is working but I managed to create local mail loops for instance), I'll try and recreate and tweak your setup next :) Hints on how to set up virtual users are most welcome (and how I can relay local user mail to those virtual users)

Anyway: Found a small typo it should say options.inc (not options.conf) in this line: "This is done by adding the dkim_signing filter to the filters key configuration in /etc/rspamd/options.conf:"

When I find more I'll edit this post. :) Thanks again Gilles!

[poolpOrg]

@mchack23 typo fixed

the source markdown for the articles are in this repository, feel free to submit pull requests for typos and such :-)

[myfirstnameispaul]

I have been using Vultr 1GB VPS for about a year with a LEMP server spitting out transactional mail without issue. A few months ago I moved my MiaB over to the same service and haven't had any real issues, never submitted a ticket for any sort of filtering. Are you using a European DC?

Have you found dnswl.org to be useful? I see a lot of the big guys are there.

I look forward to your article on Microsoft because I'm not sure it can be done, as well as the DKIM key size as this is the first I've been aware of this.

While I originally taught myself how to configure my own mail server, I moved to MiaB in order on a simple VPS because as a personal mail server, it's fantastic and trouble-free, as much as a mail server can be.

However, I am wondering if you ever cover capacity limits? For some of the organizations I think you are targeting, they may quickly max out a VPS so capacity planning seems like a very relevant topic. What is considered for CPU limits in a mail server? What is using up memory and how painful is swap? How much storage should I plan on using, especially when forecasting growth rates? Are there some rules of thumb for any of these? Can I share block storage with more than one server? Reverse proxy? Etc...

I have also wondered about IPv6. What happens when spammers can just use a new IP address for every email and blast you a million times a second into eternity? Can I get by with just IPv4?

[benwaffle]

If reading about how you can build such setups is of any interest, I could also write about it to help people build MORE hosting alternatives which is ultimately what I hope for.

Yes, I'm interested.

[superkuh]

The hard part is 5 years later when your current mailserver's OS goes end of life and you have to port everything to a new setup using different versions of things.

[cmouse]

I'd like to suggest incorporating this in your howto:

OpenSMTPD could use LMTP (or execute dovecot-lda, not sure which works in OpenSTMPD) to deliver mail via dovecot, the benefit is that then dovecot updates it's indexes immediately, and you can benefit from sieve filtering for incoming mail. Then you can make dovecot filter e.g. spam messages to Spam folder directly.

[ynakao]

Thanks for the great writeup. As @benwaffle mentioned, I'm also interested in large-scale email server setup especially about virtual users and multiple domain managements.

I will write instructions assuming the upcoming 6.6.0 release which is due in a few weeks now. It brings a ton of improvements and I don’t want the instructions to be obsoleted and need a rewrite in October.

Out of curiosity, which is the OpenSMTPD v6.6.0 specific settings? Will there be similar breaking changes on OpenSMTPD in the future?

[omoerbeek]

PR #18, I think the flag does not match the text.

[poolpOrg]

@myfirstnameispaul

I have been using Vultr 1GB VPS for about a year with a LEMP server spitting out transactional mail without issue. A few months ago I moved my MiaB over to the same service and haven't had any real issues, never submitted a ticket for any sort of filtering. Are you using a European DC?

For my own needs, I'm using multiple DC (Amsterdam, Frankfurt, London, Paris) all in Europe yes.

Have you found dnswl.org to be useful? I see a lot of the big guys are there.

In all honesty no, I know dnswl.org but I can't even recall if I ever subscribed mine, so they may be useful but you can certainly do without.

I'm 100% positive that I didn't subscribe most of the domains I plugged.

However, I am wondering if you ever cover capacity limits? For some of the organizations I think you are targeting, they may quickly max out a VPS so capacity planning seems like a very relevant topic.
What is considered for CPU limits in a mail server?
What is using up memory and how painful is swap?
How much storage should I plan on using, especially when forecasting growth rates?

I don't because it really depends on what you are going after.

I documented a very simple setup which would work fine for a lot of people on the cheapest VPS, because that's how I test these setups and even load them a little, but how people need to handle capacity planning depends on many factors.

On my own setup, Rspamd instances run on machines of their own so I can adjust them depending on CPU and RAM usage, whereas Dovecot runs on a separate machine with decent disks, whereas my OpenSMTPD instances are basically CPU-bound, delivering the messages to a remote machine and not the local disk. My planning here is per-service and since services are per-machine, this is something I'd expect a company to do but not a lambda user, my recipes won't work for them.

Then, even on a simpler setup, some users will want MariaDB to handle their accounts, others will want an elk stack to monitor, others will want a webmail and maybe some fancy log analysers, and depending on where you run these, it changes radically how you do your planning. A machine could run very fine with your mail infra and load like hell just with elastic installed, so you might need to go for a real dedicated server or much higher-frequency CPUs just for one of the components.

I think capacity planning is not very different with mail than with any other service and, as far as I'm concerned, it goes by graphing and monitoring how things evolve in time to adjust, I don't have a secret works-for-all recipe. Some more competent SREs surely have better solutions though.

A very cheap VPS will do as they are more powerful than most mail servers from the last decade and the software have not grown more hungry in resources (only speaking about opensource solutions, not the commercial ones which will plain not work on a VPS).

For Rspamd, I have no idea what the minimal resources are, but I had never had a problem running it on the cheapest VPS I could get my hands on. It mostly does CPU, that's about all.

For Dovecot, this is VERY dependant on how many users you have, how they consume their mail, if they search mailboxes and such, so again I had never had a problem running on the cheapest VPS BUT your mileage may vary and graphing resources consumption to adjust is key. It mostly does CPU and DISK but is also mostly idle if not on a busy server.

For OpenSMTPD, not only the cheapest VPS will do but SMTP servers are mostly idle, so testing the limits is quite difficult. It can run (not just build but actually run) on very cheap machines, people use Raspberry boards, as well as on some ancient or not-so-powerful architectures supported by OpenBSD (https://www.openbsd.org/plat.html). It does CPU and DISK but it can also deliver remotely (what I do) in which case it mostly does CPU when it's not idle, even on a relatively busy server.

A lot of people thing that a lot of CPU is needed for all the TLS handling but this is not correct, I could share stories about that ;-p

I don't observe variations in terms of memory usage across time and on a server that has been running for a while, the memory consumption of just mapping the Go runtime is an order of magnitude higher than the memory consumption of any process from Dovecot or OpenSMTPD for instance.

When looking into how good I'm with resources, my main concern is if I should add disk or not.

Are there some rules of thumb for any of these? Can I share block storage with more than one server? Reverse proxy? Etc...

The sharing of block storage is a tricky part and depends on what system you're using.
If using Maildir backend, you're supposedly safe when sharing block storage because Dovecot keeps its state internal to a Maildir, and OpenSMTPD delivers atomically and lockless inside Maildir.
Then questions arise if you're not only sharing block storage but also clustering / cloning the SMTP instances or the IMAP instances. Both are technically doable, not necessarily in a very tricky way, but I would not say it's easy.

Reverse proxying is simple in both cases, both Dovecot and OpenSMTPD can support proxy-v2 and as a matter of fact I'm using an haproxy myself.

I have also wondered about IPv6. What happens when spammers can just use a new IP address for every email and blast you a million times a second into eternity? Can I get by with just IPv4?

That's a problem.

But since it's not globally solved by Big Mailer Corps, IPv6 remains insignificant when it comes to mail as of today and you can certainly just get by with IPv4.

[poolpOrg]

@cmouse

I'd like to suggest incorporating this in your howto:

OpenSMTPD could use LMTP (or execute dovecot-lda, not sure which works in OpenSTMPD) to deliver mail via dovecot, the benefit is that then dovecot updates it's indexes immediately, and you can benefit from sieve filtering for incoming mail. Then you can make dovecot filter e.g. spam messages to Spam folder directly.

I'd rather see people write such howtos ;-)

[poolpOrg]

@ynakao

Thanks for the great writeup. As @benwaffle mentioned, I'm also interested in large-scale email server setup especially about virtual users and multiple domain managements.

I will write instructions assuming the upcoming 6.6.0 release which is due in a few weeks now. It brings a ton of improvements and I don’t want the instructions to be obsoleted and need a rewrite in October.

Out of curiosity, which is the OpenSMTPD v6.6.0 specific settings?

All the filter stuff comes with v6.6.0 so while you could have rspamd, dkim and such, it was done through a slighly trickier method of "queue reinjection" where you tell OpenSMTPD to relay to an Rspamd proxy or a DKIM proxy and you tell the proxy to re-enqueue in the daemon.

It was not necessarily much more complex in terms of configuration but it was a bit more complex in terms of understanding how it works and far less elegant than just adding a filter keyword to a listen line :-)

Will there be similar breaking changes on OpenSMTPD in the future?

In this case the change is not breaking, the configurations that worked before would still work, but they would be far less elegant and efficient at solving the issue, or would even be missing some of the improvements.

That being said, OpenSMTPD is a very active project with a release cycle of six months and we'll have breaking changes every now and then because we'd rather have users change a keyword than let technical debt accumulate.

We have had a very big breaking change a year ago where the configuration file was completely revamped, this was the first time since the project was first started ten years ago, so a huge breaking change should not happen before another decade :-) but it's likely that within the next year, we'll change a keyword here or there to help improve the project.

[cmouse]

I'd like to suggest incorporating this in your howto:

OpenSMTPD could use LMTP (or execute dovecot-lda, not sure which works in OpenSTMPD) to deliver mail via dovecot, the benefit is that then dovecot updates it's indexes immediately, and you can benefit from sieve filtering for incoming mail. Then you can make dovecot filter e.g. spam messages to Spam folder directly.

I'd rather see people write such howtos ;-)

Perhaps. But I think it would make sense to involve dovecot on the delivery phase always, as it would enable much more functionality in dovecot and would allow dovecot to update it's indexes during delivery.

[jirib]

My only concern for HA mail setup are mails waiting in a queue to be delivered, eg. a mail server in DMZ with paused delivery aka "deffered relaying" (cf 9.2.1 Postponing Mail Delivery in Postfix: Definitive Guide; would that be doable with smtpd -P mta?) and which waits for a manual trigger to try to deliver via SMTP to a mail server in intranet. If the mail server with this setup would fail, the mails will be lost. IIRC 'extras' used to allow Python plugins which could save mails into redis etc. to have a queue "distributed"...