Parameters to handle "high" load on smtp_server component of postal

[arthurzenika]

We're looking into an incident where it seems that the smtp-server component of postal had difficulty handling an unusual load on our servers. We have a setup where postal is running behind an AWS NLB load balancer, with two fairly large EC2 instance (r5n.large) and we had a large spike of connexions to the SMTP server where the clients trying to send email got some Network time out errors.

Using the collected metrics from prometheus we can see that the postal_smtp_server_connections_total was reaching a max of 3.6 cnx/s (which is 215 cnx/s) for each smtp_server. That smtp server cannot be easily scaled up like the workers since it is listening on a single port on the linux hosts. The linux metrics seems all OK, there doesn't seem to be any saturation on that side, the only strange metric is the number of "Time Wait" TCP sockets which spiked at 150.

We couldn't find any relevant parameters in the documentation. Are there any parameters that can be tweaked for more threads to be given to that component or more network capabilities (in ruby on rails or in the docker-compose settings) ?

We are looking into using https://github.com/iamd3vil/smtpbench to load test different settings to see if changing settings have an effect.

[willpower232]

I know others have talked about threads on the smtp server before but I can't remember off the top of my head where they got up to.

There are a lot of words and linked things here #751

[arthurzenika]

The PR is work on these sorts of issues but it has not been integrated in postal and since then NIO has been used in the smtp-server (latest code modifications by @adamcooke). I have not found any documentation in https://github.com/socketry/nio4r that indicates a way to improve the number of threads used. Maybe some with better ruby skills can find something ?

We have setup a proof of concept with two smtp-server components listening on two different ports behind the NLB, and the throughput goes from 25 msg/s to 34 msg/s and 8s average latency to à 5s, with identical specs underneath, this would seem to confirm that there is a bottleneck there.