Some nginx related changes.

[oxpa]

Hello,

I saw you post at nginx community forum and decided to get involved:)

open file cache is mostly used to battle VFS locks. Those will matter if you have a lot of open()'s along with reads and writes (usually happen with proxy_pass + temp files + closer to 1kk rps). And these were mostly solved in linux kernel since open file cache was introduced into nginx.

reuseport may cause some security "issues". As in "any application can listen to the same port as nginx and have its traffic". Not a great concern in most cases. But it ensures better (though still not ideal) load distribution across workers

If your nginx instance has a CDN in front of it - keepalive timeouts should be increased significantly. Otherwise you are reopening connections every 1000 requests (which may happen within milliseconds)

multi accept works well in synthetic tests and can easily fail you in real life.

rlimit nofile is used to make nginx get set the limit on open files properly (basically - test that OS limits are correct). If you have 1 client connection it may open 1 upstream connection and 1 file (cache or proxy temp). Both client and upstream connections are accounted into worker connections. So if you have a limit of 20k connections it gives you 10k clients with proxy, 20k clients without proxy, up to 30k open files when there is proxy, up to 40k files without proxy.

Sendfile only works if there is no postprocessing in the application. SSL, gzip, any other filters require application logic that effectively disables sendfile.

gzip on along with gzip level 1 is a wise choice though :) But if files are pregzipped - it can probably be disabled. Check out https://nginx.org/en/docs/http/ngx_http_gzip_static_module.html . Not sure how applicable to you this is.

I would also get rid of RE locations. Or try to hide them inside appropriate prefix locations. But this requires careful testing before implementation.

I didn't test the config but I'm happy to answer any questions you may have.

Cheers!

[oxpa]

Missed two things:
return 444 - closing connection may result in positive feedback loops and create connection storms. return 204 is usually a better choice.
reject ssl handshake is a more "modern" way of having null ciphers and no real key/cert. But it should also work without cert/key configured. Though this may be a false memory:)