Proposal: make state_path required

[jsha]

Hiya! Super excited to hear about this project. It will make certificate management so much easier for so many people.

I'm an engineer at Let's Encrypt (and author on RFC 8555), and based on our deployment experience, I'd recommend that state_path should be required. If it's optional, I think many people will leave it empty for simplicity's sake, not realizing that it's a major availability hazard.

We work hard to be highly available, but we do have outages, including a recent one in July that was several hours long. When that happens, all new issuance will naturally fail, but existing certificates work just fine. For an Nginx user with no state_path, our recent outage would have turned a routine Nginx restart into a complete site outage, because Nginx would not have been able to get a certificate on start.

For reference, the Caddy web server had the behavior that it would refuse to start unless it had a currently-valid certificate, which led to many operators having a surprise outage (caused by a Let's Encrypt outage) in 2017: caddyserver/caddy#1680.

Additionally, users who configure no state_path will run into rate limits very quickly if they restart their server several times in succession (5 times in 7 days). This can cause a site outage even without a corresponding Let's Encrypt outage.

I recognize there are some potential use cases where no writeable disk is available. I believe those use cases would be best served with tmpfs, or possibly an explicit config option like no_certificate_persistence, which could have the availability risks documented with it. The advantage of tmpfs is that files could persist across Nginx restarts even if not across OS restarts.

Thanks again for your work on supporting ACME. Happy to chat anytime.

[bavshin-f5]

That is a valid concern. Thanks for highlighting the importance.

I'll confess that the only reasons we don't have persistent storage enabled by default are a lack of a good platform-dependent default path and a general inconvenience of passing build configuration options to a Rust project.
For all the other cache or state path directives, we have ./configure --with-${feature}-path switches and prefix-relative defaults, so the caches are on by default (e.g. --http-proxy-temp-path=/var/cache/nginx/proxy_temp). Rather than requiring the state_path, it would be more consistent to set a default value in a similar manner.

I discussed this problem with our release engineering team, and we came up with the following plan:

• Pass NGINX_ACME_STATE_PREFIX environment variable during build to set a platform-specific state directory path (/var/cache/nginx, /var/db/nginx, etc).
• Set default value for state_path to {state_prefix}/acme-{issuer_name}.
• Add a special value state_path off to revert to the current stateless behavior.