NitroEnclave crashes Apache

[dmitri--]

Whenever I restart nutro.enclave.acm service Apache fails to start with the following errors then start up normally.
This issue causes more than a minute downtime in Web service -- not so good for production

# journalctl -u httpd.service
Mar 26 01:04:32 s2.bbb.com systemd[1]: Stopping httpd.service - The Apache HTTP Server...
Mar 26 01:05:32 s2.bbb.com systemd[1]: httpd.service: Deactivated successfully.
Mar 26 01:05:32 s2.bbb.com systemd[1]: Stopped httpd.service - The Apache HTTP Server.
Mar 26 01:05:32 s2.bbb.com systemd[1]: httpd.service: Consumed 50.705s CPU time.
Mar 26 01:05:32 s2.bbb.com systemd[1]: Starting httpd.service - The Apache HTTP Server...
Mar 26 01:05:32 s2.bbb.com httpd[246961]: AH00526: Syntax error on line 49 of /etc/httpd/conf.d/xxx.conf:
Mar 26 01:05:32 s2.bbb.com httpd[246961]: SSLCertificateFile: file '/run/nitro_enclaves/acm/httpd-cert-yyy.pem' does not exist or is empty
Mar 26 01:05:32 s2.bbb.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 01:05:32 s2.bbb.com systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 26 01:05:32 s2.bbb.com systemd[1]: Failed to start httpd.service - The Apache HTTP Server.
Mar 26 01:05:42 s2.bbb.com systemd[1]: httpd.service: Scheduled restart job, restart counter is at 1.
Mar 26 01:05:42 s2.bbb.com systemd[1]: Stopped httpd.service - The Apache HTTP Server.
Mar 26 01:05:42 s2.bbb.com systemd[1]: Starting httpd.service - The Apache HTTP Server...
Mar 26 01:05:42 s2.bbb.com httpd[247064]: AH00526: Syntax error on line 54 of /etc/httpd/conf.d/zzz.conf:
Mar 26 01:05:42 s2.bbb.com httpd[247064]: SSLCertificateFile: file '/run/nitro_enclaves/acm/httpd-cert-aaa.pem' does not exist or is empty
Mar 26 01:05:42 s2.bbb.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 01:05:42 s2.bbb.com systemd[1]: httpd.service: Failed with result 'exit-code'.
Mar 26 01:05:42 s2.bbb.com systemd[1]: Failed to start httpd.service - The Apache HTTP Server.
Mar 26 01:05:52 s2.bbb.com systemd[1]: httpd.service: Scheduled restart job, restart counter is at 2.
Mar 26 01:05:52 s2.bbb.com systemd[1]: Stopped httpd.service - The Apache HTTP Server.
Mar 26 01:05:53 s2.bbb.com systemd[1]: Starting httpd.service - The Apache HTTP Server...
Mar 26 01:05:54 s2.bbb.com systemd[1]: Started httpd.service - The Apache HTTP Server.
Mar 26 01:05:54 s2.bbb.com httpd[247107]: Server configured, listening on: port 443, port 80

corresponding logs of nitro

# journalctl -u nitro-enclaves-acm.service
Mar 26 01:05:32 s2.bbb.com p11ne-agent[44629]: |INFO  | Setting exit condition
Mar 26 01:05:32 s2.bbb.com p11ne-agent[44629]: |INFO  | Killing enclave pid=44641
Mar 26 01:05:32 s2.bbb.com p11ne-agent[44629]: |INFO  | Cleaning up p11kit config
Mar 26 01:05:32 s2.bbb.com systemd[1]: Stopping nitro-enclaves-acm.service - Nitro Enclaves ACM Agent...
Mar 26 01:05:32 s2.bbb.com systemd[1]: nitro-enclaves-acm.service: Deactivated successfully.
Mar 26 01:05:32 s2.bbb.com systemd[1]: Stopped nitro-enclaves-acm.service - Nitro Enclaves ACM Agent.
Mar 26 01:05:32 s2.bbb.com systemd[1]: nitro-enclaves-acm.service: Consumed 8min 16.930s CPU time.
Mar 26 01:05:32 s2.bbb.com systemd[1]: Starting nitro-enclaves-acm.service - Nitro Enclaves ACM Agent...
Mar 26 01:05:32 s2.bbb.com systemd[1]: Started nitro-enclaves-acm.service - Nitro Enclaves ACM Agent.
Mar 26 01:05:33 s2.bbb.com p11ne-agent[246960]: |INFO  | Setting up p11-kit config
Mar 26 01:05:33 s2.bbb.com p11ne-agent[246960]: |INFO  | Restarting vsock proxy
Mar 26 01:05:37 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token xxx-acm-token
Mar 26 01:05:40 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token yyy-acm-token
Mar 26 01:05:41 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token zzz-acm-token
Mar 26 01:05:42 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token aaa-acm-token
Mar 26 01:05:43 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token ccc-acm-token
Mar 26 01:05:44 s2.bbb.com p11ne-agent[246960]: |INFO  | Service: httpd | Force_Start: false | Reload: 0 | Sync: 600
Mar 26 01:05:44 s2.bbb.com p11ne-agent[246960]: |INFO  | Reloading HTTPD configuration.
Mar 26 01:05:44 s2.bbb.com p11ne-agent[246960]: |WARN  | Unable to reload HTTPD: it is not running and 'force_start' option is disabled.
Mar 26 01:15:38 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token xxx-acm-token
Mar 26 01:15:40 s2.bbb.com p11ne-agent[246960]: |INFO  | Refreshing token xxx-acm-token
Mar 26 01:15:41 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token yyy-acm-token
Mar 26 01:15:42 s2.bbb.com p11ne-agent[246960]: |INFO  | Refreshing token yyy-acm-token
Mar 26 01:15:42 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token zzz-acm-token
Mar 26 01:15:43 s2.bbb.com p11ne-agent[246960]: |INFO  | Refreshing token zzz-acm-token
Mar 26 01:15:44 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token aaa-acm-token
Mar 26 01:15:44 s2.bbb.com p11ne-agent[246960]: |INFO  | Refreshing token aaa-acm-token
Mar 26 01:15:45 s2.bbb.com p11ne-agent[246960]: |INFO  | Syncing token ccc-acm-token
Mar 26 01:15:46 s2.bbb.com p11ne-agent[246960]: |INFO  | Refreshing token ccc-acm-token
Mar 26 01:15:46 s2.bbb.com p11ne-agent[246960]: |INFO  | Service: httpd | Force_Start: false | Reload: 0 | Sync: 600
Mar 26 01:15:46 s2.bbb.com p11ne-agent[246960]: |INFO  | Reloading HTTPD configuration.

Clearly nitro service does not restore original SSLCertificateFile lines in two config files out of 5 and tries to restart HTTP, or there is a race between config updating and service restart.

Steps to reproduce:

• create httpd (apache) web server with 5 virtual hosts, each has its own config file which is auto-loaded from /etc/httpd/conf.d/ directory
• configure 5 certs in ACM
• configure nitro enclave for ACM
• start nitro enclave and httpd
• restart nitro using sudo systemctl restart nitro-enclaves-acm.service

ec2 type c6g.xlarge
nitro is configured to take 1 core and 256MB of memory