Agent attempts to remove mounted files

[sjberman]

As described in the discussion linked below, users may want to mount extra files via volumes into the NGINX Pod, but Agent is causing issues with this. Agent will attempt to remove any nginx files in the system that aren't provided by our control plane, to ensure our control plane is the source of truth. We do have a method to inform Agent of files that it should ignore, by setting an Unmanaged: true flag in the file metadata, as we do with some other files that are already present on the container (not provided by our control plane). We should look into a way to include any volume-mounted files in the list of ignored files so that Agent doesn't try to remove them.

Acceptance

• Any files that are provided via extra volume mounts to the nginx Pod should be marked as Unmanaged so that Agent doesn't attempt to delete them.

Discussed in #3775

^{Originally posted by michasHL August 21, 2025}
Hello NGF team,

we got another odd case where NGF is having problems applying the NGF config.
We're mounting a root certificate into the data plane deployment by using the following values in the helm chart:

nginx:
  pod:
    volumes:
    - name: root-ca
        secret:
          secretName: secret
  container:
    volumeMounts:
    - name: root-ca
        mountPath: /etc/nginx/certs/

The SnippetsFilter looks like this:

apiVersion: gateway.nginx.org/v1alpha1
kind: SnippetsFilter
metadata:
  name: test
spec:
  snippets:
  - context: http.server
    value: |
      client_header_buffer_size 2k;
      ssl_client_certificate /etc/nginx/certs/root-ca.crt;
      ssl_verify_client optional;
      ssl_verify_depth 2;

      proxy_connect_timeout 60s;
      proxy_read_timeout 60s;
      proxy_send_timeout 60s;
  - context: http.server.location
    value: "proxy_set_header x-ssl-client-cert $ssl_client_escaped_cert;\n\nlocation
      ~ ^(?!(/status/live|/canary/status/live|/instance/.+/status/live|/status/ready|/canary/status/ready|/instance/.+/status/ready))
      { \n  if ($ssl_client_verify != SUCCESS) {\n    return 403;\n  }\n}\n\nset $smp
      \"${request_method}${uri}\";\nif ( $smp ~* ^(?!(GET/status/live|GET/canary/status/live|GET/instance/.+/status/live|GET/status/ready|GET/canary/status/ready|GET/instance/.+/status/ready|))
      ) {\n  return 403;\n}\n"

After rolling the control plane, the gateway goes into programmed state for a while. It seems upon the first change we're seeing this in the Gateway resource:
Message: The Gateway is not programmed due to a failure to reload nginx with the configuration: msg: Config apply failed, rolling back config; error: error deleting file: /etc/nginx/certs/root-ca.crt error: remove /etc/nginx/certs/root-ca.crt: read-only file system

Now of course, the simple answer might be to just move the cert to a different folder, because maybe it doesn't like that the cert file is mounted to /etc/nginx/. Otherwise, I'm a little baffled because secret volumes always seem to be added as read-only: https://kubernetes.io/docs/concepts/storage/volumes/#secret.

[bjee19]

After some investigation, it was decided that this story might be unavailable to complete with the current apis presented to us by NGINX Agent. We are moving forwards with asking the Agent team to support an enhancement allowing us to mark entire directories as Unmanaged and have agent ignore them. Until then, this issue is blocked. I will keep this issue up to date with timelines on Agent.