automatically update istio-ca-root-cert when it is updated

[lcopi]

Ztunnel loads the Istio CA root certificate once at startup and never reloads it. When the root cert is rotated (e.g. via the istio-ca-root-cert ConfigMap), existing ztunnel pods must be restarted to pick up the new cert. This causes UnknownIssuer errors during CA rotation until pods are restarted.

Instead ztunnel should use a file watcher approach similar to src/tls/crl.rs so when istio-ca-root-cert is rotated, ztunnel will pick up the changes automatically.

Example logs when a rotation happens without restarting ztunnel:

error    cert_fetcher    unable to prefetch cert for "spiffe://cluster.local/ns/flux-system/sa/source-controller", skipping, SigningRequest(Status { code: Unknown, message: "client error (Connect)", source: Some(client error (Connect)

Caused by:
    invalid peer certificate: UnknownIssuer

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
   1: <ztunnel::tls::control::TlsGrpcChannel as tower_service::Service<http::request::Request<tonic::body::Body>>>::call::{{closure}}
   2: ztunnel::identity::caclient::CaClient::fetch_certificate::{{closure}}::{{closure}}
   3: <ztunnel::identity::caclient::CaClient as ztunnel::identity::manager::CaClientTrait>::fetch_certificate::{{closure}}
   4: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
   5: ztunnel::identity::manager::Worker::new::{{closure}}
   6: tokio::runtime::task::raw::poll
   7: ztunnel::main
   8: std::sys::backtrace::__rust_begin_short_backtrace
   9: main
  10: __libc_start_call_main
  11: __libc_start_main_impl
  12: _start) })

[jlojosnegros]

Hi all,

I would like to work on this issue if nobody else is doing it.

Thanks in advance

[jlojosnegros]

/assign

[keithmattix]

@jlojosnegros assigned it to you. Good luck!

[jlojosnegros]

The fix follows the same pattern already used by CrlManager for CRL hot-reload:
a filesystem watcher sets an atomic dirty flag, and the affected resource is rebuilt lazily on the next use rather than eagerly on every file event.

Concretely, a new RootCertManager watches the root cert directory and marks itself dirty on any change.
CaClient checks this flag at the start of each fetch_certificate call and rebuilds the gRPC TLS channel if needed.
Only RootCert::File gets a watcher; static and system-bundle certs are unaffected.

[jlojosnegros]

PR #1775 is up and ready for review whenever you have time.

Happy to address any feedback.