Add missing Pod IPs to created certificates

[sbernauer]

Some tools call the Pods via their PodIP, e.g. Prometheus when using a ServiceMonitor.
For this to work with HTTPS-enabled products, we need to add the Pod IP to the certificate because

1. The clients should be able to validate the cert
2. Some products (currently only NiFi) raise an "Invalid SNI" error if they can not find a matching cert in it's keystore

There is currently already code for this.
The Pod IPs are gathered here:

secret-operator/rust/operator-binary/src/backend/pod_info.rs

Lines 142 to 153 in e5224ab

	// This will generally be empty, since Kubernetes assigns pod IPs *after* CSI plugins are successful
	pod_ips: pod
	.status
	.iter()
	.flat_map(|status| &status.pod_ips)
	.flatten()
	.map(|ip| &ip.ip)
	.map(|ip| {
	ip.parse()
	.context(from_pod_error::IllegalAddressSnafu { address: ip })
	})
	.collect::<Result<_, _>>()?,

and stuffed into the certificate here

secret-operator/rust/operator-binary/src/backend/mod.rs

Line 229 in e5224ab

addrs.extend(pod_info.pod_ips.iter().copied().map(Address::Ip));

However the comment already highlights the problem:

This will generally be empty, since Kubernetes assigns pod IPs after CSI plugins are successful

Because of this, we are lacking SAN entries for the Pod IPs.

I don't know if this "is even possible" with our current architecture of secret-operator being a CSI driver.

So I though "maybe listener-op can help"? Not from the top of my head, as it itself is "only" a CSI driver, so the Pod has no IPs assigned when it is running as well. listener-op can currently only block Pod creation until a Service (such as a LoadBalancer) has an address assigned. But there might be some other clever way how listener-op can do this which I didn't though of.

Another though is that certificate hot-reloading should work, because we can add the IP to the cert after the Pod has an IP assigned.
But that's a bigger story - e.g. do all tools support this?
Run secret-op as init container? As sidecar?

[lfrancke]

Leaving this reference here:

• Add option to skip blocking pod startup if driver is not ready to create a request yet cert-manager/csi-lib#20
• ability to specify pod IP in volume attributes cert-manager/csi-driver#17