fix(proxy): promote shutdown-manager to native sidecar to close preStop drain race on k8s 1.33+

[ChrisJBurns]

On Kubernetes 1.33+, envoy and shutdown-manager are ordinary sibling containers with no termination ordering guarantee. When kubelet sends SIGTERM during a pod rollout, both containers receive it simultaneously.

Envoy's preStop hook (httpGet :19002/shutdown/ready) is designed to block until shutdown-manager confirms that in-flight connections have drained. But because shutdown-manager can exit before the hook completes its polling loop, the hook fails immediately with FailedPreStopHook rather than waiting out the drain window. Kubelet then kills envoy with requests still in flight, producing client-visible 5xx errors on every proxy rollout.

Confirmed on: Envoy AI Gateway v1.7.0, Kubernetes 1.33

Reproduce

1. Deploy a proxy with active traffic
2. Trigger a proxy pod rollout (update EnvoyProxy or any setting causing pod recreation)
3. Observe FailedPreStopHook events followed by connection-refused on port 19002:

Warning  FailedPreStopHook  pod/envoy-<hash>  preStop hook for container "envoy" failed

Surfaced during a routine gateway rollout — a cluster of 5xx responses traceable to FailedPreStopHook events immediately followed by refused connections to the shutdown-manager port.

Expected behaviour

shutdown-manager should outlive envoy so the preStop drain handshake always completes before kubelet kills the proxy process.

Proposed fix

On Kubernetes 1.33+, promote shutdown-manager from spec.containers to spec.initContainers with restartPolicy: Always. This makes it a native sidecar, which kubelet guarantees will keep running until all non-sidecar containers (envoy) have fully exited, closing the race entirely.

The change belongs in the proxy pod-spec builder (roughly internal/infrastructure/kubernetes/proxy/resource.go) behind a version gate:

if k8sVersion >= 1.33 {
    shutdownMgr.RestartPolicy = ptr.To(corev1.ContainerRestartPolicyAlways)
    spec.InitContainers = append(spec.InitContainers, shutdownMgr)
    // remove from spec.Containers
} else {
    spec.Containers = append(spec.Containers, shutdownMgr)
}

Why not make the preStop hook retry-tolerant?

If shutdown-manager has already exited, drain has not completed and Envoy should not be killed. Masking a missing shutdown-manager in the hook would paper over the race rather than fix it. The correct invariant is that shutdown-manager must be alive for the entire duration of Envoy's preStop window.

Notes

• This requires Kubernetes 1.33+ and should be a no-op on older clusters
• It would be useful to export shutdownManagerContainerName (and envoyContainerName) as package-level constants — downstream consumers currently have to hardcode "shutdown-manager" as a bare string with no compile-time contract