Add webhook readiness check to operator /readyz endpoint (#4778)

* Add webhook readiness check to operator /readyz endpoint

Add a "webhook" readyz check using controller-runtime's
StartedChecker() which verifies the webhook TLS listener is
accepting connections. The check is only registered when webhooks are
enabled, so webhooks-disabled deployments are unaffected.

* update script in chainsaw test

* use kubectl wait in tests

Author: jinja2

.chloggen/webhook-readiness-check.yaml

@@ -0,0 +1,20 @@
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the component, or a single word describing the area of concern, (e.g. collector, target allocator, auto-instrumentation, opamp, github action)
+component: operator
+
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add webhook server readiness check to the operator's /readyz endpoint so the pod is not marked ready before the webhook server is listening.
+
+# One or more tracking issues related to the change
+issues: [3772]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext: |
+  Previously the readiness probe used only healthz.Ping, causing a race where CRs 
+  created right after deployment could hit "connection refused" from the webhook.
+  Now the readyz endpoint includes a check using controller-runtime's
+  StartedChecker which verifies the webhook TLS listener is actually accepting connections.

Makefile

@@ -321,7 +321,7 @@ add-rbac-permissions-to-operator: manifests kustomize
 .PHONY: deploy
 deploy: set-image-controller
 	$(KUSTOMIZE) build config/default | kubectl apply -f -
-	go run hack/check-operator-ready.go 300
+	kubectl wait --for=condition=available deployment/opentelemetry-operator-controller-manager -n opentelemetry-operator-system --timeout=300s
 
 # Undeploy controller in the current Kubernetes context, configured in ~/.kube/config
 .PHONY: undeploy

main.go

@@ -529,6 +529,12 @@ func main() {
 		setupLog.Error(err, "unable to set up ready check")
 		os.Exit(1)
 	}
+	if cfg.EnableWebhooks {
+		if err := mgr.AddReadyzCheck("webhook", mgr.GetWebhookServer().StartedChecker()); err != nil {
+			setupLog.Error(err, "unable to set up webhook ready check")
+			os.Exit(1)
+		}
+	}
 
 	setupLog.Info("starting manager")
 	// NOTE: We enable LeaderElectionReleaseOnCancel, and to be safe we need to exit right after the manager does