OCPBUGS-100074: Use /healthz for router startup probe

[jcmoraisjr]

The startup probe previously used /healthz/ready, which requires the router to have synced its first route within the 2-minute failure threshold. On resource-constrained clusters or during slow provisioning, the bootstrap process can exceed this window, causing the router pod to crashloop.

Switching the startup probe to /healthz checks only that the router process is alive, allowing the readiness probe to gate traffic until the initial sync completes. This lets the router remain not-ready during a slow bootstrap rather than restarting repeatedly.

https://redhat.atlassian.net/browse/OCPBUGS-100074

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@jcmoraisjr: This pull request references Jira Issue OCPBUGS-100074, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

The startup probe previously used /healthz/ready, which requires the router to have synced its first route within the 2-minute failure threshold. On resource-constrained clusters or during slow provisioning, the bootstrap process can exceed this window, causing the router pod to crashloop.

Switching the startup probe to /healthz checks only that the router process is alive, allowing the readiness probe to gate traffic until the initial sync completes. This lets the router remain not-ready during a slow bootstrap rather than restarting repeatedly.

https://redhat.atlassian.net/browse/OCPBUGS-100074

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Warning

Review limit reached

@jcmoraisjr, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 48 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 18cc3fc6-2cec-46d3-bc3a-4d0dd96707b0

📥 Commits

Reviewing files that changed from the base of the PR and between 5fbdca3 and c8b5e95.

📒 Files selected for processing (2)

• pkg/manifests/assets/router/deployment.yaml
• pkg/operator/controller/ingress/deployment_test.go

📝 Walkthrough

Walkthrough

The router container’s startup probe HTTP GET path was changed from /healthz/ready to /healthz in the deployment manifest.

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Container-Privileges	❌ Error	pkg/manifests/assets/router/deployment.yaml sets allowPrivilegeEscalation: true in the haproxy init container, which this check forbids.	Remove allowPrivilegeEscalation: true from the manifest or add a clear justification if it must remain.

✅ Passed checks (14 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: switching the router startup probe to /healthz.
Description check	✅ Passed	The description matches the change and explains why moving the startup probe to /healthz helps avoid crashloops.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR only changes router deployment YAML startupProbe path; no Ginkgo tests or test titles were added or renamed, so no unstable names are present.
Test Structure And Quality	✅ Passed	No Ginkgo test code changed; the PR only updates a router manifest probe path, so the test-structure checklist is not applicable.
Microshift Test Compatibility	✅ Passed	Only pkg/manifests/assets/router/deployment.yaml changed; no new Ginkgo tests or MicroShift-unsupported APIs/features were added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	Only a router deployment manifest changed; no new Ginkgo/e2e tests or SNO-related assumptions were added.
Topology-Aware Scheduling Compatibility	✅ Passed	Only the router startupProbe path changed; no replicas, nodeSelectors, affinity, spread constraints, or topology-specific scheduling logic were added.
Ote Binary Stdout Contract	✅ Passed	Only a router manifest probe path changed; no process-level binary code or stdout logging was modified.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR only changes a router deployment manifest; no Ginkgo/e2e tests were added or modified, so the IPv4/disconnected test check is not applicable.
No-Weak-Crypto	✅ Passed	The only change is a router startupProbe path in a YAML manifest; no weak crypto, custom crypto, or secret comparisons are present.
No-Sensitive-Data-In-Logs	✅ Passed	PR only changes the router startupProbe path in a manifest; no logging code or sensitive-data exposure was added.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/manifests/assets/router/deployment.yaml`:
- Line 116: Update the startup-probe path assertion in the deployment test to
expect /healthz, matching the manifest; leave the readiness-probe assertion
expecting /healthz/ready.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 56607fdd-bb6c-40cc-8daa-0dcc6243c93c

📥 Commits

Reviewing files that changed from the base of the PR and between 90f2833 and 5fbdca3.

📒 Files selected for processing (1)

• pkg/manifests/assets/router/deployment.yaml

[gcs278]

/assign

[jcmoraisjr]

/test hypershift-e2e-aks

[jcmoraisjr]

/pipeline required

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-operator
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-upgrade
/test e2e-azure-operator
/test e2e-gcp-operator
/test e2e-hypershift
/test e2e-vsphere-static-metallb-operator-gwapi
/test e2e-vsphere-static-metallb-operator-gwapi-techpreview

[melvinjoseph86]

/retest-required

[melvinjoseph86]

Tried using cluster bot

➜    oc get deployment router-default -n openshift-ingress -o jsonpath='{.spec.template.spec.containers[?(@.name=="router")].startupProbe}' | jq .
{
  "failureThreshold": 120,
  "httpGet": {
    "path": "/healthz",
    "port": 1936,
    "scheme": "HTTP"
  },
  "periodSeconds": 1,
  "successThreshold": 1,
  "timeoutSeconds": 1
}
➜   oc get deployment router-default -n openshift-ingress -o jsonpath='{.spec.template.spec.containers[?(@.name=="router")].readinessProbe}' | jq .
{
  "failureThreshold": 3,
  "httpGet": {
    "path": "/healthz/ready",
    "port": 1936,
    "scheme": "HTTP"
  },
  "periodSeconds": 10,
  "successThreshold": 1,
  "timeoutSeconds": 1
}

Checked with normal reboot of the router pod and there is no Unhealthy or Killing events seen, related to the startup probe.

But not able to stimulate a resource-constrained environment, as it crashed by terminal when tried to stimulate the same.

Tried a more simple one  by creating a test IngressController and watch startup phases
➜  oc create -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
  name: test-startup-probe
  namespace: openshift-ingress-operator
spec:
  domain: test-startup-probe.example.com
  endpointPublishingStrategy:
    type: Private
  replicas: 1
EOF
ingresscontroller.operator.openshift.io/test-startup-probe created

➜  Downloads  oc get pods -n openshift-ingress -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=test-startup-probe -w
NAME                                        READY   STATUS    RESTARTS   AGE
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Pending   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Pending   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Pending   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:1/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:1/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     PodInitializing   0          2s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     PodInitializing   0          2s
router-test-startup-probe-dcbddfcf8-frzrc   1/2     PodInitializing   0          2s
router-test-startup-probe-dcbddfcf8-frzrc   1/2     Running           0          2s
router-test-startup-probe-dcbddfcf8-frzrc   1/2     Running           0          3s
router-test-startup-probe-dcbddfcf8-frzrc   2/2     Running           0          3s
router-test-startup-probe-dcbddfcf8-frzrc   2/2     Running           0          4s

No CrashLoopBackOff or restarts

➜  Downloads POD=$(oc get pods -n openshift-ingress -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=test-startup-probe -o jsonpath='{.items[0].metadata.name}')
  oc describe pod $POD -n openshift-ingress | grep -E "Unhealthy|Killing|Back-off"

No output, hence no probe failures.

Marking as verified
/verified by @mjoseph

[openshift-ci-robot]

@melvinjoseph86: This PR has been marked as verified by @mjoseph.

Details

In response to this:

Tried using cluster bot

➜    oc get deployment router-default -n openshift-ingress -o jsonpath='{.spec.template.spec.containers[?(@.name=="router")].startupProbe}' | jq .
{
 "failureThreshold": 120,
 "httpGet": {
   "path": "/healthz",
   "port": 1936,
   "scheme": "HTTP"
 },
 "periodSeconds": 1,
 "successThreshold": 1,
 "timeoutSeconds": 1
}
➜   oc get deployment router-default -n openshift-ingress -o jsonpath='{.spec.template.spec.containers[?(@.name=="router")].readinessProbe}' | jq .
{
 "failureThreshold": 3,
 "httpGet": {
   "path": "/healthz/ready",
   "port": 1936,
   "scheme": "HTTP"
 },
 "periodSeconds": 10,
 "successThreshold": 1,
 "timeoutSeconds": 1
}

Checked with normal reboot of the router pod and there is no Unhealthy or Killing events seen, related to the startup probe.

But not able to stimulate a resource-constrained environment, as it crashed by terminal when tried to stimulate the same.

Tried a more simple one  by creating a test IngressController and watch startup phases
➜  oc create -f - <<EOF
apiVersion: operator.openshift.io/v1
kind: IngressController
metadata:
 name: test-startup-probe
 namespace: openshift-ingress-operator
spec:
 domain: test-startup-probe.example.com
 endpointPublishingStrategy:
   type: Private
 replicas: 1
EOF
ingresscontroller.operator.openshift.io/test-startup-probe created

➜  Downloads  oc get pods -n openshift-ingress -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=test-startup-probe -w
NAME                                        READY   STATUS    RESTARTS   AGE
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Pending   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Pending   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Pending   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          0s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:0/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:1/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     Init:1/2   0          1s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     PodInitializing   0          2s
router-test-startup-probe-dcbddfcf8-frzrc   0/2     PodInitializing   0          2s
router-test-startup-probe-dcbddfcf8-frzrc   1/2     PodInitializing   0          2s
router-test-startup-probe-dcbddfcf8-frzrc   1/2     Running           0          2s
router-test-startup-probe-dcbddfcf8-frzrc   1/2     Running           0          3s
router-test-startup-probe-dcbddfcf8-frzrc   2/2     Running           0          3s
router-test-startup-probe-dcbddfcf8-frzrc   2/2     Running           0          4s

No CrashLoopBackOff or restarts

➜  Downloads POD=$(oc get pods -n openshift-ingress -l ingresscontroller.operator.openshift.io/deployment-ingresscontroller=test-startup-probe -o jsonpath='{.items[0].metadata.name}')
 oc describe pod $POD -n openshift-ingress | grep -E "Unhealthy|Killing|Back-off"

No output, hence no probe failures.

Marking as verified
/verified by @mjoseph

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[gcs278]

We discussed this in the bug status call today. I think this is a good fix, but I'm hesitant to merge something that isn't driven by a production bug this late in the 5.0 release, especially since we've been relying on this crashloop behavior since 4.9. For example, what if there's a bug in the router's internal API retry logic that we haven't caught precisely because the crashloop has been clearing state and masking it?

Let's hold off until after branching — unless something comes up that gives us a concrete reason to prioritize it

/approve
/lgtm
/hold

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-operator
/test e2e-aws-ovn
/test e2e-aws-ovn-hypershift-conformance
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-upgrade
/test e2e-azure-operator
/test e2e-gcp-operator
/test e2e-hypershift
/test e2e-vsphere-static-metallb-operator-gwapi
/test e2e-vsphere-static-metallb-operator-gwapi-techpreview

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gcs278

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [gcs278]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@jcmoraisjr: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-operator	c8b5e95	link	true	/test e2e-aws-operator
ci/prow/e2e-aws-ovn-hypershift-conformance	c8b5e95	link	true	/test e2e-aws-ovn-hypershift-conformance
ci/prow/e2e-aws-ovn	c8b5e95	link	true	/test e2e-aws-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bentito]

@redhat-chai-bot are the e2e check fails due to infra fails?