OCPBUGS-61508: IngressOperator not exposing some metrics for degraded… #1290

davidesalerno · 2025-10-06T09:56:58Z

… IngressController

This PR is fixing the issue we have got into the ingress controller that is not exposing ingress_operator_conditions metrics for existing ingress controllers if the operator pod is restarted till the ingress controllers status don't change.

We can achieve this simply moving the SetIngressControllerConditionsMetric away from the code block inside the if statement checking if the ingress status is changed or not.

Added an e2e test that is verifying that the metrics are there before and after the operator pod restart. The e2e test can't be executed in parallel with other tests because it is restarting the ingress operator pod and so it could affect in a non predictable way their results.

openshift-ci-robot · 2025-10-06T09:57:05Z

@davidesalerno: This pull request references Jira Issue OCPBUGS-61508, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

… IngressController

This PR is fixing the issue we have got into the ingress controller that is not exposing ingress_operator_conditions metrics for existing ingress controllers if the operator pod is restarted till the ingress controllers status don't change.

We can achieve this simply moving the SetIngressControllerConditionsMetric away from the code block inside the if statement checking if the ingress status is changed or not.

Added an e2e test that is verifying that the metrics are there before and after the operator pod restart. The e2e test can't be executed in parallel with other tests because it is restarting the ingress operator pod and so it could affect in a non predictable way their results.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-10-06T09:58:14Z

@davidesalerno: This pull request references Jira Issue OCPBUGS-61508, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

davidesalerno · 2025-10-06T14:31:48Z

/retest

test/e2e/ic_conditions_metric_test.go

davidesalerno · 2025-10-08T15:06:20Z

/retest

candita · 2025-10-09T22:16:57Z

/assign @alebedev87
/asssign @grzpiotrowski

candita · 2025-10-09T22:17:57Z

/assign @grzpiotrowski

davidesalerno · 2025-10-10T07:50:46Z

/jira refresh

openshift-ci-robot · 2025-10-10T07:50:51Z

@davidesalerno: This pull request references Jira Issue OCPBUGS-61508, which is invalid:

expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

davidesalerno · 2025-10-10T12:11:07Z

/test okd-scos-e2e-aws-ovn

davidesalerno · 2025-10-10T12:11:23Z

/jira refresh

openshift-ci-robot · 2025-10-10T12:11:29Z

@davidesalerno: This pull request references Jira Issue OCPBUGS-61508, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @anuragthehatter

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

alebedev87

The code change looks fine. Some comments about the testing.

test/e2e/ic_conditions_metric_test.go

alebedev87

Second look.

test/e2e/ic_conditions_metric_test.go

openshift-ci · 2025-11-04T10:59:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alebedev87

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [alebedev87]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

lihongan · 2025-11-05T02:01:39Z

/jira refresh

openshift-ci-robot · 2025-11-05T02:01:44Z

@lihongan: This pull request references Jira Issue OCPBUGS-61508, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.21.0) matches configured target version for branch (4.21.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @ShudiLi

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

davidesalerno · 2025-11-05T09:50:06Z

/retest

ShudiLi · 2025-11-07T07:54:39Z

tested it with 4.21.0-0-2025-11-07-061132-test-ci-ln-z5rxwg2-latest

1.
% oc get clusterversion
NAME      VERSION                                                AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.21.0-0-2025-11-07-061132-test-ci-ln-z5rxwg2-latest   True        False         7m      Cluster version is 4.21.0-0-2025-11-07-061132-test-ci-ln-z5rxwg2-latest

2. ingresscontroller int33 had the same domain as that of int22
 % oc -n openshift-ingress-operator get ingresscontroller                

NAME      AGE
default   4m6s
int1      34m
int22     33m
int33     20m

3.  delete the ingresscontroller pod, the metrics of in default, int1 and int22 were available on the new ingresscontroller pod.
4.  delete the default ingresscontroller
5. 
sh-5.1$ curl -s localhost:60000/metrics | grep -v controller_runtime | grep -v workqueue | grep -v go_ | grep controller
# HELP ingress_controller_aws_nlb_active Report the number of active NLBs on AWS clusters.
# TYPE ingress_controller_aws_nlb_active gauge
ingress_controller_aws_nlb_active{name="int1"} 0
ingress_controller_aws_nlb_active{name="int22"} 0
# HELP ingress_controller_conditions Report the conditions for ingress controllers. 0 is False and 1 is True.
# TYPE ingress_controller_conditions gauge
ingress_controller_conditions{condition="Available",name="int1"} 1
ingress_controller_conditions{condition="Available",name="int22"} 1
ingress_controller_conditions{condition="Degraded",name="int1"} 0
ingress_controller_conditions{condition="Degraded",name="int22"} 0
# HELP route_metrics_controller_routes_per_shard Report the number of routes for shards (ingress controllers).
# TYPE route_metrics_controller_routes_per_shard gauge
route_metrics_controller_routes_per_shard{shard_name="int1"} 8
route_metrics_controller_routes_per_shard{shard_name="int22"} 8
sh-5.1$ 
sh-5.1$ curl -s localhost:60000/metrics | grep -v controller_runtime | grep -v workqueue | grep -v go_ | grep controller
# HELP ingress_controller_aws_nlb_active Report the number of active NLBs on AWS clusters.
# TYPE ingress_controller_aws_nlb_active gauge
ingress_controller_aws_nlb_active{name="default"} 0
ingress_controller_aws_nlb_active{name="int1"} 0
ingress_controller_aws_nlb_active{name="int22"} 0
# HELP ingress_controller_conditions Report the conditions for ingress controllers. 0 is False and 1 is True.
# TYPE ingress_controller_conditions gauge
ingress_controller_conditions{condition="Available",name="default"} 0
ingress_controller_conditions{condition="Available",name="int1"} 1
ingress_controller_conditions{condition="Available",name="int22"} 1
ingress_controller_conditions{condition="Degraded",name="default"} 0
ingress_controller_conditions{condition="Degraded",name="int1"} 0
ingress_controller_conditions{condition="Degraded",name="int22"} 0
# HELP route_metrics_controller_routes_per_shard Report the number of routes for shards (ingress controllers).
# TYPE route_metrics_controller_routes_per_shard gauge
route_metrics_controller_routes_per_shard{shard_name="default"} 8
route_metrics_controller_routes_per_shard{shard_name="int1"} 8
route_metrics_controller_routes_per_shard{shard_name="int22"} 8
sh-5.1$ curl -s localhost:60000/metrics | grep -v controller_runtime | grep -v workqueue | grep -v go_ | grep controller
# HELP ingress_controller_aws_nlb_active Report the number of active NLBs on AWS clusters.
# TYPE ingress_controller_aws_nlb_active gauge
ingress_controller_aws_nlb_active{name="default"} 0
ingress_controller_aws_nlb_active{name="int1"} 0
ingress_controller_aws_nlb_active{name="int22"} 0
# HELP ingress_controller_conditions Report the conditions for ingress controllers. 0 is False and 1 is True.
# TYPE ingress_controller_conditions gauge
ingress_controller_conditions{condition="Available",name="default"} 1
ingress_controller_conditions{condition="Available",name="int1"} 1
ingress_controller_conditions{condition="Available",name="int22"} 1
ingress_controller_conditions{condition="Degraded",name="default"} 0
ingress_controller_conditions{condition="Degraded",name="int1"} 0
ingress_controller_conditions{condition="Degraded",name="int22"} 0
# HELP route_metrics_controller_routes_per_shard Report the number of routes for shards (ingress controllers).
# TYPE route_metrics_controller_routes_per_shard gauge
route_metrics_controller_routes_per_shard{shard_name="default"} 8
route_metrics_controller_routes_per_shard{shard_name="int1"} 8
route_metrics_controller_routes_per_shard{shard_name="int22"} 8
sh-5.1$ %

ShudiLi · 2025-11-07T07:55:38Z

/label qe-approved
thanks

grzpiotrowski · 2025-11-07T09:42:40Z

/retest

davidesalerno · 2025-11-07T14:05:51Z

/test okd-scos-e2e-aws-ovn

davidesalerno · 2025-11-12T09:53:02Z

/test e2e-aws-pre-release-ossm

rhamini3 · 2025-11-13T18:04:21Z

/verified by @ShudiLi comment

openshift-ci-robot · 2025-11-13T18:04:33Z

@rhamini3: This PR has been marked as verified by @ShudiLi [comment](https://github.com/openshift/cluster-ingress-operator/pull/1290#issuecomment-3501165490).

In response to this:

/verified by @ShudiLi comment

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2025-11-13T20:00:22Z

/retest-required

Remaining retests: 0 against base HEAD 072c1cd and 2 for PR HEAD 00810f9 in total

davidesalerno · 2025-11-14T08:21:21Z

/test e2e-aws-ovn-serial-1of2

davidesalerno · 2025-11-14T14:02:30Z

/test e2e-aws-ovn

rhamini3 · 2025-11-14T17:28:15Z

/test e2e-gcp-operator

openshift-ci · 2025-11-14T19:59:36Z

@davidesalerno: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-11-14T20:04:12Z

@davidesalerno: Jira Issue Verification Checks: Jira Issue OCPBUGS-61508
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-61508 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

In response to this:

… IngressController

This PR is fixing the issue we have got into the ingress controller that is not exposing ingress_operator_conditions metrics for existing ingress controllers if the operator pod is restarted till the ingress controllers status don't change.

We can achieve this simply moving the SetIngressControllerConditionsMetric away from the code block inside the if statement checking if the ingress status is changed or not.

Added an e2e test that is verifying that the metrics are there before and after the operator pod restart. The e2e test can't be executed in parallel with other tests because it is restarting the ingress operator pod and so it could affect in a non predictable way their results.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

davidesalerno · 2025-11-17T10:50:17Z

/cherry-pick release-4.20

openshift-cherrypick-robot · 2025-11-17T10:51:07Z

@davidesalerno: new pull request created: #1305

In response to this:

/cherry-pick release-4.20

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

davidesalerno · 2025-11-18T15:48:48Z

/cherry-pick release-4.19

openshift-cherrypick-robot · 2025-11-18T15:49:30Z

@davidesalerno: new pull request created: #1308

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-merge-robot · 2025-11-20T13:50:15Z

Fix included in accepted release 4.21.0-0.nightly-2025-11-15-144034

openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels Oct 6, 2025

openshift-ci-robot added the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Oct 6, 2025

openshift-ci bot requested review from Thealisyed and candita October 6, 2025 09:57

Miciah reviewed Oct 8, 2025

View reviewed changes

test/e2e/ic_conditions_metric_test.go Show resolved Hide resolved

openshift-ci bot assigned alebedev87 Oct 9, 2025

openshift-ci bot assigned grzpiotrowski Oct 9, 2025

davidesalerno force-pushed the ocpbugs61508onmaster branch from 38d8184 to 8864da1 Compare October 10, 2025 07:31

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Oct 10, 2025

openshift-ci bot requested a review from anuragthehatter October 10, 2025 12:11

alebedev87 reviewed Oct 15, 2025

View reviewed changes

davidesalerno force-pushed the ocpbugs61508onmaster branch from 8864da1 to b90a8c4 Compare October 16, 2025 12:24

alebedev87 reviewed Oct 20, 2025

View reviewed changes

davidesalerno force-pushed the ocpbugs61508onmaster branch 3 times, most recently from b90a8c4 to 3896e07 Compare October 21, 2025 10:15

grzpiotrowski reviewed Oct 24, 2025

View reviewed changes

test/e2e/ic_conditions_metric_test.go Show resolved Hide resolved

test/e2e/ic_conditions_metric_test.go Show resolved Hide resolved

test/e2e/ic_conditions_metric_test.go Outdated Show resolved Hide resolved

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 4, 2025

openshift-ci bot requested a review from ShudiLi November 5, 2025 02:01

openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Nov 7, 2025

grzpiotrowski approved these changes Nov 7, 2025

View reviewed changes

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Nov 13, 2025

openshift-merge-bot bot merged commit 8773f8f into openshift:master Nov 14, 2025
18 checks passed

openshift-cherrypick-robot mentioned this pull request Nov 17, 2025

[release-4.20] OCPBUGS-65664: IngressOperator not exposing some metrics for degraded… #1305

Open

openshift-cherrypick-robot mentioned this pull request Nov 18, 2025

[release-4.19] OCPBUGS-65723: IngressOperator not exposing some metrics for degraded… #1308

Open

OCPBUGS-61508: IngressOperator not exposing some metrics for degraded… #1290

OCPBUGS-61508: IngressOperator not exposing some metrics for degraded… #1290

Uh oh!

Conversation

davidesalerno commented Oct 6, 2025

Uh oh!

openshift-ci-robot commented Oct 6, 2025

Uh oh!

openshift-ci-robot commented Oct 6, 2025

Uh oh!

davidesalerno commented Oct 6, 2025

Uh oh!

Uh oh!

davidesalerno commented Oct 8, 2025

Uh oh!

candita commented Oct 9, 2025

Uh oh!

candita commented Oct 9, 2025

Uh oh!

davidesalerno commented Oct 10, 2025

Uh oh!

openshift-ci-robot commented Oct 10, 2025

Uh oh!

davidesalerno commented Oct 10, 2025

Uh oh!

davidesalerno commented Oct 10, 2025

Uh oh!

openshift-ci-robot commented Oct 10, 2025

Uh oh!

alebedev87 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alebedev87 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci bot commented Nov 4, 2025

Uh oh!

lihongan commented Nov 5, 2025

Uh oh!

openshift-ci-robot commented Nov 5, 2025

Uh oh!

davidesalerno commented Nov 5, 2025

Uh oh!

ShudiLi commented Nov 7, 2025

Uh oh!

ShudiLi commented Nov 7, 2025

Uh oh!

grzpiotrowski commented Nov 7, 2025

Uh oh!

davidesalerno commented Nov 7, 2025

Uh oh!

davidesalerno commented Nov 12, 2025

Uh oh!

rhamini3 commented Nov 13, 2025

Uh oh!

openshift-ci-robot commented Nov 13, 2025

Uh oh!

openshift-ci-robot commented Nov 13, 2025

Uh oh!

davidesalerno commented Nov 14, 2025

Uh oh!

davidesalerno commented Nov 14, 2025

alebedev87 left a comment •

edited

Loading