OCPBUGS-34568,OCPBUGS-35095,OCPBUGS-60689,OCPBUGS-60691,OCPBUGS-60692: non-HA alert cases #2630

rexagod · 2025-07-29T21:27:25Z

Pulls in changes from 1, which refactors alerts to accomodate for non-HA cases.

I added CHANGELOG entry for this change.
No user facing changes, so no entry in CHANGELOG was needed.

openshift-ci-robot · 2025-07-29T21:27:34Z

@rexagod: This pull request references Jira Issue OCPBUGS-34568, which is invalid:

expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-35095, which is invalid:

expected the bug to target either version "4.20." or "openshift-4.20.", but it targets "4.19.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Pulls in changes from 1, which refactors alerts to accomodate for non-HA cases.

I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rexagod · 2025-07-29T21:29:49Z

/jira refresh

openshift-ci-robot · 2025-07-29T21:29:56Z

@rexagod: This pull request references Jira Issue OCPBUGS-34568, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

This pull request references Jira Issue OCPBUGS-35095, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rexagod · 2025-08-04T08:37:02Z

/retest-required

assets/control-plane/prometheus-rule.yaml

simonpasquier · 2025-08-07T07:13:40Z

@rexagod could you rebase on main to get rid of the version changes which make it harder to review?

Pulls in changes from [1], which refactors alerts to accomodate for non-HA cases. [1]: kubernetes-monitoring/kubernetes-mixin#1010 Signed-off-by: Pranshu Srivastava <[email protected]>

simonpasquier

/lgtm
/hold
I see that the bump adds a couple of alerting rules (most of them at info level). Maybe it's worth listing them in the CHANGELOG? Also how confident are we that these new rules won't interfere with the CI (e.g. the origin e2e tests will fail if they detect firing alerts)?

simonpasquier · 2025-08-12T12:39:40Z

jsonnet/main.jsonnet

            pvExcludedSelector: 'label_alerts_k8s_io_kube_persistent_volume_filling_up="disabled"',
            containerfsSelector: 'id!=""',
            clusterLabel: $.values.common.dashboardClusterLabel,
+            showMultiCluster: false,  // Opt-out of multi-cluster dashboards (opted-in by midstream kube-prometheus)


rexagod · 2025-08-13T08:44:53Z

/payload-aggregate periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn 3

openshift-ci · 2025-08-13T08:44:57Z

@rexagod: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c82b9a40-7821-11f0-9bd2-835c3a7ddd94-0

rexagod · 2025-08-13T11:33:17Z

/payload-job periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn

/payload-aggregate periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn 6

openshift-ci · 2025-08-13T11:33:20Z

@rexagod: trigger 2 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn
periodic-ci-openshift-hypershift-release-4.20-periodics-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/4f47f430-7839-11f0-849e-5581308be2cc-0

rexagod · 2025-08-17T21:17:36Z

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn

openshift-ci · 2025-08-17T21:17:39Z

@rexagod: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9966f4c0-7baf-11f0-8d35-5666afe3a7b2-0

rexagod · 2025-08-18T07:24:45Z

Also how confident are we that these new rules won't interfere with the CI (e.g. the origin e2e tests will fail if they detect firing alerts)?

@simonpasquier All origin monitoring tests are passing (such as shouldn't report any alerts in firing state apart from Watchdog and AlertmanagerReceiversNotConfigured [Early]), which should help establish confidence IIUC.

simonpasquier

The jsonnet bump also brings a few (useful!) fixes:

Do you know if we have OCPBUGS tickets opened for them? If not, should we create them and link to this PR for traceability?

/lgtm
/hold

openshift-ci · 2025-08-20T10:17:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rexagod, simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rexagod,simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

simonpasquier · 2025-08-20T11:54:11Z

/retest-required
/skip

openshift-ci-robot · 2025-08-20T14:52:55Z

@rexagod: This pull request references Jira Issue OCPBUGS-34568, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-35095, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-60689, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-60691, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

The bug has been updated to refer to the pull request using the external bug tracker.

This pull request references Jira Issue OCPBUGS-60692, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.20.0) matches configured target version for branch (4.20.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @juzhao

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Pulls in changes from 1, which refactors alerts to accomodate for non-HA cases.

I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

rexagod · 2025-08-20T15:02:49Z

Created tickets for the aforementioned PRs (since there were no existing ones that tracked them), PTAL.

simonpasquier · 2025-08-21T08:25:38Z

/hold cancel

openshift-ci-robot · 2025-08-21T08:31:07Z

/retest-required

Remaining retests: 0 against base HEAD d81d23d and 2 for PR HEAD d1e492a in total

openshift-ci-robot · 2025-08-21T14:23:35Z

/retest-required

Remaining retests: 0 against base HEAD d81d23d and 2 for PR HEAD d1e492a in total

openshift-ci-robot · 2025-08-21T22:33:12Z

/retest-required

Remaining retests: 0 against base HEAD d81d23d and 2 for PR HEAD d1e492a in total

openshift-ci · 2025-08-22T01:31:28Z

@rexagod: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node	`d1e492a`	link	false	`/test e2e-aws-ovn-single-node`
ci/prow/okd-scos-e2e-aws-ovn	`d1e492a`	link	false	`/test okd-scos-e2e-aws-ovn`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot · 2025-08-22T15:02:36Z

/retest-required

Remaining retests: 0 against base HEAD d81d23d and 2 for PR HEAD d1e492a in total

openshift-ci-robot · 2025-08-22T17:54:05Z

@rexagod: Jira Issue OCPBUGS-34568: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#2630

Jira Issue OCPBUGS-34568 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-35095: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-35095 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-60689: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#2630

Jira Issue OCPBUGS-60689 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-60691: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#2630

Jira Issue OCPBUGS-60691 has been moved to the MODIFIED state.

Jira Issue OCPBUGS-60692: All pull requests linked via external trackers have merged:

openshift/cluster-monitoring-operator#2630

Jira Issue OCPBUGS-60692 has been moved to the MODIFIED state.

In response to this:

Pulls in changes from 1, which refactors alerts to accomodate for non-HA cases.

I added CHANGELOG entry for this change.

No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-bot · 2025-08-23T00:43:05Z

[ART PR BUILD NOTIFIER]

Distgit: cluster-monitoring-operator
This PR has been included in build cluster-monitoring-operator-container-v4.21.0-202508230015.p0.g3ebd14a.assembly.stream.el9.
All builds following this will include this PR.

rexagod · 2025-10-10T16:46:38Z

/cherrypick release-4.19,release-4.18,release-4.17,release-4.16

openshift-cherrypick-robot · 2025-10-10T16:47:26Z

@rexagod: cannot checkout release-4.19,release-4.18,release-4.17,release-4.16: error checking out "release-4.19,release-4.18,release-4.17,release-4.16": exit status 1 error: pathspec 'release-4.19,release-4.18,release-4.17,release-4.16' did not match any file(s) known to git

In response to this:

/cherrypick release-4.19,release-4.18,release-4.17,release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rexagod · 2025-10-10T16:48:24Z

/cherrypick release-4.19 release-4.18 release-4.17 release-4.16

openshift-cherrypick-robot · 2025-10-10T16:49:04Z

@rexagod: #2630 failed to apply on top of branch "release-4.19":

Applying: OCPBUGS-34568,OCPBUGS-35095: non-HA alert cases
Using index info to reconstruct a base tree...
M	CHANGELOG.md
M	jsonnet/jsonnetfile.lock.json
Falling back to patching base and 3-way merge...
Auto-merging jsonnet/jsonnetfile.lock.json
Auto-merging CHANGELOG.md
CONFLICT (content): Merge conflict in CHANGELOG.md
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 OCPBUGS-34568,OCPBUGS-35095: non-HA alert cases

In response to this:

/cherrypick release-4.19 release-4.18 release-4.17 release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci bot requested review from marioferh and simonpasquier July 29, 2025 21:28

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2025

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 29, 2025

openshift-ci bot requested a review from juzhao July 29, 2025 21:29

rexagod force-pushed the OCPBUGS-34568,OCPBUGS-35095 branch 2 times, most recently from 682569b to f93c51c Compare July 29, 2025 23:20

simonpasquier reviewed Aug 4, 2025

View reviewed changes

assets/control-plane/prometheus-rule.yaml Outdated Show resolved Hide resolved

rexagod force-pushed the OCPBUGS-34568,OCPBUGS-35095 branch 2 times, most recently from 4c7aca4 to 8d1fadc Compare August 5, 2025 10:14

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 9, 2025

rexagod added 4 commits August 12, 2025 14:22

OCPBUGS-34568,OCPBUGS-35095: non-HA alert cases

0ad7686

Pulls in changes from [1], which refactors alerts to accomodate for non-HA cases. [1]: kubernetes-monitoring/kubernetes-mixin#1010 Signed-off-by: Pranshu Srivastava <[email protected]>

bugfix: opt-out of multi-cluster control plane rules

1424512

chore: make generate

3c01065

chore: make jsonnet-fmt

49eed1d

rexagod force-pushed the OCPBUGS-34568,OCPBUGS-35095 branch from a34b00d to 49eed1d Compare August 12, 2025 09:06

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 12, 2025

simonpasquier reviewed Aug 12, 2025

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 12, 2025

openshift-ci bot assigned simonpasquier Aug 12, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 12, 2025

rexagod requested a review from simonpasquier August 20, 2025 08:49

simonpasquier reviewed Aug 20, 2025

View reviewed changes

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Aug 20, 2025

rexagod changed the title ~~OCPBUGS-34568,OCPBUGS-35095: non-HA alert cases~~ OCPBUGS-34568,OCPBUGS-35095,OCPBUGS-60689,OCPBUGS-60691,OCPBUGS-60692: non-HA alert cases Aug 20, 2025

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 21, 2025

openshift-merge-bot bot merged commit 3ebd14a into openshift:main Aug 22, 2025
21 checks passed

rexagod mentioned this pull request Oct 11, 2025

OCPBUGS-62965,OCPBUGS-62966,OCPBUGS-62967,OCPBUGS-62968,OCPBUGS-62969: [Backport] non-HA alert cases #2709

Merged

2 tasks

OCPBUGS-34568,OCPBUGS-35095,OCPBUGS-60689,OCPBUGS-60691,OCPBUGS-60692: non-HA alert cases #2630

OCPBUGS-34568,OCPBUGS-35095,OCPBUGS-60689,OCPBUGS-60691,OCPBUGS-60692: non-HA alert cases #2630

Uh oh!

Conversation

rexagod commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Jul 29, 2025

Uh oh!

rexagod commented Jul 29, 2025

Uh oh!

openshift-ci-robot commented Jul 29, 2025

Uh oh!

rexagod commented Aug 4, 2025

Uh oh!

Uh oh!

simonpasquier commented Aug 7, 2025

Uh oh!

simonpasquier left a comment

Choose a reason for hiding this comment

Uh oh!

simonpasquier Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

rexagod commented Aug 13, 2025

Uh oh!

openshift-ci bot commented Aug 13, 2025

Uh oh!

rexagod commented Aug 13, 2025

Uh oh!

openshift-ci bot commented Aug 13, 2025

Uh oh!

rexagod commented Aug 17, 2025

Uh oh!

openshift-ci bot commented Aug 17, 2025

Uh oh!

rexagod commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

simonpasquier left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Aug 20, 2025

Uh oh!

simonpasquier commented Aug 20, 2025

Uh oh!

openshift-ci-robot commented Aug 20, 2025

Uh oh!

rexagod commented Aug 20, 2025

Uh oh!

simonpasquier commented Aug 21, 2025

Uh oh!

openshift-ci-robot commented Aug 21, 2025

Uh oh!

openshift-ci-robot commented Aug 21, 2025

Uh oh!

openshift-ci-robot commented Aug 21, 2025

Uh oh!

openshift-ci bot commented Aug 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 22, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Aug 22, 2025

Uh oh!

openshift-bot commented Aug 23, 2025

Uh oh!

rexagod commented Oct 10, 2025

Uh oh!

openshift-cherrypick-robot commented Oct 10, 2025

Uh oh!

rexagod commented Oct 10, 2025

Uh oh!

openshift-cherrypick-robot commented Oct 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

rexagod commented Jul 29, 2025 •

edited

Loading

rexagod commented Aug 18, 2025 •

edited

Loading

openshift-ci bot commented Aug 22, 2025 •

edited

Loading