Skip to content

OCPBUGS-58145: reload serving cert on rotation #1285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

OCPBUGS-58145: reload serving cert on rotation #1285

wants to merge 1 commit into from

Conversation

@bentito
Copy link
Contributor

@bentito bentito commented Sep 25, 2025

This PR attempts to fix this bug, here's an overview of what it's doing:

• Compute TLS secret hash (tls.crt, tls.key, optional ca.crt)
• Annotate pod template with canary-serving-cert hash
• Watch canary serving cert Secret to trigger reconcile
• Emit event when cert hash changes (traceability)
• Update desiredCanaryDaemonSet(canaryImage, certHash)
• Reconcile logic compares/propagates hash annotation
• Add unit tests for hash compute and DaemonSet change detection

@bentito
OCPBUGS-58145: reload serving cert on rotation
2f8f5ac 
• Compute TLS secret hash (tls.crt, tls.key, optional ca.crt)
• Annotate pod template with canary-serving-cert hash
• Watch canary serving cert Secret to trigger reconcile
• Emit event when cert hash changes (traceability)
• Update desiredCanaryDaemonSet(canaryImage, certHash)
• Reconcile logic compares/propagates hash annotation
• Add unit tests for hash compute and DaemonSet change detection

Signed-off-by: Brett Tofel <[email protected]>
@openshift-ci-robot openshift-ci-robot added jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 25, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 25, 2025

@bentito: This pull request references Jira Issue OCPBUGS-58145, which is invalid:

  • expected the bug to target the "4.21.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This PR attempts to fix this bug, here's an overview of what it's doing:

• Compute TLS secret hash (tls.crt, tls.key, optional ca.crt)
• Annotate pod template with canary-serving-cert hash
• Watch canary serving cert Secret to trigger reconcile
• Emit event when cert hash changes (traceability)
• Update desiredCanaryDaemonSet(canaryImage, certHash)
• Reconcile logic compares/propagates hash annotation
• Add unit tests for hash compute and DaemonSet change detection

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bentito
Copy link
Contributor Author

bentito commented Sep 25, 2025

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 25, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 25, 2025

@bentito: This pull request references Jira Issue OCPBUGS-58145, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Sep 25, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign gcs278 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@bentito
Copy link
Contributor Author

bentito commented Sep 25, 2025

/retest

7 similar comments
@bentito
Copy link
Contributor Author

bentito commented Sep 26, 2025

/retest

@bentito
Copy link
Contributor Author

bentito commented Sep 26, 2025

/retest

@bentito
Copy link
Contributor Author

bentito commented Sep 26, 2025

/retest

@bentito
Copy link
Contributor Author

bentito commented Sep 27, 2025

/retest

@bentito
Copy link
Contributor Author

bentito commented Sep 27, 2025

/retest

@bentito
Copy link
Contributor Author

bentito commented Sep 27, 2025

/retest

@bentito
Copy link
Contributor Author

bentito commented Sep 29, 2025

/retest

@melvinjoseph86
Copy link

melvinjoseph86 commented Sep 30, 2025

/test e2e-aws-ovn-serial

@melvinjoseph86
Copy link

melvinjoseph86 commented Sep 30, 2025

Tested with cluster bot


mjoseph@mjoseph-mac Downloads % oc get clusterversion
NAME      VERSION                                                AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         25m     Cluster version is 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest
1) Set some default variables 
mjoseph@mjoseph-mac Downloads % NS=openshift-ingress-canary
OP_NS=openshift-ingress-operator
DS=ingress-canary
SECRET=canary-serving-cert

2) Record the current PodTemplate annotation and current pods' annotations
mjoseph@mjoseph-mac Downloads % oc -n $NS get ds $DS -o jsonpath='{.spec.template.metadata.annotations.ingress\.operator\.openshift\.io/canary-serving-cert-hash}' || echo "(none)"
087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b%                                                                                                                                                   mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $NS get pods -o custom-columns=NAME:.metadata.name,HASH:.metadata.annotations.ingress\\.operator\\.openshift\\.io/canary-serving-cert-hash --no-headers
ingress-canary-gpds9   087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b
ingress-canary-kln85   087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b
ingress-canary-v4flf   087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b

3) Simulate a rotation of the serving cert by create new base64-encoded values (portable):
mjoseph@mjoseph-mac Downloads % NEWCRT_B64=$(echo -n "rotation-$(date -u +%s)" | base64 | tr -d '\n')
mjoseph@mjoseph-mac Downloads % NEWKEY_B64=$(echo -n "rotation-key-$(date -u +%s)" | base64 | tr -d '\n')
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
- Patch the Secret in-place to replace `tls.crt` and `tls.key`:
mjoseph@mjoseph-mac Downloads % oc -n $NS patch secret $SECRET --type=json -p "$(jq -n --arg c "$NEWCRT_B64" --arg k "$NEWKEY_B64" '[{op:"replace",path:"/data/tls.crt",value:$c},{op:"replace",path:"/data/tls.key",value:$k}]')"
secret/canary-serving-cert patched
mjoseph@mjoseph-mac Downloads % oc get secret -n openshift-ingress-canary
NAME                       TYPE                      DATA   AGE
builder-dockercfg-9vb4h    kubernetes.io/dockercfg   1      44m
canary-serving-cert        kubernetes.io/tls         2      44m
default-dockercfg-r4c64    kubernetes.io/dockercfg   1      44m
deployer-dockercfg-tzxg5   kubernetes.io/dockercfg   1      44m

mjoseph@mjoseph-mac Downloads % oc get secret -n openshift-ingress-canary canary-serving-cert -oyaml
apiVersion: v1
data:
  tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVFVENDQXZtZ0F3SUJBZ0lJWklWMXdEYVdYLzR3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRjMU9USTFNRGMxTURBZQpGdzB5TlRBNU16QXhOek01TXpaYUZ3MHlOekE1TXpBeE56TTVNemRhTURZeE5EQXlCZ05WQkFNVEsybHVaM0psCmMzTXRZMkZ1WVhKNUxtOXdaVzV6YUdsbWRDMXBibWR5WlhOekxXTmhibUZ5ZVM1emRtTXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3lRbHFsY2J1WmY2SjZjQ0JjaWxSVVFwaitWbXl4MEw5WgpPb1gvU2JwalFiQm0rekhGVDRpYzBLclRIM2NLeTJJT0o0MVF0STJRWHRRdXhyMEJCcmRoRnZxMytqaTFoV3VDCkdOVUt5WDRJZTV5MXNDUWF5di9RcUl6TXNDOVNKUGd3V0VaUmJyZ1ZQRWFuak8yRVFRRzdnSHpJWVdHZ0ZtR0cKak1RT3pOQWtCODZlQWFDYXFxRU54dkJEVXE3Uzh3a1gyb0tpWm4xZGF5UXlKRmRBVWlsRWlldEdrUXNRVzFlcAo0VHIwVlFxWGtOblp4YVFDN0JXRmpaMkkySW8zTkVUcTNJaVZYWC95dEhtbGhLL1Y5ZmlJRmZKZnVPRGhxUE1PCkxZU0xkcEhkajRSdE1aOVZCdHZSaXZlT1FtQVBtTk9yQkxuS3dKM0tPYm5ubVNHZERGLzlBZ01CQUFHamdnRWgKTUlJQkhUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVApBUUgvQkFJd0FEQWRCZ05WSFE0RUZnUVU4b0V2N3AxcDdLcUhsei9ZNi9tTGFTRGpnemN3SHdZRFZSMGpCQmd3CkZvQVVYSGE2NzNTekVLaVZpMjFOWk1majBlT0xUQkl3Y1FZRFZSMFJCR293YUlJcmFXNW5jbVZ6Y3kxallXNWgKY25rdWIzQmxibk5vYVdaMExXbHVaM0psYzNNdFkyRnVZWEo1TG5OMlk0STVhVzVuY21WemN5MWpZVzVoY25rdQpiM0JsYm5Ob2FXWjBMV2x1WjNKbGMzTXRZMkZ1WVhKNUxuTjJZeTVqYkhWemRHVnlMbXh2WTJGc01EVUdDeXNHCkFRUUJrZ2dSWkFJQkJDWVRKREkzTlRsbU16WmpMVE00WW1VdE5EVTBNUzFoTkRjeExURmxZamczWWpFek5XRXoKWkRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVFmc2xNU08wWCtzeDlmcldzajdxakF0czBEcU4xd0p6cjVuYgpHaTh3TENwbXd6dkJaM1FuU1JhdXBqL1Z6NDRQUmZLTGRuT2ZTRkNzSTd5U0YzNHkyckI0eVpaRjhIRUsxZ1pJCk9jU3B0aFVDQTRPOEhVV245WDBMSWo0T2ZGYUxFUEIxQUxtb1RxMGI0T3c4YzlFN284T0ZUR1VvWEJPZC85YVkKenByaGRhZTRiTFFTOG1TeVRIK0t4czR5UExMSEZMR2hmaFo1Y0lyaGFuazBQV0V2QnJHNzdsNzFtVE05aG1WRwowc3BpOElicDZWTGtuZVNjdUxuMnZteUZWUHh1TC96bjBBdE11R3pvOTkyN0FmSkkreEhkTFFGZGRDRUFRZjFMCk5FUXorTG1YeWxkSWJLV3dCYUl1eTRTbFBLZXRnM2hOVnZ4Q29KaEluT3JaYi9ENG1nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRFVUQ0NBam1nQXdJQkFnSUlWYmx1WmJFTVlWNHdEUVlKS29aSWh2Y05BUUVMQlFBd05qRTBNRElHQTFVRQpBd3dyYjNCbGJuTm9hV1owTFhObGNuWnBZMlV0YzJWeWRtbHVaeTF6YVdkdVpYSkFNVGMxT1RJMU1EYzFNREFlCkZ3MHlOVEE1TXpBeE5qUTFORGxhRncweU56RXhNamt4TmpRMU5UQmFNRFl4TkRBeUJnTlZCQU1NSzI5d1pXNXoKYUdsbWRDMXpaWEoyYVdObExYTmxjblpwYm1jdGMybG5ibVZ5UURFM05Ua3lOVEEzTlRBd2dnRWlNQTBHQ1NxRwpTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDaS9UVzlaU1FNZ0UyMXFLdlg2T24rek9kSkJ0V2hHa1YxCnJiUDRFU1lOd3hZNnJZWXFPSG0ybFpLUmc1eitDODlYUUZhUkFjOUpUVkMwbGZwanJxWU95bUhnTTdwWjZ5MEgKZTRrblhRL0M4TUgzZGR2dVI2V1dBU1liNnduRGJaaE0za2RxZFNRb3VOVk8zNlkzZXByekF4UE9ST3JRVTQ0UwowUmR4NUcvQThod3hNL3QwajJqWjhWMlVjSCtLalkvL1Azc1pjQWlVL3V4VEVSWkZhTFBhcHBLTUZ4QjRud0QxClNFNVNyRk1oZGNGZEVVRm1VVVlpeG9mNTJscTB0M3Z0dnFhZFVOZUYzc0F1d29Tazc1N29YZWFoVW9jQ3FoQngKd05aQ3Y3UFBlTHM0OFBndGRBMHlvNy9IWmxqcWQydzduem5SajQxQ2VPamxVaUlxblNlWEFnTUJBQUdqWXpCaApNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUmNkcnJ2CmRMTVFxSldMYlUxa3grUFI0NHRNRWpBZkJnTlZIU01FR0RBV2dCUmNkcnJ2ZExNUXFKV0xiVTFreCtQUjQ0dE0KRWpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVJCeTVqV1puZ1YzUy90SGJKQzR1dUJDSXBYazczb1V5RGpqTQpKTzYwRnNMMWNTSFM0SVdwN251T1VEekowTjBrcERwanl0WTUzMlpFeENncUFVUTVNOUhzTlFYS2pnQUdTZVJxCnNTUnBGZzN5aU5yazVyY3ZtNlNzYzNJTjJsanplQlRZdFRUd2N2dmY3aVBpalVvRmFmaVZiUXNCRm5MdmQ0aDAKVSszR0ZORzRTeldURVBUeG8vVURRL3hpSUdnbTRaSHJwMkJjWjhBYUx5UG5HZlI1NytOSERQdThMM1NTT05aWApmTVhsN0lhaFRyNWIvZDBqQ2pDZ2hlRytzMkpodEIyNTJVVEdINmI2QWFEem1OSTFNOWd1Z29mNWgra3RMMVpXCmxiNVd6MGZFVkpTQkxqWjB1aCtodHNXNCtqNk01WkxLWjJBcExzVmU3T0JuYmFIM21RPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb1FJQkFBS0NBUUVBc2tKYXBYRzdtWCtpZW5BZ1hJcFVWRUtZL2xac3NkQy9XVHFGLzBtNlkwR3dadnN4CnhVK0luTkNxMHg5M0NzdGlEaWVOVUxTTmtGN1VMc2E5QVFhM1lSYjZ0L280dFlWcmdoalZDc2wrQ0h1Y3RiQWsKR3NyLzBLaU16TEF2VWlUNE1GaEdVVzY0RlR4R3A0enRoRUVCdTRCOHlHRmhvQlpoaG96RURzelFKQWZPbmdHZwptcXFoRGNid1ExS3Uwdk1KRjlxQ29tWjlYV3NrTWlSWFFGSXBSSW5yUnBFTEVGdFhxZUU2OUZVS2w1RFoyY1drCkF1d1ZoWTJkaU5pS056UkU2dHlJbFYxLzhyUjVwWVN2MWZYNGlCWHlYN2pnNGFqekRpMkVpM2FSM1krRWJUR2YKVlFiYjBZcjNqa0pnRDVqVHF3UzV5c0NkeWptNTU1a2huUXhmL1FJREFRQUJBb0gvSGlEM0NjTmNwc3pEVnRDcgpUYUdqcFdzbWtUdGNtZ3liNUcrWHJnVUQ5cGNzV20xWmNpVDlkZVNmM2JkUzREeTZ2TU5RQStxQWFwbmh1cnVCCjBpcnQySWxPbVhDbXBua0xld1ZsR3lhNFpJV0c5aGhYL0ZTdDNjaTRHYXFSL0dRSWJaNk9pVGhyWmdqWGlOSVEKbFhHdlNRUWtIaVN0UEsxVndWaUJJMFJ2QW4rVUtmWXA3dmc5dHM1QTBnd3ZCMFVCbVpOK2x4QzJXVWNtRWNHRQpMU3dEWC96eHN5VEliMEVMOTFuaVVDWmFLTVB4KzdPdmVZbm9BRDZiR2NxVFpjTlFYK1I3eGxvVlMvT0llQlZDCmFEMmhleko5RU8vcEU0Z2hxeUI4T1J2N0lOMXh5eHZ0L1hYSlp3R2E0K09wc3RGVzE2OGxvekFmWlZEZzdiaXQKN3VncEFvR0JBTitsa1NiZGY1VTJldVZhNFZpSGg5Rk55Smt3OFdRSWt3bExmU2JsMFFwNGVkM29oZmhhNFVjMAp1bndzS0V2VmRKUGlpZjVyL0ViajBBQVNQVCtCemlFSDRUSUNyU3BnSWpudG55aDhOU1BKZWpjenJsSGpjVks5CllPMWxEOTVqUUV4MGZXM2RPamZWZE01Y09YTmZjWUk4YmdwWDlwc3FpamtReHZleW84L2xBb0dCQU13TDdNb24KQ1BZUDNlQ3VuY2dsS0lJb2NDazhzKzk1SndpSEtaQjA4VG9pcHVPK2VWMm9NNUdVckdCY3FZaGNaeFVUTDNDTApGTEtaQTFsdGNkMEtOUm56VFQvdWo0LzJXeGp3MU5Gcm9UZU1PWFRXclFROFh0WXh2RjYrWGdLZnE4ZTdlcGhTCjRjZ0VlWndxQkYrQUJGVGJkeWh5MUhMRDhwdG1WZWJwZVY0NUFvR0FRWVJnWEhGaEVkWk85R2ZHUWtnWitxLzkKeHhyMmJHYXlwSE52NjNoQjMvNWhQU2dFTHpoNGJJSys2dWVJdW9hQ3kxdi9Md0IxZkZMNnVUcWVwTDhqSXJ3TApOYnBnL2x3QlN0VVkzaVJjOE9ueUdyL1NIbjB4MHBwV0RkV2dhM2JKL0Z2WDd0cHpWcTM2dFFXQmN4SVB6NlppCldqZGdzQjRyTmdKN04wOG0vQjBDZ1lFQXJVQmExL0o0KzJUUmdoc0J3WlU3WmZpcjgzZWxXR3BJS0VlTGxTcVcKUUVSMzY5d285TXNDcmxQTVlTMk1qVW5ZQk43N1l0WFV4eWlBM1llK0xHNmtVOXB2OXg2Nkt3U0VxUktqd1FWZwpKK2dhU1FpV2M0aEhCaHZJMDdGVDVXcHAwdmR1eXJObEhnNGVxK2hKT2tWYWFDckliRUhEL1lZL0VvKzcxSXJkCkVKRUNnWUJURXovdDh4S1JGdlExRU5DTjZjZkN0d2w3OXVGdGJWT3RXT3RRN00ybFJ4RTdQRXMya0lmS3liT3MKUVAxMlNEY004OWN0Nnl5aG5jQm9DQUFpU3F6Q0tCWFVxMTVqRXQ5ZjBwdEpndXNHakpvb1hRR29BRnRoNG8veQpYUmJKWnNROFpYRUVCVVhieXd1dU1pbXFmejNHeERISFZGY08wU2d4bTUwYzNUSUxhdz09Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
  annotations:
    openshift.io/description: 'Secret contains a pair signed serving certificate/key
      that is generated by Service CA operator for service/ingress-canary with hostname
      ingress-canary.openshift-ingress-canary.svc and is annotated to the service
      with annotating a service resource with ''service.beta.openshift.io/serving-cert-secret-name:
      canary-serving-cert''. The certificate is valid for 2 years.'
    openshift.io/owning-component: service-ca
    service.alpha.openshift.io/expiry: "2027-09-30T17:39:37Z"
    service.beta.openshift.io/expiry: "2027-09-30T17:39:37Z"
    service.beta.openshift.io/originating-service-name: ingress-canary
    service.beta.openshift.io/originating-service-uid: 2759f36c-38be-4541-a471-1eb87b135a3d
  creationTimestamp: "2025-09-30T16:56:45Z"
  name: canary-serving-cert
  namespace: openshift-ingress-canary
  ownerReferences:
  - apiVersion: v1
    kind: Service
    name: ingress-canary
    uid: 2759f36c-38be-4541-a471-1eb87b135a3d
  resourceVersion: "39537"
  uid: e3da7a32-7159-4283-9a65-cda4c761cf1f
type: kubernetes.io/tls

4) Watch operator logs for the rollout trigger and look for the event/log indicating a canary cert-driven update
The event emission has a `updated canary daemonset` log line with the  reasone `CanaryCertRotated`.
mjoseph@mjoseph-mac Downloads % oc -n $OP_NS get pods -l name=ingress-operator -o name || oc -n $OP_NS get pods | grep ingress-operator
pod/ingress-operator-76457bd877-66k7l
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $OP_NS logs -f ingress-operator-76457bd877-66k7l | sed -n '/CanaryCertRotated/,/updated canary daemonset/p'
Defaulted container "ingress-operator" out of: ingress-operator, kube-rbac-proxy
2025-09-30T16:56:45.579Z	DEBUG	operator.init.events	record/event.go:418	Canary serving cert rotated, updated pod template annotation hash: 08783236	{"type": "Normal", "object": {"kind":"DaemonSet","namespace":"openshift-ingress-canary","name":"ingress-canary","uid":"9d849944-b277-4943-980a-25e4c3f03514","apiVersion":"apps/v1","resourceVersion":"21570"}, "reason": "CanaryCertRotated"}
2025-09-30T16:56:45.673Z	INFO	operator.canary_controller	canary/service.go:29	updated canary service	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.Service{\n  \tTypeMeta: {},\n  \tObjectMeta: v1.ObjectMeta{\n  \t\t... // 9 identical fields\n  \t\tDeletionGracePeriodSeconds: nil,\n  \t\tLabels:                     {\"ingress.openshift.io/canary\": \"canary_controller\"},\n  \t\tAnnotations: map[string]string{\n- \t\t\t\"service.alpha.openshift.io/serving-cert-signed-by\":  \"openshift-service-serving-signer@1759250750\",\n  \t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"canary-serving-cert\",\n- \t\t\t\"service.beta.openshift.io/serving-cert-signed-by\":   \"openshift-service-serving-signer@1759250750\",\n  \t\t},\n  \t\tOwnerReferences: {{APIVersion: \"apps/v1\", Kind: \"daemonset\", Name: \"ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ...}},\n  \t\tFinalizers:      nil,\n  \t\tManagedFields:   {{Manager: \"ingress-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 16:56:45 +0000 UTC\", ...}, {Manager: \"service-ca-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 16:56:45 +0000 UTC\", ...}},\n  \t},\n  \tSpec:   {Ports: {{Name: \"8443-tcp\", Protocol: \"TCP\", Port: 8443, TargetPort: {IntVal: 8443}, ...}, {Name: \"8888-tcp\", Protocol: \"TCP\", Port: 8888, TargetPort: {IntVal: 8888}, ...}}, Selector: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}, ClusterIP: \"172.30.225.114\", ClusterIPs: {\"172.30.225.114\"}, ...},\n  \tStatus: {},\n  }\n"}
2025-09-30T16:56:45.718Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/4 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/4 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 3 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:56:45.818Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:45.835Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:45.867Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:56:57.952Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:56:57.952Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.716Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:56:58.716Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:56:58.716Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.942Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:56:58.942Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.958Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:59.032Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:02.954Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:57:02.955Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:02.973Z	INFO	operator.configurable_route_controller	controller/controller.go:119	reconciling	{"request": {"name":"cluster"}}
2025-09-30T16:57:02.973Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:02.974Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:02.974Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:03.173Z	INFO	operator.configurable_route_controller	controller/controller.go:119	reconciling	{"request": {"name":"cluster"}}
2025-09-30T16:57:03.173Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:03.173Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:03.334Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:57:03.334Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:03.434Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:57:27.004Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:27.004Z	INFO	operator.configurable_route_controller	controller/controller.go:119	reconciling	{"request": {"name":"cluster"}}
2025-09-30T16:57:27.004Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:27.005Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:27.131Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:57:29.697Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:57:29.698Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:57:29.698Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.794Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "15.222877954s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False, CanaryChecksSucceeding=Unknown"}
2025-09-30T16:57:29.794Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.832Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.899Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "15.101561369s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False, CanaryChecksSucceeding=Unknown"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:45.002Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.093Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)"}
2025-09-30T16:57:45.093Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.111Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.185Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:45.383Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.383Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.384Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.384Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.384Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.482Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:45.482Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.483Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.486Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "49m24.530179972s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False"}
2025-09-30T16:57:45.486Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.489Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.506Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.593Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "49m24.408255308s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:58:00.617Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:00.617Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:00.617Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:00.850Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "49m9.150861082s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False"}
2025-09-30T16:58:05.460Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:05.460Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:05.515Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:05.515Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:05.555Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:05.555Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:06.575Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:06.575Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.630Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:30.631Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:30.631Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.710Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:30.710Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.722Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:30.722Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.837Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.875Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:58:45.094Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:25.439Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.440Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.440Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:25.441Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.441Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.441Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:25.454Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.454Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.454Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:26.475Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:26.475Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:26.475Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T17:09:53.214Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T17:09:53.263Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.dns_controller	handler/enqueue_mapped.go:139	queueing dnsrecord	{"name": "default-wildcard", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.dns_controller	handler/enqueue_mapped.go:139	queueing dnsrecord	{"name": "default-wildcard", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.dns_controller	controller/controller.go:119	reconciling	{"request": {"name":"default-wildcard","namespace":"openshift-ingress-operator"}}
2025-09-30T17:09:53.274Z	INFO	operator.dns_controller	dns/controller.go:183	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org.","targets":["a3eb70b6b492b4cc3bc5ffdcafd560fc-439759926.us-east-1.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30,"dnsManagementPolicy":"Managed"}, "dnszone": {"tags":{"Name":"ci-ln-hg0gydt-76ef8-nm482-int","kubernetes.io/cluster/ci-ln-hg0gydt-76ef8-nm482":"owned"}}}
2025-09-30T17:09:53.274Z	INFO	operator.dns_controller	dns/controller.go:183	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org.","targets":["a3eb70b6b492b4cc3bc5ffdcafd560fc-439759926.us-east-1.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30,"dnsManagementPolicy":"Managed"}, "dnszone": {"id":"Z00287062J1ITQ61DDU2Z"}}
2025-09-30T17:09:53.305Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T17:39:37.351Z	INFO	operator.canary_controller	canary/daemonset.go:53	updated canary daemonset	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.DaemonSet{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"ingress-canary\", Namespace: \"openshift-ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ResourceVersion: \"38469\", ...},\n  \tSpec: v1.DaemonSetSpec{\n  \t\tSelector: &{MatchLabels: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}},\n  \t\tTemplate: v1.PodTemplateSpec{\n  \t\t\tObjectMeta: v1.ObjectMeta{\n  \t\t\t\t... // 9 identical fields\n  \t\t\t\tDeletionGracePeriodSeconds: nil,\n  \t\t\t\tLabels:                     {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"},\n  \t\t\t\tAnnotations: map[string]string{\n  \t\t\t\t\t\"ingress.operator.openshift.io/canary-serving-cert-hash\": strings.Join({\n- \t\t\t\t\t\t\"087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b\",\n+ \t\t\t\t\t\t\"ad7b4a2e5ec6effbc7959b551bab4006d553c52e1d931d147a8300d99b09b065\",\n  \t\t\t\t\t}, \"\"),\n  \t\t\t\t\t\"openshift.io/required-scc\":               \"restricted-v2\",\n  \t\t\t\t\t\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`,\n  \t\t\t\t},\n  \t\t\t\tOwnerReferences: nil,\n  \t\t\t\tFinalizers:      nil,\n  \t\t\t\tManagedFields:   nil,\n  \t\t\t},\n  \t\t\tSpec: {Volumes: {{Name: \"cert\", VolumeSource: {Secret: &{SecretName: \"canary-serving-cert\", DefaultMode: &272}}}}, Containers: {{Name: \"serve-healthcheck-canary\", Image: \"registry.build11.ci.openshift.org/ci-ln-hg0gydt/stable@sha256:a6\"..., Command: {\"ingress-operator\", \"serve-healthcheck\"}, Ports: {{ContainerPort: 8443, Protocol: \"TCP\"}, {ContainerPort: 8888, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n  \t\t},\n  \t\tUpdateStrategy:       {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"10%\"}, MaxSurge: &{}}},\n  \t\tMinReadySeconds:      0,\n  \t\tRevisionHistoryLimit: &10,\n  \t},\n  \tStatus: {CurrentNumberScheduled: 3, DesiredNumberScheduled: 3, NumberReady: 3, ObservedGeneration: 2, ...},\n  }\n"}
2025-09-30T17:39:37.351Z	DEBUG	operator.init.events	record/event.go:418	Canary serving cert rotated, updated pod template annotation hash: ad7b4a2e	{"type": "Normal", "object": {"kind":"DaemonSet","namespace":"openshift-ingress-canary","name":"ingress-canary","uid":"9d849944-b277-4943-980a-25e4c3f03514","apiVersion":"apps/v1","resourceVersion":"39516"}, "reason": "CanaryCertRotated"}
2025-09-30T17:39:37.980Z	INFO	operator.canary_controller	canary/daemonset.go:53	updated canary daemonset	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.DaemonSet{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"ingress-canary\", Namespace: \"openshift-ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ResourceVersion: \"39533\", ...},\n  \tSpec: v1.DaemonSetSpec{\n  \t\tSelector: &{MatchLabels: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}},\n  \t\tTemplate: v1.PodTemplateSpec{\n  \t\t\tObjectMeta: v1.ObjectMeta{\n  \t\t\t\t... // 9 identical fields\n  \t\t\t\tDeletionGracePeriodSeconds: nil,\n  \t\t\t\tLabels:                     {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"},\n  \t\t\t\tAnnotations: map[string]string{\n  \t\t\t\t\t\"ingress.operator.openshift.io/canary-serving-cert-hash\": strings.Join({\n- \t\t\t\t\t\t\"ad7b4a2e5ec6effbc7959b551bab4006d553c52e1d931d147a8300d99b09b065\",\n+ \t\t\t\t\t\t\"7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482\",\n  \t\t\t\t\t}, \"\"),\n  \t\t\t\t\t\"openshift.io/required-scc\":               \"restricted-v2\",\n  \t\t\t\t\t\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`,\n  \t\t\t\t},\n  \t\t\t\tOwnerReferences: nil,\n  \t\t\t\tFinalizers:      nil,\n  \t\t\t\tManagedFields:   nil,\n  \t\t\t},\n  \t\t\tSpec: {Volumes: {{Name: \"cert\", VolumeSource: {Secret: &{SecretName: \"canary-serving-cert\", DefaultMode: &272}}}}, Containers: {{Name: \"serve-healthcheck-canary\", Image: \"registry.build11.ci.openshift.org/ci-ln-hg0gydt/stable@sha256:a6\"..., Command: {\"ingress-operator\", \"serve-healthcheck\"}, Ports: {{ContainerPort: 8443, Protocol: \"TCP\"}, {ContainerPort: 8888, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n  \t\t},\n  \t\tUpdateStrategy:       {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"10%\"}, MaxSurge: &{}}},\n  \t\tMinReadySeconds:      0,\n  \t\tRevisionHistoryLimit: &10,\n  \t},\n  \tStatus: {CurrentNumberScheduled: 3, DesiredNumberScheduled: 3, NumberReady: 2, ObservedGeneration: 3, ...},\n  }\n"}
2025-09-30T17:39:37.980Z	DEBUG	operator.init.events	record/event.go:418	Canary serving cert rotated, updated pod template annotation hash: 7fc52b64	{"type": "Normal", "object": {"kind":"DaemonSet","namespace":"openshift-ingress-canary","name":"ingress-canary","uid":"9d849944-b277-4943-980a-25e4c3f03514","apiVersion":"apps/v1","resourceVersion":"39539"}, "reason": "CanaryCertRotated"}
2025-09-30T17:39:38.009Z	INFO	operator.canary_controller	canary/service.go:29	updated canary service	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.Service{\n  \tTypeMeta: {},\n  \tObjectMeta: v1.ObjectMeta{\n  \t\t... // 9 identical fields\n  \t\tDeletionGracePeriodSeconds: nil,\n  \t\tLabels:                     {\"ingress.openshift.io/canary\": \"canary_controller\"},\n  \t\tAnnotations: map[string]string{\n- \t\t\t\"service.alpha.openshift.io/serving-cert-signed-by\":  \"openshift-service-serving-signer@1759250750\",\n  \t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"canary-serving-cert\",\n- \t\t\t\"service.beta.openshift.io/serving-cert-signed-by\":   \"openshift-service-serving-signer@1759250750\",\n  \t\t},\n  \t\tOwnerReferences: {{APIVersion: \"apps/v1\", Kind: \"daemonset\", Name: \"ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ...}},\n  \t\tFinalizers:      nil,\n  \t\tManagedFields:   {{Manager: \"ingress-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 16:56:45 +0000 UTC\", ...}, {Manager: \"service-ca-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 17:39:37 +0000 UTC\", ...}},\n  \t},\n  \tSpec:   {Ports: {{Name: \"8443-tcp\", Protocol: \"TCP\", Port: 8443, TargetPort: {IntVal: 8443}, ...}, {Name: \"8888-tcp\", Protocol: \"TCP\", Port: 8888, TargetPort: {IntVal: 8888}, ...}}, Selector: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}, ClusterIP: \"172.30.225.114\", ClusterIPs: {\"172.30.225.114\"}, ...},\n  \tStatus: {},\n  }\n"}
oc -n $NS rollout status ds/$DS
^C
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 

5) Observe the DaemonSet rolling update
mjoseph@mjoseph-mac Downloads % oc -n $NS rollout status ds/$DS
daemon set "ingress-canary" successfully rolled out
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 

6) Confirm the annotation changed and pods were recreated
mjoseph@mjoseph-mac Downloads % oc -n $NS get ds $DS -o jsonpath='{.spec.template.metadata.annotations.ingress\.operator\.openshift\.io/canary-serving-cert-hash}'
7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482%                                                                                                                                                   mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $NS get pods -o custom-columns=NAME:.metadata.name,HASH:.metadata.annotations.ingress\\.operator\\.openshift\\.io/canary-serving-cert-hash --no-headers
ingress-canary-7bjx2   7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482
ingress-canary-gsjdd   7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482
ingress-canary-ntbxb   7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482

7) Inspect Events for traceability
mjoseph@mjoseph-mac Downloads % oc -n $NS get events --field-selector involvedObject.kind=DaemonSet,involvedObject.name=$DS -o json | jq '.items[] | select(.reason=="CanaryCertRotated")'
{
  "apiVersion": "v1",
  "count": 1,
  "eventTime": null,
  "firstTimestamp": "2025-09-30T16:56:45Z",
  "involvedObject": {
    "apiVersion": "apps/v1",
    "kind": "DaemonSet",
    "name": "ingress-canary",
    "namespace": "openshift-ingress-canary",
    "resourceVersion": "21570",
    "uid": "9d849944-b277-4943-980a-25e4c3f03514"
  },
  "kind": "Event",
  "lastTimestamp": "2025-09-30T16:56:45Z",
  "message": "Canary serving cert rotated, updated pod template annotation hash: 08783236",
  "metadata": {
    "creationTimestamp": "2025-09-30T16:56:45Z",
    "name": "ingress-canary.186a1dd51c252bb6",
    "namespace": "openshift-ingress-canary",
    "resourceVersion": "21571",
    "uid": "f796189f-6fed-4e17-9f09-ccaedc30a949"
  },
  "reason": "CanaryCertRotated",
  "reportingComponent": "canary_controller",
  "reportingInstance": "",
  "source": {
    "component": "canary_controller"
  },
  "type": "Normal"
}
{
  "apiVersion": "v1",
  "count": 1,
  "eventTime": null,
  "firstTimestamp": "2025-09-30T17:39:37Z",
  "involvedObject": {
    "apiVersion": "apps/v1",
    "kind": "DaemonSet",
    "name": "ingress-canary",
    "namespace": "openshift-ingress-canary",
    "resourceVersion": "39516",
    "uid": "9d849944-b277-4943-980a-25e4c3f03514"
  },
  "kind": "Event",
  "lastTimestamp": "2025-09-30T17:39:37Z",
  "message": "Canary serving cert rotated, updated pod template annotation hash: ad7b4a2e",
  "metadata": {
    "creationTimestamp": "2025-09-30T17:39:37Z",
    "name": "ingress-canary.186a202be5b6a42d",
    "namespace": "openshift-ingress-canary",
    "resourceVersion": "39518",
    "uid": "44a7ebf7-a7f3-420e-a257-0a18c4f76866"
  },
  "reason": "CanaryCertRotated",
  "reportingComponent": "canary_controller",
  "reportingInstance": "",
  "source": {
    "component": "canary_controller"
  },
  "type": "Normal"
}
{
  "apiVersion": "v1",
  "count": 1,
  "eventTime": null,
  "firstTimestamp": "2025-09-30T17:39:37Z",
  "involvedObject": {
    "apiVersion": "apps/v1",
    "kind": "DaemonSet",
    "name": "ingress-canary",
    "namespace": "openshift-ingress-canary",
    "resourceVersion": "39539",
    "uid": "9d849944-b277-4943-980a-25e4c3f03514"
  },
  "kind": "Event",
  "lastTimestamp": "2025-09-30T17:39:37Z",
  "message": "Canary serving cert rotated, updated pod template annotation hash: 7fc52b64",
  "metadata": {
    "creationTimestamp": "2025-09-30T17:39:37Z",
    "name": "ingress-canary.186a202c0b391244",
    "namespace": "openshift-ingress-canary",
    "resourceVersion": "39540",
    "uid": "99568d26-b3d7-4397-ba34-0ed52e87a96d"
  },
  "reason": "CanaryCertRotated",
  "reportingComponent": "canary_controller",
  "reportingInstance": "",
  "source": {
    "component": "canary_controller"
  },
  "type": "Normal"
}
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $NS get events | grep CanaryCertRotated
49m         Normal    CanaryCertRotated   daemonset/ingress-canary   Canary serving cert rotated, updated pod template annotation hash: 08783236
6m25s       Normal    CanaryCertRotated   daemonset/ingress-canary   Canary serving cert rotated, updated pod template annotation hash: ad7b4a2e
6m25s       Normal    CanaryCertRotated   daemonset/ingress-canary   Canary serving cert rotated, updated pod template annotation hash: 7fc52b64

mjoseph@mjoseph-mac Downloads % oc get co
NAME                                       VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      36m     
baremetal                                  4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
cloud-controller-manager                   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      62m     
cloud-credential                           4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      64m     
cluster-autoscaler                         4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
config-operator                            4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      60m     
console                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      43m     
control-plane-machine-set                  4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      55m     
csi-snapshot-controller                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
dns                                        4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      58m     
etcd                                       4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      58m     
image-registry                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      49m     
ingress                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      48m     
insights                                   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
kube-apiserver                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      54m     
kube-controller-manager                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      54m     
kube-scheduler                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      56m     
kube-storage-version-migrator              4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      60m     
machine-api                                4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      48m     
machine-approver                           4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
machine-config                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      57m     
marketplace                                4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
monitoring                                 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      46m     
network                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      61m     
node-tuning                                4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      11m     
olm                                        4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
openshift-apiserver                        4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      49m     
openshift-controller-manager               4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      49m     
openshift-samples                          4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      48m     
operator-lifecycle-manager                 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
operator-lifecycle-manager-catalog         4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
operator-lifecycle-manager-packageserver   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      54m     
service-ca                                 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      60m     
storage                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      58m

Hence marking as verified
/label qe-approved
/verified by @mjoseph

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Sep 30, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 30, 2025

@bentito: This pull request references Jira Issue OCPBUGS-58145, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.21.0) matches configured target version for branch (4.21.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @melvinjoseph86

In response to this:

This PR attempts to fix this bug, here's an overview of what it's doing:

• Compute TLS secret hash (tls.crt, tls.key, optional ca.crt)
• Annotate pod template with canary-serving-cert hash
• Watch canary serving cert Secret to trigger reconcile
• Emit event when cert hash changes (traceability)
• Update desiredCanaryDaemonSet(canaryImage, certHash)
• Reconcile logic compares/propagates hash annotation
• Add unit tests for hash compute and DaemonSet change detection

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Sep 30, 2025
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 30, 2025

@melvinjoseph86: This PR has been marked as verified by @mjoseph.

In response to this:

Tested with cluster bot


mjoseph@mjoseph-mac Downloads % oc get clusterversion
NAME      VERSION                                                AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         25m     Cluster version is 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest
1) Set some default variables 
mjoseph@mjoseph-mac Downloads % NS=openshift-ingress-canary
OP_NS=openshift-ingress-operator
DS=ingress-canary
SECRET=canary-serving-cert

2) Record the current PodTemplate annotation and current pods' annotations
mjoseph@mjoseph-mac Downloads % oc -n $NS get ds $DS -o jsonpath='{.spec.template.metadata.annotations.ingress\.operator\.openshift\.io/canary-serving-cert-hash}' || echo "(none)"
087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b%                                                                                                                                                   mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $NS get pods -o custom-columns=NAME:.metadata.name,HASH:.metadata.annotations.ingress\\.operator\\.openshift\\.io/canary-serving-cert-hash --no-headers
ingress-canary-gpds9   087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b
ingress-canary-kln85   087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b
ingress-canary-v4flf   087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b

3) Simulate a rotation of the serving cert by create new base64-encoded values (portable):
mjoseph@mjoseph-mac Downloads % NEWCRT_B64=$(echo -n "rotation-$(date -u +%s)" | base64 | tr -d '\n')
mjoseph@mjoseph-mac Downloads % NEWKEY_B64=$(echo -n "rotation-key-$(date -u +%s)" | base64 | tr -d '\n')
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
- Patch the Secret in-place to replace `tls.crt` and `tls.key`:
mjoseph@mjoseph-mac Downloads % oc -n $NS patch secret $SECRET --type=json -p "$(jq -n --arg c "$NEWCRT_B64" --arg k "$NEWKEY_B64" '[{op:"replace",path:"/data/tls.crt",value:$c},{op:"replace",path:"/data/tls.key",value:$k}]')"
secret/canary-serving-cert patched
mjoseph@mjoseph-mac Downloads % oc get secret -n openshift-ingress-canary
NAME                       TYPE                      DATA   AGE
builder-dockercfg-9vb4h    kubernetes.io/dockercfg   1      44m
canary-serving-cert        kubernetes.io/tls         2      44m
default-dockercfg-r4c64    kubernetes.io/dockercfg   1      44m
deployer-dockercfg-tzxg5   kubernetes.io/dockercfg   1      44m

mjoseph@mjoseph-mac Downloads % oc get secret -n openshift-ingress-canary canary-serving-cert -oyaml
apiVersion: v1
data:
 tls.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVFVENDQXZtZ0F3SUJBZ0lJWklWMXdEYVdYLzR3RFFZSktvWklodmNOQVFFTEJRQXdOakUwTURJR0ExVUUKQXd3cmIzQmxibk5vYVdaMExYTmxjblpwWTJVdGMyVnlkbWx1WnkxemFXZHVaWEpBTVRjMU9USTFNRGMxTURBZQpGdzB5TlRBNU16QXhOek01TXpaYUZ3MHlOekE1TXpBeE56TTVNemRhTURZeE5EQXlCZ05WQkFNVEsybHVaM0psCmMzTXRZMkZ1WVhKNUxtOXdaVzV6YUdsbWRDMXBibWR5WlhOekxXTmhibUZ5ZVM1emRtTXdnZ0VpTUEwR0NTcUcKU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLQW9JQkFRQ3lRbHFsY2J1WmY2SjZjQ0JjaWxSVVFwaitWbXl4MEw5WgpPb1gvU2JwalFiQm0rekhGVDRpYzBLclRIM2NLeTJJT0o0MVF0STJRWHRRdXhyMEJCcmRoRnZxMytqaTFoV3VDCkdOVUt5WDRJZTV5MXNDUWF5di9RcUl6TXNDOVNKUGd3V0VaUmJyZ1ZQRWFuak8yRVFRRzdnSHpJWVdHZ0ZtR0cKak1RT3pOQWtCODZlQWFDYXFxRU54dkJEVXE3Uzh3a1gyb0tpWm4xZGF5UXlKRmRBVWlsRWlldEdrUXNRVzFlcAo0VHIwVlFxWGtOblp4YVFDN0JXRmpaMkkySW8zTkVUcTNJaVZYWC95dEhtbGhLL1Y5ZmlJRmZKZnVPRGhxUE1PCkxZU0xkcEhkajRSdE1aOVZCdHZSaXZlT1FtQVBtTk9yQkxuS3dKM0tPYm5ubVNHZERGLzlBZ01CQUFHamdnRWgKTUlJQkhUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0V3WURWUjBsQkF3d0NnWUlLd1lCQlFVSEF3RXdEQVlEVlIwVApBUUgvQkFJd0FEQWRCZ05WSFE0RUZnUVU4b0V2N3AxcDdLcUhsei9ZNi9tTGFTRGpnemN3SHdZRFZSMGpCQmd3CkZvQVVYSGE2NzNTekVLaVZpMjFOWk1majBlT0xUQkl3Y1FZRFZSMFJCR293YUlJcmFXNW5jbVZ6Y3kxallXNWgKY25rdWIzQmxibk5vYVdaMExXbHVaM0psYzNNdFkyRnVZWEo1TG5OMlk0STVhVzVuY21WemN5MWpZVzVoY25rdQpiM0JsYm5Ob2FXWjBMV2x1WjNKbGMzTXRZMkZ1WVhKNUxuTjJZeTVqYkhWemRHVnlMbXh2WTJGc01EVUdDeXNHCkFRUUJrZ2dSWkFJQkJDWVRKREkzTlRsbU16WmpMVE00WW1VdE5EVTBNUzFoTkRjeExURmxZamczWWpFek5XRXoKWkRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVFmc2xNU08wWCtzeDlmcldzajdxakF0czBEcU4xd0p6cjVuYgpHaTh3TENwbXd6dkJaM1FuU1JhdXBqL1Z6NDRQUmZLTGRuT2ZTRkNzSTd5U0YzNHkyckI0eVpaRjhIRUsxZ1pJCk9jU3B0aFVDQTRPOEhVV245WDBMSWo0T2ZGYUxFUEIxQUxtb1RxMGI0T3c4YzlFN284T0ZUR1VvWEJPZC85YVkKenByaGRhZTRiTFFTOG1TeVRIK0t4czR5UExMSEZMR2hmaFo1Y0lyaGFuazBQV0V2QnJHNzdsNzFtVE05aG1WRwowc3BpOElicDZWTGtuZVNjdUxuMnZteUZWUHh1TC96bjBBdE11R3pvOTkyN0FmSkkreEhkTFFGZGRDRUFRZjFMCk5FUXorTG1YeWxkSWJLV3dCYUl1eTRTbFBLZXRnM2hOVnZ4Q29KaEluT3JaYi9ENG1nPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRFVUQ0NBam1nQXdJQkFnSUlWYmx1WmJFTVlWNHdEUVlKS29aSWh2Y05BUUVMQlFBd05qRTBNRElHQTFVRQpBd3dyYjNCbGJuTm9hV1owTFhObGNuWnBZMlV0YzJWeWRtbHVaeTF6YVdkdVpYSkFNVGMxT1RJMU1EYzFNREFlCkZ3MHlOVEE1TXpBeE5qUTFORGxhRncweU56RXhNamt4TmpRMU5UQmFNRFl4TkRBeUJnTlZCQU1NSzI5d1pXNXoKYUdsbWRDMXpaWEoyYVdObExYTmxjblpwYm1jdGMybG5ibVZ5UURFM05Ua3lOVEEzTlRBd2dnRWlNQTBHQ1NxRwpTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDaS9UVzlaU1FNZ0UyMXFLdlg2T24rek9kSkJ0V2hHa1YxCnJiUDRFU1lOd3hZNnJZWXFPSG0ybFpLUmc1eitDODlYUUZhUkFjOUpUVkMwbGZwanJxWU95bUhnTTdwWjZ5MEgKZTRrblhRL0M4TUgzZGR2dVI2V1dBU1liNnduRGJaaE0za2RxZFNRb3VOVk8zNlkzZXByekF4UE9ST3JRVTQ0UwowUmR4NUcvQThod3hNL3QwajJqWjhWMlVjSCtLalkvL1Azc1pjQWlVL3V4VEVSWkZhTFBhcHBLTUZ4QjRud0QxClNFNVNyRk1oZGNGZEVVRm1VVVlpeG9mNTJscTB0M3Z0dnFhZFVOZUYzc0F1d29Tazc1N29YZWFoVW9jQ3FoQngKd05aQ3Y3UFBlTHM0OFBndGRBMHlvNy9IWmxqcWQydzduem5SajQxQ2VPamxVaUlxblNlWEFnTUJBQUdqWXpCaApNQTRHQTFVZER3RUIvd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUmNkcnJ2CmRMTVFxSldMYlUxa3grUFI0NHRNRWpBZkJnTlZIU01FR0RBV2dCUmNkcnJ2ZExNUXFKV0xiVTFreCtQUjQ0dE0KRWpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVJCeTVqV1puZ1YzUy90SGJKQzR1dUJDSXBYazczb1V5RGpqTQpKTzYwRnNMMWNTSFM0SVdwN251T1VEekowTjBrcERwanl0WTUzMlpFeENncUFVUTVNOUhzTlFYS2pnQUdTZVJxCnNTUnBGZzN5aU5yazVyY3ZtNlNzYzNJTjJsanplQlRZdFRUd2N2dmY3aVBpalVvRmFmaVZiUXNCRm5MdmQ0aDAKVSszR0ZORzRTeldURVBUeG8vVURRL3hpSUdnbTRaSHJwMkJjWjhBYUx5UG5HZlI1NytOSERQdThMM1NTT05aWApmTVhsN0lhaFRyNWIvZDBqQ2pDZ2hlRytzMkpodEIyNTJVVEdINmI2QWFEem1OSTFNOWd1Z29mNWgra3RMMVpXCmxiNVd6MGZFVkpTQkxqWjB1aCtodHNXNCtqNk01WkxLWjJBcExzVmU3T0JuYmFIM21RPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
 tls.key: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFb1FJQkFBS0NBUUVBc2tKYXBYRzdtWCtpZW5BZ1hJcFVWRUtZL2xac3NkQy9XVHFGLzBtNlkwR3dadnN4CnhVK0luTkNxMHg5M0NzdGlEaWVOVUxTTmtGN1VMc2E5QVFhM1lSYjZ0L280dFlWcmdoalZDc2wrQ0h1Y3RiQWsKR3NyLzBLaU16TEF2VWlUNE1GaEdVVzY0RlR4R3A0enRoRUVCdTRCOHlHRmhvQlpoaG96RURzelFKQWZPbmdHZwptcXFoRGNid1ExS3Uwdk1KRjlxQ29tWjlYV3NrTWlSWFFGSXBSSW5yUnBFTEVGdFhxZUU2OUZVS2w1RFoyY1drCkF1d1ZoWTJkaU5pS056UkU2dHlJbFYxLzhyUjVwWVN2MWZYNGlCWHlYN2pnNGFqekRpMkVpM2FSM1krRWJUR2YKVlFiYjBZcjNqa0pnRDVqVHF3UzV5c0NkeWptNTU1a2huUXhmL1FJREFRQUJBb0gvSGlEM0NjTmNwc3pEVnRDcgpUYUdqcFdzbWtUdGNtZ3liNUcrWHJnVUQ5cGNzV20xWmNpVDlkZVNmM2JkUzREeTZ2TU5RQStxQWFwbmh1cnVCCjBpcnQySWxPbVhDbXBua0xld1ZsR3lhNFpJV0c5aGhYL0ZTdDNjaTRHYXFSL0dRSWJaNk9pVGhyWmdqWGlOSVEKbFhHdlNRUWtIaVN0UEsxVndWaUJJMFJ2QW4rVUtmWXA3dmc5dHM1QTBnd3ZCMFVCbVpOK2x4QzJXVWNtRWNHRQpMU3dEWC96eHN5VEliMEVMOTFuaVVDWmFLTVB4KzdPdmVZbm9BRDZiR2NxVFpjTlFYK1I3eGxvVlMvT0llQlZDCmFEMmhleko5RU8vcEU0Z2hxeUI4T1J2N0lOMXh5eHZ0L1hYSlp3R2E0K09wc3RGVzE2OGxvekFmWlZEZzdiaXQKN3VncEFvR0JBTitsa1NiZGY1VTJldVZhNFZpSGg5Rk55Smt3OFdRSWt3bExmU2JsMFFwNGVkM29oZmhhNFVjMAp1bndzS0V2VmRKUGlpZjVyL0ViajBBQVNQVCtCemlFSDRUSUNyU3BnSWpudG55aDhOU1BKZWpjenJsSGpjVks5CllPMWxEOTVqUUV4MGZXM2RPamZWZE01Y09YTmZjWUk4YmdwWDlwc3FpamtReHZleW84L2xBb0dCQU13TDdNb24KQ1BZUDNlQ3VuY2dsS0lJb2NDazhzKzk1SndpSEtaQjA4VG9pcHVPK2VWMm9NNUdVckdCY3FZaGNaeFVUTDNDTApGTEtaQTFsdGNkMEtOUm56VFQvdWo0LzJXeGp3MU5Gcm9UZU1PWFRXclFROFh0WXh2RjYrWGdLZnE4ZTdlcGhTCjRjZ0VlWndxQkYrQUJGVGJkeWh5MUhMRDhwdG1WZWJwZVY0NUFvR0FRWVJnWEhGaEVkWk85R2ZHUWtnWitxLzkKeHhyMmJHYXlwSE52NjNoQjMvNWhQU2dFTHpoNGJJSys2dWVJdW9hQ3kxdi9Md0IxZkZMNnVUcWVwTDhqSXJ3TApOYnBnL2x3QlN0VVkzaVJjOE9ueUdyL1NIbjB4MHBwV0RkV2dhM2JKL0Z2WDd0cHpWcTM2dFFXQmN4SVB6NlppCldqZGdzQjRyTmdKN04wOG0vQjBDZ1lFQXJVQmExL0o0KzJUUmdoc0J3WlU3WmZpcjgzZWxXR3BJS0VlTGxTcVcKUUVSMzY5d285TXNDcmxQTVlTMk1qVW5ZQk43N1l0WFV4eWlBM1llK0xHNmtVOXB2OXg2Nkt3U0VxUktqd1FWZwpKK2dhU1FpV2M0aEhCaHZJMDdGVDVXcHAwdmR1eXJObEhnNGVxK2hKT2tWYWFDckliRUhEL1lZL0VvKzcxSXJkCkVKRUNnWUJURXovdDh4S1JGdlExRU5DTjZjZkN0d2w3OXVGdGJWT3RXT3RRN00ybFJ4RTdQRXMya0lmS3liT3MKUVAxMlNEY004OWN0Nnl5aG5jQm9DQUFpU3F6Q0tCWFVxMTVqRXQ5ZjBwdEpndXNHakpvb1hRR29BRnRoNG8veQpYUmJKWnNROFpYRUVCVVhieXd1dU1pbXFmejNHeERISFZGY08wU2d4bTUwYzNUSUxhdz09Ci0tLS0tRU5EIFJTQSBQUklWQVRFIEtFWS0tLS0tCg==
kind: Secret
metadata:
 annotations:
   openshift.io/description: 'Secret contains a pair signed serving certificate/key
     that is generated by Service CA operator for service/ingress-canary with hostname
     ingress-canary.openshift-ingress-canary.svc and is annotated to the service
     with annotating a service resource with ''service.beta.openshift.io/serving-cert-secret-name:
     canary-serving-cert''. The certificate is valid for 2 years.'
   openshift.io/owning-component: service-ca
   service.alpha.openshift.io/expiry: "2027-09-30T17:39:37Z"
   service.beta.openshift.io/expiry: "2027-09-30T17:39:37Z"
   service.beta.openshift.io/originating-service-name: ingress-canary
   service.beta.openshift.io/originating-service-uid: 2759f36c-38be-4541-a471-1eb87b135a3d
 creationTimestamp: "2025-09-30T16:56:45Z"
 name: canary-serving-cert
 namespace: openshift-ingress-canary
 ownerReferences:
 - apiVersion: v1
   kind: Service
   name: ingress-canary
   uid: 2759f36c-38be-4541-a471-1eb87b135a3d
 resourceVersion: "39537"
 uid: e3da7a32-7159-4283-9a65-cda4c761cf1f
type: kubernetes.io/tls

4) Watch operator logs for the rollout trigger and look for the event/log indicating a canary cert-driven update
The event emission has a `updated canary daemonset` log line with the  reasone `CanaryCertRotated`.
mjoseph@mjoseph-mac Downloads % oc -n $OP_NS get pods -l name=ingress-operator -o name || oc -n $OP_NS get pods | grep ingress-operator
pod/ingress-operator-76457bd877-66k7l
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $OP_NS logs -f ingress-operator-76457bd877-66k7l | sed -n '/CanaryCertRotated/,/updated canary daemonset/p'
Defaulted container "ingress-operator" out of: ingress-operator, kube-rbac-proxy
2025-09-30T16:56:45.579Z	DEBUG	operator.init.events	record/event.go:418	Canary serving cert rotated, updated pod template annotation hash: 08783236	{"type": "Normal", "object": {"kind":"DaemonSet","namespace":"openshift-ingress-canary","name":"ingress-canary","uid":"9d849944-b277-4943-980a-25e4c3f03514","apiVersion":"apps/v1","resourceVersion":"21570"}, "reason": "CanaryCertRotated"}
2025-09-30T16:56:45.673Z	INFO	operator.canary_controller	canary/service.go:29	updated canary service	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.Service{\n  \tTypeMeta: {},\n  \tObjectMeta: v1.ObjectMeta{\n  \t\t... // 9 identical fields\n  \t\tDeletionGracePeriodSeconds: nil,\n  \t\tLabels:                     {\"ingress.openshift.io/canary\": \"canary_controller\"},\n  \t\tAnnotations: map[string]string{\n- \t\t\t\"service.alpha.openshift.io/serving-cert-signed-by\":  \"openshift-service-serving-signer@1759250750\",\n  \t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"canary-serving-cert\",\n- \t\t\t\"service.beta.openshift.io/serving-cert-signed-by\":   \"openshift-service-serving-signer@1759250750\",\n  \t\t},\n  \t\tOwnerReferences: {{APIVersion: \"apps/v1\", Kind: \"daemonset\", Name: \"ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ...}},\n  \t\tFinalizers:      nil,\n  \t\tManagedFields:   {{Manager: \"ingress-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 16:56:45 +0000 UTC\", ...}, {Manager: \"service-ca-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 16:56:45 +0000 UTC\", ...}},\n  \t},\n  \tSpec:   {Ports: {{Name: \"8443-tcp\", Protocol: \"TCP\", Port: 8443, TargetPort: {IntVal: 8443}, ...}, {Name: \"8888-tcp\", Protocol: \"TCP\", Port: 8888, TargetPort: {IntVal: 8888}, ...}}, Selector: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}, ClusterIP: \"172.30.225.114\", ClusterIPs: {\"172.30.225.114\"}, ...},\n  \tStatus: {},\n  }\n"}
2025-09-30T16:56:45.718Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/4 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/4 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 3 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:56:45.818Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:45.835Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:45.867Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:56:57.952Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:56:57.952Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.716Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:56:58.716Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:56:58.716Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.939Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.942Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:56:58.942Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:58.958Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:56:59.032Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:02.954Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:57:02.955Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:02.973Z	INFO	operator.configurable_route_controller	controller/controller.go:119	reconciling	{"request": {"name":"cluster"}}
2025-09-30T16:57:02.973Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:02.974Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:02.974Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:03.173Z	INFO	operator.configurable_route_controller	controller/controller.go:119	reconciling	{"request": {"name":"cluster"}}
2025-09-30T16:57:03.173Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:03.173Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:03.334Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:57:03.334Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:03.434Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:57:27.004Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:27.004Z	INFO	operator.configurable_route_controller	controller/controller.go:119	reconciling	{"request": {"name":"cluster"}}
2025-09-30T16:57:27.004Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T16:57:27.005Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:27.131Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: DeploymentAvailable=False (DeploymentUnavailable: The deployment has Available status condition set to False (reason: MinimumReplicasUnavailable) with message: Deployment does not have minimum availability.), DeploymentReplicasMinAvailable=False (DeploymentMinimumReplicasNotMet: 0/2 of replicas are available, max unavailable is 1: Some pods are not scheduled: Pod \"router-default-767b8b698b-8lt99\" cannot be scheduled: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/5 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 4 Preemption is not helpful for scheduling. Make sure you have sufficient worker nodes.)"}
2025-09-30T16:57:29.697Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:57:29.698Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:57:29.698Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.790Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.794Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "15.222877954s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False, CanaryChecksSucceeding=Unknown"}
2025-09-30T16:57:29.794Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.832Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:29.899Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "15.101561369s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False, CanaryChecksSucceeding=Unknown"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:45.002Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.087Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.093Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)"}
2025-09-30T16:57:45.093Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.111Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.185Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "1m0s", "error": "IngressController is degraded: CanaryChecksSucceeding=Unknown (CanaryRouteNotAdmitted: Canary route is not admitted by the default ingress controller)"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:45.383Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.383Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.384Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.384Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.384Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.482Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:57:45.482Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.483Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.486Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "49m24.530179972s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False"}
2025-09-30T16:57:45.486Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.489Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.506Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:57:45.593Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "49m24.408255308s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False"}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:58:00.617Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:00.617Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:00.617Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:00.850Z	ERROR	operator.ingress_controller	controller/controller.go:119	got retryable error; requeueing	{"after": "49m9.150861082s", "error": "IngressController may become degraded soon: DeploymentReplicasAllAvailable=False"}
2025-09-30T16:58:05.460Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:05.460Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:05.515Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:05.515Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:05.555Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:05.555Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:06.575Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:06.575Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.630Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:30.631Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingress	{"name": "default", "related": ""}
2025-09-30T16:58:30.631Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.710Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:30.710Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.722Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:58:30.722Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.ingressclass_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.certificate_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.817Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.837Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:58:30.875Z	INFO	operator.status_controller	controller/controller.go:119	Reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
Validity period of the certificate for "*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org" is unset, resetting to 17520h0m0s!
2025-09-30T16:58:45.094Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:25.439Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.440Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.440Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:25.441Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.441Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.441Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:25.454Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.454Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:25.454Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T16:59:26.475Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:26.475Z	INFO	operator.route_metrics_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default"}
2025-09-30T16:59:26.475Z	INFO	operator.route_metrics_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T17:09:53.214Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T17:09:53.263Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.dns_controller	handler/enqueue_mapped.go:139	queueing dnsrecord	{"name": "default-wildcard", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.ingress_controller	handler/enqueue_mapped.go:139	queueing ingresscontroller	{"name": "default", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.dns_controller	handler/enqueue_mapped.go:139	queueing dnsrecord	{"name": "default-wildcard", "related": ""}
2025-09-30T17:09:53.263Z	INFO	operator.dns_controller	controller/controller.go:119	reconciling	{"request": {"name":"default-wildcard","namespace":"openshift-ingress-operator"}}
2025-09-30T17:09:53.274Z	INFO	operator.dns_controller	dns/controller.go:183	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org.","targets":["a3eb70b6b492b4cc3bc5ffdcafd560fc-439759926.us-east-1.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30,"dnsManagementPolicy":"Managed"}, "dnszone": {"tags":{"Name":"ci-ln-hg0gydt-76ef8-nm482-int","kubernetes.io/cluster/ci-ln-hg0gydt-76ef8-nm482":"owned"}}}
2025-09-30T17:09:53.274Z	INFO	operator.dns_controller	dns/controller.go:183	skipping zone to which the DNS record is already published	{"record": {"dnsName":"*.apps.ci-ln-hg0gydt-76ef8.aws-2.ci.openshift.org.","targets":["a3eb70b6b492b4cc3bc5ffdcafd560fc-439759926.us-east-1.elb.amazonaws.com"],"recordType":"CNAME","recordTTL":30,"dnsManagementPolicy":"Managed"}, "dnszone": {"id":"Z00287062J1ITQ61DDU2Z"}}
2025-09-30T17:09:53.305Z	INFO	operator.ingress_controller	controller/controller.go:119	reconciling	{"request": {"name":"default","namespace":"openshift-ingress-operator"}}
2025-09-30T17:39:37.351Z	INFO	operator.canary_controller	canary/daemonset.go:53	updated canary daemonset	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.DaemonSet{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"ingress-canary\", Namespace: \"openshift-ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ResourceVersion: \"38469\", ...},\n  \tSpec: v1.DaemonSetSpec{\n  \t\tSelector: &{MatchLabels: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}},\n  \t\tTemplate: v1.PodTemplateSpec{\n  \t\t\tObjectMeta: v1.ObjectMeta{\n  \t\t\t\t... // 9 identical fields\n  \t\t\t\tDeletionGracePeriodSeconds: nil,\n  \t\t\t\tLabels:                     {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"},\n  \t\t\t\tAnnotations: map[string]string{\n  \t\t\t\t\t\"ingress.operator.openshift.io/canary-serving-cert-hash\": strings.Join({\n- \t\t\t\t\t\t\"087832368955c8e824189777bc96d99f32762e504dcd0ae13007891e2bf7b36b\",\n+ \t\t\t\t\t\t\"ad7b4a2e5ec6effbc7959b551bab4006d553c52e1d931d147a8300d99b09b065\",\n  \t\t\t\t\t}, \"\"),\n  \t\t\t\t\t\"openshift.io/required-scc\":               \"restricted-v2\",\n  \t\t\t\t\t\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`,\n  \t\t\t\t},\n  \t\t\t\tOwnerReferences: nil,\n  \t\t\t\tFinalizers:      nil,\n  \t\t\t\tManagedFields:   nil,\n  \t\t\t},\n  \t\t\tSpec: {Volumes: {{Name: \"cert\", VolumeSource: {Secret: &{SecretName: \"canary-serving-cert\", DefaultMode: &272}}}}, Containers: {{Name: \"serve-healthcheck-canary\", Image: \"registry.build11.ci.openshift.org/ci-ln-hg0gydt/stable@sha256:a6\"..., Command: {\"ingress-operator\", \"serve-healthcheck\"}, Ports: {{ContainerPort: 8443, Protocol: \"TCP\"}, {ContainerPort: 8888, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n  \t\t},\n  \t\tUpdateStrategy:       {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"10%\"}, MaxSurge: &{}}},\n  \t\tMinReadySeconds:      0,\n  \t\tRevisionHistoryLimit: &10,\n  \t},\n  \tStatus: {CurrentNumberScheduled: 3, DesiredNumberScheduled: 3, NumberReady: 3, ObservedGeneration: 2, ...},\n  }\n"}
2025-09-30T17:39:37.351Z	DEBUG	operator.init.events	record/event.go:418	Canary serving cert rotated, updated pod template annotation hash: ad7b4a2e	{"type": "Normal", "object": {"kind":"DaemonSet","namespace":"openshift-ingress-canary","name":"ingress-canary","uid":"9d849944-b277-4943-980a-25e4c3f03514","apiVersion":"apps/v1","resourceVersion":"39516"}, "reason": "CanaryCertRotated"}
2025-09-30T17:39:37.980Z	INFO	operator.canary_controller	canary/daemonset.go:53	updated canary daemonset	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.DaemonSet{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"ingress-canary\", Namespace: \"openshift-ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ResourceVersion: \"39533\", ...},\n  \tSpec: v1.DaemonSetSpec{\n  \t\tSelector: &{MatchLabels: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}},\n  \t\tTemplate: v1.PodTemplateSpec{\n  \t\t\tObjectMeta: v1.ObjectMeta{\n  \t\t\t\t... // 9 identical fields\n  \t\t\t\tDeletionGracePeriodSeconds: nil,\n  \t\t\t\tLabels:                     {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"},\n  \t\t\t\tAnnotations: map[string]string{\n  \t\t\t\t\t\"ingress.operator.openshift.io/canary-serving-cert-hash\": strings.Join({\n- \t\t\t\t\t\t\"ad7b4a2e5ec6effbc7959b551bab4006d553c52e1d931d147a8300d99b09b065\",\n+ \t\t\t\t\t\t\"7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482\",\n  \t\t\t\t\t}, \"\"),\n  \t\t\t\t\t\"openshift.io/required-scc\":               \"restricted-v2\",\n  \t\t\t\t\t\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`,\n  \t\t\t\t},\n  \t\t\t\tOwnerReferences: nil,\n  \t\t\t\tFinalizers:      nil,\n  \t\t\t\tManagedFields:   nil,\n  \t\t\t},\n  \t\t\tSpec: {Volumes: {{Name: \"cert\", VolumeSource: {Secret: &{SecretName: \"canary-serving-cert\", DefaultMode: &272}}}}, Containers: {{Name: \"serve-healthcheck-canary\", Image: \"registry.build11.ci.openshift.org/ci-ln-hg0gydt/stable@sha256:a6\"..., Command: {\"ingress-operator\", \"serve-healthcheck\"}, Ports: {{ContainerPort: 8443, Protocol: \"TCP\"}, {ContainerPort: 8888, Protocol: \"TCP\"}}, ...}}, RestartPolicy: \"Always\", TerminationGracePeriodSeconds: &30, ...},\n  \t\t},\n  \t\tUpdateStrategy:       {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"10%\"}, MaxSurge: &{}}},\n  \t\tMinReadySeconds:      0,\n  \t\tRevisionHistoryLimit: &10,\n  \t},\n  \tStatus: {CurrentNumberScheduled: 3, DesiredNumberScheduled: 3, NumberReady: 2, ObservedGeneration: 3, ...},\n  }\n"}
2025-09-30T17:39:37.980Z	DEBUG	operator.init.events	record/event.go:418	Canary serving cert rotated, updated pod template annotation hash: 7fc52b64	{"type": "Normal", "object": {"kind":"DaemonSet","namespace":"openshift-ingress-canary","name":"ingress-canary","uid":"9d849944-b277-4943-980a-25e4c3f03514","apiVersion":"apps/v1","resourceVersion":"39539"}, "reason": "CanaryCertRotated"}
2025-09-30T17:39:38.009Z	INFO	operator.canary_controller	canary/service.go:29	updated canary service	{"namespace": "openshift-ingress-canary", "name": "ingress-canary", "diff": "  &v1.Service{\n  \tTypeMeta: {},\n  \tObjectMeta: v1.ObjectMeta{\n  \t\t... // 9 identical fields\n  \t\tDeletionGracePeriodSeconds: nil,\n  \t\tLabels:                     {\"ingress.openshift.io/canary\": \"canary_controller\"},\n  \t\tAnnotations: map[string]string{\n- \t\t\t\"service.alpha.openshift.io/serving-cert-signed-by\":  \"openshift-service-serving-signer@1759250750\",\n  \t\t\t\"service.beta.openshift.io/serving-cert-secret-name\": \"canary-serving-cert\",\n- \t\t\t\"service.beta.openshift.io/serving-cert-signed-by\":   \"openshift-service-serving-signer@1759250750\",\n  \t\t},\n  \t\tOwnerReferences: {{APIVersion: \"apps/v1\", Kind: \"daemonset\", Name: \"ingress-canary\", UID: \"9d849944-b277-4943-980a-25e4c3f03514\", ...}},\n  \t\tFinalizers:      nil,\n  \t\tManagedFields:   {{Manager: \"ingress-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 16:56:45 +0000 UTC\", ...}, {Manager: \"service-ca-operator\", Operation: \"Update\", APIVersion: \"v1\", Time: s\"2025-09-30 17:39:37 +0000 UTC\", ...}},\n  \t},\n  \tSpec:   {Ports: {{Name: \"8443-tcp\", Protocol: \"TCP\", Port: 8443, TargetPort: {IntVal: 8443}, ...}, {Name: \"8888-tcp\", Protocol: \"TCP\", Port: 8888, TargetPort: {IntVal: 8888}, ...}}, Selector: {\"ingresscanary.operator.openshift.io/daemonset-ingresscanary\": \"canary_controller\"}, ClusterIP: \"172.30.225.114\", ClusterIPs: {\"172.30.225.114\"}, ...},\n  \tStatus: {},\n  }\n"}
oc -n $NS rollout status ds/$DS
^C
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 

5) Observe the DaemonSet rolling update
mjoseph@mjoseph-mac Downloads % oc -n $NS rollout status ds/$DS
daemon set "ingress-canary" successfully rolled out
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 

6) Confirm the annotation changed and pods were recreated
mjoseph@mjoseph-mac Downloads % oc -n $NS get ds $DS -o jsonpath='{.spec.template.metadata.annotations.ingress\.operator\.openshift\.io/canary-serving-cert-hash}'
7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482%                                                                                                                                                   mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $NS get pods -o custom-columns=NAME:.metadata.name,HASH:.metadata.annotations.ingress\\.operator\\.openshift\\.io/canary-serving-cert-hash --no-headers
ingress-canary-7bjx2   7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482
ingress-canary-gsjdd   7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482
ingress-canary-ntbxb   7fc52b64806b77d13a35dbcd76a7c28985f04fd33c2698ff5194eaa01a3ca482

7) Inspect Events for traceability
mjoseph@mjoseph-mac Downloads % oc -n $NS get events --field-selector involvedObject.kind=DaemonSet,involvedObject.name=$DS -o json | jq '.items[] | select(.reason=="CanaryCertRotated")'
{
 "apiVersion": "v1",
 "count": 1,
 "eventTime": null,
 "firstTimestamp": "2025-09-30T16:56:45Z",
 "involvedObject": {
   "apiVersion": "apps/v1",
   "kind": "DaemonSet",
   "name": "ingress-canary",
   "namespace": "openshift-ingress-canary",
   "resourceVersion": "21570",
   "uid": "9d849944-b277-4943-980a-25e4c3f03514"
 },
 "kind": "Event",
 "lastTimestamp": "2025-09-30T16:56:45Z",
 "message": "Canary serving cert rotated, updated pod template annotation hash: 08783236",
 "metadata": {
   "creationTimestamp": "2025-09-30T16:56:45Z",
   "name": "ingress-canary.186a1dd51c252bb6",
   "namespace": "openshift-ingress-canary",
   "resourceVersion": "21571",
   "uid": "f796189f-6fed-4e17-9f09-ccaedc30a949"
 },
 "reason": "CanaryCertRotated",
 "reportingComponent": "canary_controller",
 "reportingInstance": "",
 "source": {
   "component": "canary_controller"
 },
 "type": "Normal"
}
{
 "apiVersion": "v1",
 "count": 1,
 "eventTime": null,
 "firstTimestamp": "2025-09-30T17:39:37Z",
 "involvedObject": {
   "apiVersion": "apps/v1",
   "kind": "DaemonSet",
   "name": "ingress-canary",
   "namespace": "openshift-ingress-canary",
   "resourceVersion": "39516",
   "uid": "9d849944-b277-4943-980a-25e4c3f03514"
 },
 "kind": "Event",
 "lastTimestamp": "2025-09-30T17:39:37Z",
 "message": "Canary serving cert rotated, updated pod template annotation hash: ad7b4a2e",
 "metadata": {
   "creationTimestamp": "2025-09-30T17:39:37Z",
   "name": "ingress-canary.186a202be5b6a42d",
   "namespace": "openshift-ingress-canary",
   "resourceVersion": "39518",
   "uid": "44a7ebf7-a7f3-420e-a257-0a18c4f76866"
 },
 "reason": "CanaryCertRotated",
 "reportingComponent": "canary_controller",
 "reportingInstance": "",
 "source": {
   "component": "canary_controller"
 },
 "type": "Normal"
}
{
 "apiVersion": "v1",
 "count": 1,
 "eventTime": null,
 "firstTimestamp": "2025-09-30T17:39:37Z",
 "involvedObject": {
   "apiVersion": "apps/v1",
   "kind": "DaemonSet",
   "name": "ingress-canary",
   "namespace": "openshift-ingress-canary",
   "resourceVersion": "39539",
   "uid": "9d849944-b277-4943-980a-25e4c3f03514"
 },
 "kind": "Event",
 "lastTimestamp": "2025-09-30T17:39:37Z",
 "message": "Canary serving cert rotated, updated pod template annotation hash: 7fc52b64",
 "metadata": {
   "creationTimestamp": "2025-09-30T17:39:37Z",
   "name": "ingress-canary.186a202c0b391244",
   "namespace": "openshift-ingress-canary",
   "resourceVersion": "39540",
   "uid": "99568d26-b3d7-4397-ba34-0ed52e87a96d"
 },
 "reason": "CanaryCertRotated",
 "reportingComponent": "canary_controller",
 "reportingInstance": "",
 "source": {
   "component": "canary_controller"
 },
 "type": "Normal"
}
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % 
mjoseph@mjoseph-mac Downloads % oc -n $NS get events | grep CanaryCertRotated
49m         Normal    CanaryCertRotated   daemonset/ingress-canary   Canary serving cert rotated, updated pod template annotation hash: 08783236
6m25s       Normal    CanaryCertRotated   daemonset/ingress-canary   Canary serving cert rotated, updated pod template annotation hash: ad7b4a2e
6m25s       Normal    CanaryCertRotated   daemonset/ingress-canary   Canary serving cert rotated, updated pod template annotation hash: 7fc52b64

mjoseph@mjoseph-mac Downloads % oc get co
NAME                                       VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      36m     
baremetal                                  4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
cloud-controller-manager                   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      62m     
cloud-credential                           4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      64m     
cluster-autoscaler                         4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
config-operator                            4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      60m     
console                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      43m     
control-plane-machine-set                  4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      55m     
csi-snapshot-controller                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
dns                                        4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      58m     
etcd                                       4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      58m     
image-registry                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      49m     
ingress                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      48m     
insights                                   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
kube-apiserver                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      54m     
kube-controller-manager                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      54m     
kube-scheduler                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      56m     
kube-storage-version-migrator              4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      60m     
machine-api                                4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      48m     
machine-approver                           4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
machine-config                             4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      57m     
marketplace                                4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
monitoring                                 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      46m     
network                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      61m     
node-tuning                                4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      11m     
olm                                        4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
openshift-apiserver                        4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      49m     
openshift-controller-manager               4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      49m     
openshift-samples                          4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      48m     
operator-lifecycle-manager                 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
operator-lifecycle-manager-catalog         4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      59m     
operator-lifecycle-manager-packageserver   4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      54m     
service-ca                                 4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      60m     
storage                                    4.20.0-0-2025-09-30-162337-test-ci-ln-hg0gydt-latest   True        False         False      58m

Hence marking as verified
/label qe-approved
/verified by @mjoseph

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@melvinjoseph86
Copy link

melvinjoseph86 commented Sep 30, 2025

/verified by @melvinjoseph86

@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Sep 30, 2025

@melvinjoseph86: This PR has been marked as verified by @melvinjoseph86.

In response to this:

/verified by @melvinjoseph86

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bentito
Copy link
Contributor Author

bentito commented Oct 1, 2025

/retest

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 1, 2025

@bentito: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-aws-ovn-serial 2f8f5ac link true /test e2e-aws-ovn-serial

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Miciah
Copy link
Contributor

Miciah commented Oct 8, 2025

/cc @rfredette

Brett's PR touches the same code as your PR, #1155.

@openshift-ci openshift-ci bot requested a review from rfredette October 8, 2025 14:51
@Miciah
Copy link
Contributor

Miciah commented Oct 8, 2025

/assign

@candita
Copy link
Contributor

candita commented Oct 8, 2025

/assign @rfredette

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@melvinjoseph86 melvinjoseph86 Awaiting requested review from melvinjoseph86

@gcs278 gcs278 Awaiting requested review from gcs278

@Thealisyed Thealisyed Awaiting requested review from Thealisyed

@rfredette rfredette Awaiting requested review from rfredette

Assignees

@rfredette rfredette

@Miciah Miciah

Labels

jira/severity-moderate Referenced Jira bug's severity is moderate for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. qe-approved Signifies that QE has signed off on this PR verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@bentito @openshift-ci-robot @melvinjoseph86 @Miciah @candita @rfredette