[Flake e2e (0)] Fail Forward Upgrades when a CSV resource is in a failed state [It] eventually reports a successful state when using skip ranges

[tmshort]

Flaky Test Report

Failure Log Link
Failure Log

Relevant Failure Log

------------------------------
• [FAILED] [398.692 seconds]
Fail Forward Upgrades when a CSV resource is in a failed state [It] eventually reports a successful state when using skip ranges
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:389

  Captured StdOut/StdErr Output >>
  waiting 103.714261ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - NO_CONNECTION
  waiting 1.705552809s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - IDLE
  waiting 203.34317ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - TRANSIENT_FAILURE
  waiting 594.885807ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - CONNECTING
  waiting 200.681952ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - TRANSIENT_FAILURE
  waiting 12.99946988s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - READY
  probing catalog mc-csv-failed-7h6c9 pod with address mc-csv-failed-7h6c9.fail-forward-e2e-w6cdz.svc:50051
  skipping health check
  waiting 103.496755ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - NO_CONNECTION
  waiting 2.26095134s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - IDLE
  waiting 1.188746873s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - CONNECTING
  waiting 18.594727433s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - TRANSIENT_FAILURE
  waiting 999.441847ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - READY
  probing catalog mc-csv-failed-7h6c9 pod with address mc-csv-failed-7h6c9.fail-forward-e2e-w6cdz.svc:50051
  skipping health check
  waiting 106.62133ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - NO_CONNECTION
  waiting 3.604420787s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - IDLE
  waiting 810.787389ms for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - CONNECTING
  waiting 18.387474948s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - TRANSIENT_FAILURE
  waiting 1.000335935s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - CONNECTING
  waiting 19.999753332s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - TRANSIENT_FAILURE
  waiting 1.646247834s for catalog pod fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9 to be available (for sync) - READY
  probing catalog mc-csv-failed-7h6c9 pod with address mc-csv-failed-7h6c9.fail-forward-e2e-w6cdz.svc:50051
  skipping health check
  Using the kubectl kubectl binary
  Using the artifacts/fail-forward-e2e-w6cdz output directory
  Storing the test artifact output in the artifacts/fail-forward-e2e-w6cdz directory
  Collecting get catalogsources -o yaml output...
  Collecting get subscriptions -o yaml output...
  Collecting get operatorgroups -o yaml output...
  Collecting get clusterserviceversions -o yaml output...
  Collecting get installplans -o yaml output...
  Collecting get pods -o wide output...
  Collecting get events --sort-by .lastTimestamp output...
  << Captured StdOut/StdErr Output

  Timeline >>
  STEP: creating the testing namespace with an OG that enabled fail forward behavior @ 11/06/23 13:55:52.872
  created the fail-forward-e2e-w6cdz testing namespace
  STEP: deploying the testing catalog @ 11/06/23 13:55:52.949
  Creating catalog source mc-csv-failed-7h6c9 in namespace fail-forward-e2e-w6cdz...
  Catalog source mc-csv-failed-7h6c9 created
  STEP: creating the testing subscription @ 11/06/23 13:56:08.797
  STEP: waiting until the subscription has an IP reference @ 11/06/23 13:56:08.805
  13:56:13.0237: subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub state: UpgradePending (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-w6cdz", Name:"install-gcpf4", UID:"e0a9e18b-3acd-4551-8b2c-a2fd84f44f8e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"2373", FieldPath:""}
  waiting 4.21[780](https://github.com/operator-framework/operator-lifecycle-manager/actions/runs/6771572912/job/18402234628#step:5:781)5022s for subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-w6cdz", Name:"install-gcpf4", UID:"e0a9e18b-3acd-4551-8b2c-a2fd84f44f8e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"2373", FieldPath:""}
  STEP: waiting for the v0.1.0 CSV to report a succeeded phase @ 11/06/23 13:56:13.023
  waiting for CSV fail-forward-e2e-w6cdz/example-operator.v0.1.0 to reach condition
  waited 196.299543ms for csv fail-forward-e2e-w6cdz/example-operator.v0.1.0 - Installing (InstallSucceeded): waiting for install components to report healthy
  waited 196.310643ms for CSV fail-forward-e2e-w6cdz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallSucceeded): waiting for install components to report healthy
  waited 189.081397ms for csv fail-forward-e2e-w6cdz/example-operator.v0.1.0 - Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: waiting for spec update of deployment "example-operator-controller-manager" to be observed...
  waited 189.085397ms for CSV fail-forward-e2e-w6cdz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: waiting for spec update of deployment "example-operator-controller-manager" to be observed...
  waited 189.807402ms for csv fail-forward-e2e-w6cdz/example-operator.v0.1.0 - Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
  waited 189.812302ms for CSV fail-forward-e2e-w6cdz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
  waited 1.598454321s for csv fail-forward-e2e-w6cdz/example-operator.v0.1.0 - Succeeded (InstallSucceeded): install strategy completed with no errors
  waited 1.598455722s for CSV fail-forward-e2e-w6cdz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Succeeded (InstallSucceeded): install strategy completed with no errors
  STEP: updating the catalog with a broken v0.2.0 csv @ 11/06/23 13:56:15.197
  STEP: removing the existing catalog source @ 11/06/23 13:56:15.197
  STEP: removing the previous catalog source pod(s) @ 11/06/23 13:56:15.399
  waiting for the catalog source mc-csv-failed-7h6c9-z2w6v pod to be deleted...
  STEP: updating the catalog with a new bundle images @ 11/06/23 13:56:16.421
  Creating catalog source mc-csv-failed-7h6c9 in namespace fail-forward-e2e-w6cdz...
  Catalog source mc-csv-failed-7h6c9 created
  STEP: verifying the subscription has installed the current csv @ 11/06/23 13:56:39.597
  13:56:39.7981: subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub state: AtLatestKnown (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-w6cdz", Name:"install-gcpf4", UID:"e0a9e18b-3acd-4551-8b2c-a2fd84f44f8e", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"2373", FieldPath:""}
  13:56:40.7982: subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub state: UpgradePending (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-w6cdz", Name:"install-jx7xj", UID:"ae2c98d5-214a-4697-8fc0-2c3be4758933", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"2590", FieldPath:""}
  STEP: waiting for the bad CSV to report a failed state @ 11/06/23 13:56:40.798
  waiting for CSV fail-forward-e2e-w6cdz/example-operator.v0.2.0 to reach condition
  waited 208.0[791](https://github.com/operator-framework/operator-lifecycle-manager/actions/runs/6771572912/job/18402234628#step:5:792)72ms for csv fail-forward-e2e-w6cdz/example-operator.v0.2.0 - Failed (InstallComponentFailed): install strategy failed: Deployment.apps "example-operator-controller-manager&invalid" is invalid: metadata.name: Invalid value: "example-operator-controller-manager&invalid": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
  waited 8m43.288344407s for CSV fail-forward-e2e-w6cdz/example-operator.v0.2.0: to be in phases [Failed], in phase Failed (InstallComponentFailed): install strategy failed: Deployment.apps "example-operator-controller-manager&invalid" is invalid: metadata.name: Invalid value: "example-operator-controller-manager&invalid": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
  STEP: patching the catalog with a fixed version @ 11/06/23 13:56:41.006
  STEP: removing the existing catalog source @ 11/06/23 13:56:41.006
  STEP: removing the previous catalog source pod(s) @ 11/06/23 13:56:41.21
  waiting for the catalog source mc-csv-failed-7h6c9-rdgd7 pod to be deleted...
  STEP: updating the catalog with a new bundle images @ 11/06/23 13:56:42.271
  Creating catalog source mc-csv-failed-7h6c9 in namespace fail-forward-e2e-w6cdz...
  Catalog source mc-csv-failed-7h6c9 created
  STEP: waiting for the subscription to have the example-operator.v0.3.0 status.currentCSV @ 11/06/23 13:57:27.845
  13:57:28.046: subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub state: AtLatestKnown (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-w6cdz", Name:"install-jx7xj", UID:"ae2c98d5-214a-4697-8fc0-2c3be4758933", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"2590", FieldPath:""}
  14:02:28.1992: subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"example-operator.v0.2.0", InstalledCSV:"example-operator.v0.2.0", Install:(*v1alpha1.InstallPlanReference)(0xc000033700), State:"AtLatestKnown", Reason:"", InstallPlanGeneration:2, InstallPlanRef:(*v1.ObjectReference)(0xc000913730), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc000913650), LastUpdated:time.Date(2023, time.November, 6, 13, 56, 44, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0009136c0), LastUpdated:time.Date(2023, time.November, 6, 13, 56, 44, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 6, 13, 56, 44, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"BundleUnpacking", Status:"False", Reason:"", Message:"", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, @existing/fail-forward-e2e-w6cdz//example-operator.v0.2.0, @existing/fail-forward-e2e-w6cdz//example-operator.v0.1.0 and mc-csv-failed-7h6c9/fail-forward-e2e-w6cdz/stable/example-operator.v0.3.0 originate from package packageA, subscription mc-csv-failed-7h6c9-sub exists, subscription mc-csv-failed-7h6c9-sub requires at least one of mc-csv-failed-7h6c9/fail-forward-e2e-w6cdz/stable/example-operator.v0.3.0 or @existing/fail-forward-e2e-w6cdz//example-operator.v0.2.0", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 6, 13, 57, 27, 0, time.Local)}
  14:02:28.1993: subscription fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub spec: &v1alpha1.SubscriptionSpec{CatalogSource:"mc-csv-failed-7h6c9", CatalogSourceNamespace:"fail-forward-e2e-w6cdz", Package:"packageA", Channel:"stable", StartingCSV:"", InstallPlanApproval:"", Config:(*v1alpha1.SubscriptionConfig)(nil)}
  [FAILED] in [It] - /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:399 @ 11/06/23 14:02:28.204
  STEP: removing the testing catalog resources @ 11/06/23 14:02:28.204
  Deleting config map mc-csv-failed-7h6c9-configmap...
  Deleting catalog source mc-csv-failed-7h6c9...
  waiting for the catalog source mc-csv-failed-7h6c9-7mnhz pod to be deleted...
  Deleting config map mc-csv-failed-7h6c9-configmap...
  Deleting catalog source mc-csv-failed-7h6c9...
  Deleting config map mc-csv-failed-7h6c9-configmap...
  Deleting catalog source mc-csv-failed-7h6c9...
  STEP: deleting the testing namespace @ 11/06/23 14:02:29.488
  collecting the fail-forward-e2e-w6cdz namespace artifacts as the 'eventually reports a successful state when using skip ranges' test case failed
  collecting logs in the ./artifacts/ artifacts directory
  tearing down the fail-forward-e2e-w6cdz namespace
  resetting e2e kube client
  deleting fail-forward-e2e-w6cdz/mc-csv-failed-7h6c9-sub
  deleting fail-forward-e2e-w6cdz/fail-forward-e2e-w6cdz-operatorgroup
  deleting <global>/fail-forward-e2e-w6cdz
  garbage collecting CRDs
  << Timeline

  [FAILED] Expected
      <wait.errInterrupted>: 
      timed out waiting for the condition
      {
          cause: <*errors.errorString | 0xc0001ae110>{
              s: "timed out waiting for the condition",
          },
      }
  to be nil
  In [It] at: /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:399 @ 11/06/23 14:02:28.204

  Full Stack Trace
    github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func14.4.3()
    	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:399 +0x2a3
------------------------------

[tmshort]

Took 8 attempts to fail; 12.5% occurrance.

[tmshort]

This is likely due to an error updating the subscription, possibly at a critical time. But, I do see similar errors during a success.
This is from the catalog-operator pod:

2023-11-07T19:43:34.379154025Z stderr F time="2023-11-07T19:43:34Z" level=debug msg="resolving subscriptions in namespace" id=TjnDL namespace=fail-forward-e2e-t25w6
2023-11-07T19:43:34.387130717Z stderr F time="2023-11-07T19:43:34Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"mc-csv-failed-qn8mm-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
2023-11-07T19:43:34.388610715Z stderr F E1107 19:43:34.388510       1 queueinformer_operator.go:319] sync {"update" "fail-forward-e2e-t25w6/mc-csv-failed-qn8mm-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "mc-csv-failed-qn8mm-sub": the object has been modified; please apply your changes to the latest version and try again
2023-11-07T19:43:34.388813015Z stderr F time="2023-11-07T19:43:34Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=

From the e2e log:

  19:44:38.6016: subscription fail-forward-e2e-t25w6/mc-csv-failed-qn8mm-sub state: AtLatestKnown (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-t25w6", Name:"install-kktrz", UID:"0666549d-ad6f-4d65-abd7-2ab374236509", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"1122", FieldPath:""}
  19:49:38.5972: subscription fail-forward-e2e-t25w6/mc-csv-failed-qn8mm-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"example-operator.v0.2.0", InstalledCSV:"example-operator.v0.2.0", Install:(*v1alpha1.InstallPlanReference)(0xc000fc3f40), State:"AtLatestKnown", Reason:"", InstallPlanGeneration:2, InstallPlanRef:(*v1.ObjectReference)(0xc0005fc690), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0005fc5b0), LastUpdated:time.Date(2023, time.November, 7, 19, 44, 17, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0005fc620), LastUpdated:time.Date(2023, time.November, 7, 19, 44, 17, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 7, 19, 44, 17, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"BundleUnpacking", Status:"False", Reason:"", Message:"", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, @existing/fail-forward-e2e-t25w6//example-operator.v0.2.0, @existing/fail-forward-e2e-t25w6//example-operator.v0.1.0 and mc-csv-failed-qn8mm/fail-forward-e2e-t25w6/stable/example-operator.v0.3.0 originate from package packageA, subscription mc-csv-failed-qn8mm-sub exists, subscription mc-csv-failed-qn8mm-sub requires at least one of mc-csv-failed-qn8mm/fail-forward-e2e-t25w6/stable/example-operator.v0.3.0 or @existing/fail-forward-e2e-t25w6//example-operator.v0.2.0", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 7, 19, 44, 38, 0, time.Local)}

[stevekuznetsov]

 Fail Forward Upgrades when a CSV resource is in a failed state eventually reports a successful state when using skips
/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:402
  STEP: creating the testing namespace with an OG that enabled fail forward behavior @ 11/14/23 02:22:51.908
  created the fail-forward-e2e-hm8m2 testing namespace
  STEP: deploying the testing catalog @ 11/14/23 02:22:51.999
  Creating catalog source mc-csv-failed-b47k7 in namespace fail-forward-e2e-hm8m2...
  Catalog source mc-csv-failed-b47k7 created
waiting 148.179373ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - NO_CONNECTION
waiting 590.956103ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - IDLE
waiting 200.019269ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 602.234668ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 197.393456ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 15.656604807s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - READY
probing catalog mc-csv-failed-b47k7 pod with address mc-csv-failed-b47k7.fail-forward-e2e-hm8m2.svc:50051
skipping health check
  STEP: creating the testing subscription @ 11/14/23 02:23:09.484
  STEP: waiting until the subscription has an IP reference @ 11/14/23 02:23:09.526
  02:23:10.8861: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: UpgradePending (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-sl4wl", UID:"1df970d6-3803-406d-8dc4-214c36af1b22", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128538", FieldPath:""}
  waiting 1.359020737s for subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-sl4wl", UID:"1df970d6-3803-406d-8dc4-214c36af1b22", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128538", FieldPath:""}
  STEP: waiting for the v0.1.0 CSV to report a succeeded phase @ 11/14/23 02:23:10.886
  waiting for CSV fail-forward-e2e-hm8m2/example-operator.v0.1.0 to reach condition
  waited 199.608397ms for csv fail-forward-e2e-hm8m2/example-operator.v0.1.0 - Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
  waited 199.697425ms for CSV fail-forward-e2e-hm8m2/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
  waited 1.601659075s for csv fail-forward-e2e-hm8m2/example-operator.v0.1.0 - Succeeded (InstallSucceeded): install strategy completed with no errors
  waited 1.601733023s for CSV fail-forward-e2e-hm8m2/example-operator.v0.1.0: to be in phases [Succeeded], in phase Succeeded (InstallSucceeded): install strategy completed with no errors
  STEP: updating the catalog with a broken v0.2.0 csv @ 11/14/23 02:23:12.688
  STEP: removing the existing catalog source @ 11/14/23 02:23:12.688
  STEP: removing the previous catalog source pod(s) @ 11/14/23 02:23:12.892
  waiting for the catalog source mc-csv-failed-b47k7-8ckq6 pod to be deleted...
  STEP: updating the catalog with a new bundle images @ 11/14/23 02:23:13.976
  Creating catalog source mc-csv-failed-b47k7 in namespace fail-forward-e2e-hm8m2...
  Catalog source mc-csv-failed-b47k7 created
waiting 138.410284ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - NO_CONNECTION
waiting 2.26910633s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 2.598714655s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 200.99772ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 12.400568791s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 199.285203ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - READY
probing catalog mc-csv-failed-b47k7 pod with address mc-csv-failed-b47k7.fail-forward-e2e-hm8m2.svc:50051
skipping health check
  STEP: verifying the subscription has installed the current csv @ 11/14/23 02:23:31.885
  02:23:32.0872: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: AtLatestKnown (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-sl4wl", UID:"1df970d6-3803-406d-8dc4-214c36af1b22", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128538", FieldPath:""}
  02:23:32.2868: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: UpgradePending (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-m4jh9", UID:"4d3d9467-23cd-4055-884c-34ffc0c08b28", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128987", FieldPath:""}
  STEP: waiting for the bad CSV to report a failed state @ 11/14/23 02:23:32.286
  waiting for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0 to reach condition
  waited 207.233924ms for csv fail-forward-e2e-hm8m2/example-operator.v0.2.0 - Pending (RequirementsUnknown): requirements not yet checked
  waited 1m12.23119933s for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0: to be in phases [Failed], in phase Pending (RequirementsUnknown): requirements not yet checked
  waited 593.392252ms for csv fail-forward-e2e-hm8m2/example-operator.v0.2.0 - InstallReady (AllRequirementsMet): all requirements found, attempting install
  waited 593.428936ms for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0: to be in phases [Failed], in phase InstallReady (AllRequirementsMet): all requirements found, attempting install
  waited 398.113032ms for csv fail-forward-e2e-hm8m2/example-operator.v0.2.0 - Failed (InstallComponentFailed): install strategy failed: Deployment.apps "example-operator-controller-manager&invalid" is invalid: metadata.name: Invalid value: "example-operator-controller-manager&invalid": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
  waited 398.154075ms for CSV fail-forward-e2e-hm8m2/example-operator.v0.2.0: to be in phases [Failed], in phase Failed (InstallComponentFailed): install strategy failed: Deployment.apps "example-operator-controller-manager&invalid" is invalid: metadata.name: Invalid value: "example-operator-controller-manager&invalid": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
  STEP: patching the catalog with a fixed version @ 11/14/23 02:23:33.486
  STEP: removing the existing catalog source @ 11/14/23 02:23:33.486
  STEP: removing the previous catalog source pod(s) @ 11/14/23 02:23:33.698
  waiting for the catalog source mc-csv-failed-b47k7-297mr pod to be deleted...
  STEP: updating the catalog with a new bundle images @ 11/14/23 02:23:34.793
  Creating catalog source mc-csv-failed-b47k7 in namespace fail-forward-e2e-hm8m2...
  Catalog source mc-csv-failed-b47k7 created
waiting 138.763895ms for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - NO_CONNECTION
waiting 4.050054318s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - CONNECTING
waiting 19.996456256s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - TRANSIENT_FAILURE
waiting 1.000994366s for catalog pod fail-forward-e2e-hm8m2/mc-csv-failed-b47k7 to be available (for sync) - READY
probing catalog mc-csv-failed-b47k7 pod with address mc-csv-failed-b47k7.fail-forward-e2e-hm8m2.svc:50051
skipping health check
  STEP: waiting for the subscription to have the example-operator.v0.3.0 status.currentCSV @ 11/14/23 02:24:00.085
  02:24:00.2865: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub state: AtLatestKnown (csv example-operator.v0.2.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-hm8m2", Name:"install-m4jh9", UID:"4d3d9467-23cd-4055-884c-34ffc0c08b28", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"128987", FieldPath:""}
  02:29:00.4869: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"example-operator.v0.2.0", InstalledCSV:"example-operator.v0.2.0", Install:(*v1alpha1.InstallPlanReference)(0xc001b13e40), State:"AtLatestKnown", Reason:"", InstallPlanGeneration:2, InstallPlanRef:(*v1.ObjectReference)(0xc00087ec40), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087e930), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087e9a0), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087ea80), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087eaf0), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc00087eb60), LastUpdated:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 14, 2, 23, 41, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"BundleUnpacking", Status:"False", Reason:"", Message:"", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: @existing/fail-forward-e2e-hm8m2//example-operator.v0.2.0, @existing/fail-forward-e2e-hm8m2//example-operator.v0.1.0 and mc-csv-failed-b47k7/fail-forward-e2e-hm8m2/stable/example-operator.v0.3.0 originate from package packageA, clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, subscription mc-csv-failed-b47k7-sub exists, subscription mc-csv-failed-b47k7-sub requires at least one of mc-csv-failed-b47k7/fail-forward-e2e-hm8m2/stable/example-operator.v0.3.0 or @existing/fail-forward-e2e-hm8m2//example-operator.v0.2.0", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 14, 2, 28, 34, 0, time.Local)}
  02:29:00.4871: subscription fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub spec: &v1alpha1.SubscriptionSpec{CatalogSource:"mc-csv-failed-b47k7", CatalogSourceNamespace:"fail-forward-e2e-hm8m2", Package:"packageA", Channel:"stable", StartingCSV:"", InstallPlanApproval:"", Config:(*v1alpha1.SubscriptionConfig)(nil)}
  [FAILED] in [It] - /go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:412 @ 11/14/23 02:29:00.488
  STEP: removing the testing catalog resources @ 11/14/23 02:29:00.489
  Deleting config map mc-csv-failed-mkq2q-configmap...
  Deleting catalog source mc-csv-failed-mkq2q...
  Deleting config map mc-csv-failed-mkq2q-configmap...
  Deleting catalog source mc-csv-failed-mkq2q...
  Deleting config map mc-csv-failed-mkq2q-configmap...
  Deleting catalog source mc-csv-failed-mkq2q...
  Deleting config map mc-csv-failed-b47k7-configmap...
  Deleting catalog source mc-csv-failed-b47k7...
  waiting for the catalog source mc-csv-failed-b47k7-29zm8 pod to be deleted...
  Deleting config map mc-csv-failed-b47k7-configmap...
  Deleting catalog source mc-csv-failed-b47k7...
  Deleting config map mc-csv-failed-b47k7-configmap...
  Deleting catalog source mc-csv-failed-b47k7...
  STEP: deleting the testing namespace @ 11/14/23 02:29:02.588
  collecting the fail-forward-e2e-hm8m2 namespace artifacts as the 'eventually reports a successful state when using skips' test case failed
  collecting logs in the /logs/artifacts artifacts directory
  unable to determine kubeconfig path so defaulting to the $KUBECONFIG value
Using the oc kubectl binary
Using the /logs/artifacts/fail-forward-e2e-hm8m2 output directory
Storing the test artifact output in the /logs/artifacts/fail-forward-e2e-hm8m2 directory
Collecting get catalogsources -o yaml output...
Collecting get subscriptions -o yaml output...
Collecting get operatorgroups -o yaml output...
Collecting get clusterserviceversions -o yaml output...
Collecting get installplans -o yaml output...
Collecting get pods -o wide output...
Collecting get events --sort-by .lastTimestamp output...
  tearing down the fail-forward-e2e-hm8m2 namespace
  resetting e2e kube client
  deleting fail-forward-e2e-hm8m2/mc-csv-failed-b47k7-sub
  deleting fail-forward-e2e-hm8m2/fail-forward-e2e-hm8m2-operatorgroup
  deleting <global>/fail-forward-e2e-hm8m2
  garbage collecting CRDs
• [FAILED] [375.072 seconds]
Fail Forward Upgrades when a CSV resource is in a failed state [It] eventually reports a successful state when using skips
/go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:402
  [FAILED] Expected
      <wait.errInterrupted>: 
      timed out waiting for the condition
      {
          cause: <*errors.errorString | 0xc000250190>{
              s: "timed out waiting for the condition",
          },
      }
  to be nil
  In [It] at: /go/src/github.com/openshift/operator-framework-olm/staging/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:412 @ 11/14/23 02:29:00.488 

likely similar culprit in different test

[tmshort]

Closing after a year. Our flakes have much improved.