[Flake e2e (0)] Fail Forward Upgrades when a CSV resource is in a failed state [BeforeEach] eventually reports a successful state when using skips #3126
Description
Flaky Test Report
Failure Log Link
Failure Log
You can get the link by clicking on the line number in the job logs and copying the url from your browser.
Relevant Failure Log
Fail Forward Upgrades when a CSV resource is in a failed state [BeforeEach] eventually reports a successful state when using skips
[BeforeEach] /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:332
[It] /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:403
Captured StdOut/StdErr Output >>
waiting 102.3187ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - NO_CONNECTION
waiting 2.161734782s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - IDLE
waiting 399.999906ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
waiting 19.399318051s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
waiting 999.950818ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
waiting 20.000381764s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
waiting 1.799416334s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - READY
probing catalog mc-csv-failed-vpd7n pod with address mc-csv-failed-vpd7n.fail-forward-e2e-bhsmz.svc:50051
skipping health check
waiting 102.482175ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - NO_CONNECTION
waiting 3.08146498s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - IDLE
waiting 1.000002758s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
waiting 18.002226617s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
waiting 998.20021ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
waiting 20.00012404s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
waiting 1.800247429s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - READY
probing catalog mc-csv-failed-vpd7n pod with address mc-csv-failed-vpd7n.fail-forward-e2e-bhsmz.svc:50051
skipping health check
Using the kubectl kubectl binary
Using the artifacts/fail-forward-e2e-bhsmz output directory
Storing the test artifact output in the artifacts/fail-forward-e2e-bhsmz directory
Collecting get catalogsources -o yaml output...
Collecting get subscriptions -o yaml output...
Collecting get operatorgroups -o yaml output...
Collecting get clusterserviceversions -o yaml output...
Collecting get installplans -o yaml output...
Collecting get pods -o wide output...
Collecting get events --sort-by .lastTimestamp output...
<< Captured StdOut/StdErr Output
Timeline >>
STEP: creating the testing namespace with an OG that enabled fail forward behavior @ 11/30/23 15:10:54.178
created the fail-forward-e2e-bhsmz testing namespace
STEP: deploying the testing catalog @ 11/30/23 15:10:54.193
Creating catalog source mc-csv-failed-vpd7n in namespace fail-forward-e2e-bhsmz...
Catalog source mc-csv-failed-vpd7n created
STEP: creating the testing subscription @ 11/30/23 15:11:39.065
STEP: waiting until the subscription has an IP reference @ 11/30/23 15:11:39.069
15:11:41.8669: subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub state: UpgradePending (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-bhsmz", Name:"install-7b6j8", UID:"0f9b478e-f55c-4354-b374-b440b4b035ce", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"6838", FieldPath:""}
waiting 2.797254321s for subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-bhsmz", Name:"install-7b6j8", UID:"0f9b478e-f55c-4354-b374-b440b4b035ce", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"6838", FieldPath:""}
STEP: waiting for the v0.1.0 CSV to report a succeeded phase @ 11/30/23 15:11:41.867
waiting for CSV fail-forward-e2e-bhsmz/example-operator.v0.1.0 to reach condition
waited 199.903672ms for csv fail-forward-e2e-bhsmz/example-operator.v0.1.0 - Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
waited 199.91354ms for CSV fail-forward-e2e-bhsmz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
waited 398.714801ms for csv fail-forward-e2e-bhsmz/example-operator.v0.1.0 - Succeeded (InstallSucceeded): install strategy completed with no errors
waited 398.716824ms for CSV fail-forward-e2e-bhsmz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Succeeded (InstallSucceeded): install strategy completed with no errors
STEP: updating the catalog with a broken v0.2.0 csv @ 11/30/23 15:11:42.465
STEP: removing the existing catalog source @ 11/30/23 15:11:42.465
STEP: removing the previous catalog source pod(s) @ 11/30/23 15:11:42.667
waiting for the catalog source mc-csv-failed-vpd7n-s8rpm pod to be deleted...
STEP: updating the catalog with a new bundle images @ 11/30/23 15:11:43.673
Creating catalog source mc-csv-failed-vpd7n in namespace fail-forward-e2e-bhsmz...
Catalog source mc-csv-failed-vpd7n created
STEP: verifying the subscription has installed the current csv @ 11/30/23 15:12:28.666
15:17:29.0655: subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"", InstalledCSV:"", Install:(*v1alpha1.InstallPlanReference)(nil), State:"", Reason:"", InstallPlanGeneration:0, InstallPlanRef:(*v1.ObjectReference)(nil), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0007129a0), LastUpdated:time.Date(2023, time.November, 30, 15, 12, 40, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc000712a10), LastUpdated:time.Date(2023, time.November, 30, 15, 12, 40, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 30, 15, 12, 40, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: @existing/fail-forward-e2e-bhsmz//example-operator.v0.1.0, mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.2.0 and mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.1.0 originate from package packageA, clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, subscription mc-csv-failed-vpd7n-sub requires at least one of mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.2.0 or mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.1.0, subscription mc-csv-failed-vpd7n-sub exists", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 30, 15, 15, 56, 0, time.Local)}
15:17:29.0655: subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub spec: &v1alpha1.SubscriptionSpec{CatalogSource:"mc-csv-failed-vpd7n", CatalogSourceNamespace:"fail-forward-e2e-bhsmz", Package:"packageA", Channel:"stable", StartingCSV:"", InstallPlanApproval:"", Config:(*v1alpha1.SubscriptionConfig)(nil)}
[FAILED] in [BeforeEach] - /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:374 @ 11/30/23 15:17:29.065
STEP: removing the testing catalog resources @ 11/30/23 15:17:29.065
Deleting config map mc-csv-failed-vpd7n-configmap...
Deleting catalog source mc-csv-failed-vpd7n...
waiting for the catalog source mc-csv-failed-vpd7n-tw2dk pod to be deleted...
Deleting config map mc-csv-failed-vpd7n-configmap...
Deleting catalog source mc-csv-failed-vpd7n...
STEP: deleting the testing namespace @ 11/30/23 15:17:30.28
collecting the fail-forward-e2e-bhsmz namespace artifacts as the 'eventually reports a successful state when using skips' test case failed
collecting logs in the ./artifacts/ artifacts directory
tearing down the fail-forward-e2e-bhsmz namespace
resetting e2e kube client
deleting fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub
deleting fail-forward-e2e-bhsmz/fail-forward-e2e-bhsmz-operatorgroup
deleting <global>/fail-forward-e2e-bhsmz
garbage collecting CRDs
<< Timeline
[FAILED] Expected
<wait.errInterrupted>:
timed out waiting for the condition
{
cause: <*errors.errorString | 0xc000138140>{
s: "timed out waiting for the condition",
},
}
to be nil
In [BeforeEach] at: /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:374 @ 11/30/23 15:17:29.065
Full Stack Trace
github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func14.4.1()
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:374 +0x827