[FLAKE e2e (2)] Operator Group [It] insufficient permissions resolve via RBAC

[tmshort]

Flaky Test Report

Also; too many similar/identical logs.

Failure Log Link
Failure Log

Relevant Failure Log

------------------------------
• [FAILED] [302.403 seconds]
Operator Group [It] insufficient permissions resolve via RBAC
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/operator_groups_e2e_test.go:1612

  Timeline >>
  created the operator-group-e2e-m679s testing namespace
  14:59:21.6505: Creating CRD
  14:59:21.6708: Creating operator group
  14:59:21.7011: Creating CSV
  14:59:21.8668: wait for CSV to fail
  14:59:21.9798:  (): 
  waited 9.38511127s for CSV operator-group-e2e-m679s-zr5ms/another-csv-jwr88: to be in phases [Failed], in phase  (): 
  14:59:22.0721:  (): 
  14:59:22.1724:  (): 
  14:59:22.282:  (): 
  14:59:22.3772:  (): 
  14:59:22.473:  (): 
  14:59:22.5726:  (): 
  14:59:22.7073:  (): 
  14:59:22.9107:  (): 
  14:59:23.1086: InstallReady (AllRequirementsMet): all requirements found, attempting install
  waited 1.128787182s for CSV operator-group-e2e-m679s-zr5ms/another-csv-jwr88: to be in phases [Failed], in phase InstallReady (AllRequirementsMet): all requirements found, attempting install
  14:59:23.3095: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  waited 200.854629ms for CSV operator-group-e2e-m679s-zr5ms/another-csv-jwr88: to be in phases [Failed], in phase Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  14:59:23.3211: wait for CSV to succeeed
  14:59:23.5102: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  waited 5.402819873s for CSV operator-group-e2e-m679s-zr5ms/another-csv-jwr88: to be in phases [Succeeded], in phase Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  14:59:23.7102: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  14:59:23.9075: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  14:59:24.1069: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"

... (lots of repeated lines)

  15:04:23.3076: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  15:04:23.5087: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  15:04:23.708: Failed (InstallComponentFailedNoRetry): install strategy failed: install strategy failed: deployments.apps "operator-deploymentzkkxr" is forbidden: User "system:serviceaccount:operator-group-e2e-m679s-zr5ms:nginx-sansp5b" cannot get resource "deployments" in API group "apps" in the namespace "operator-group-e2e-m679s-zr5ms"
  [FAILED] in [It] - /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/operator_groups_e2e_test.go:1746 @ 11/06/23 15:04:23.858
  cleaning up ephemeral test resources...
  deleting test subscriptions...
  deleting test installplans...
  deleting test catalogsources...
  deleting test crds...
  deleting test csvs...
  test resources deleted
  << Timeline

  [FAILED] 
  	Error Trace:	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/operator_groups_e2e_test.go:1746
  	            				/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/onsi/ginkgo/v2/internal/node.go:463
  	            				/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/onsi/ginkgo/v2/internal/suite.go:863
  	            				/opt/hostedtoolcache/go/1.20.10/x64/src/runtime/asm_amd64.s:1598
  	Error:      	Received unexpected error:
  	            	timed out waiting for the condition
  	Test:       	Operator Group insufficient permissions resolve via RBAC
  
  In [It] at: /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/operator_groups_e2e_test.go:1746 @ 11/06/23 15:04:23.858

  Full Stack Trace
    github.com/stretchr/testify/require.NoError({0x7fa3ac8bddb0, 0xc000cf5440}, {0x40901c0, 0xc0001ba160}, {0x0, 0x0, 0x0})
    	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/github.com/stretchr/testify/require/require.go:1357 +0x96
    github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func20.9()
    	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/operator_groups_e2e_test.go:1746 +0x15b5
------------------------------

[tmshort]

Investigating, not sure this is a testing issue, the test is fairly simple and seems OK.

In the success case, the CSV gets updated due to a change in the service account (operatorgroup.go):

Requeuing CSV due to detected service account change

not due to a change in RBAC (operator.go). The failure case (about 25% of the time, or so), does not contain the above log message.

[tmshort]

The occurrence rate is about 25% or less.
EDIT: Success 10/10 times, so less than 10% occurrence.

[tmshort]

Closing after a year. Our flakes have much improved.