Exception during reconciliation loop of CSV installation #3634
Description
Bug Report
What did you do?
I tried to deploy an Operator in the bundle format with OLM by
$ ./operator-sdk run bundle quay.io/airlock/microgateway-bundle:4.7.0 --timeout 10m --verbose
DEBU[0000] Debug logging is set
DEBU[0000] resolving host=quay.io
...
omit some logs
...
INFO[0007] Creating a File-Based Catalog of the bundle "quay.io/airlock/microgateway-bundle:4.7.0"
INFO[0008] Generated a valid File-Based Catalog
INFO[0015] Created registry pod: quay-io-airlock-microgateway-bundle-4-7-0
INFO[0015] Created CatalogSource: airlock-microgateway-catalog
INFO[0015] OperatorGroup "operator-sdk-og" created
INFO[0015] Created Subscription: airlock-microgateway-v4-7-0-sub
INFO[0021] Approved InstallPlan install-g78mv for the Subscription: airlock-microgateway-v4-7-0-sub
INFO[0021] Waiting for ClusterServiceVersion "default/airlock-microgateway.v4.7.0" to reach 'Succeeded' phase
INFO[0022] Waiting for ClusterServiceVersion "default/airlock-microgateway.v4.7.0" to appear
INFO[0023] Found ClusterServiceVersion "default/airlock-microgateway.v4.7.0" phase: Pending
FATA[0600] Failed to run bundle: error waiting for CSV to install: deployment airlock-microgateway-operator has error: client rate limiter Wait returned an error: context deadline exceeded
What did you expect to see?
I would expect to see some useful information from the logs of the olm-operator
What did you see instead? Under which circumstances?
I did see a stacktrace during the reconciliation loop:
time="2025-08-28T11:33:24Z" level=info msg="log level info"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=serviceaccounts"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=configmaps"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="rbac.authorization.k8s.io/v1, Resource=roles"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="batch/v1, Resource=jobs"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="rbac.authorization.k8s.io/v1, Resource=rolebindings"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=services"
time="2025-08-28T11:33:24Z" level=info msg="skipping irrelevant gvr" gvr="/v1, Resource=pods"
time="2025-08-28T11:33:24Z" level=info msg="detected ability to filter informers" canFilter=true
time="2025-08-28T11:33:24Z" level=info msg="connection established. cluster-version: v1.31.4+k3s1"
time="2025-08-28T11:33:24Z" level=info msg="operator ready"
time="2025-08-28T11:33:24Z" level=info msg="starting informers..."
time="2025-08-28T11:33:24Z" level=info msg="informers started"
time="2025-08-28T11:33:24Z" level=info msg="waiting for caches to sync..."
time="2025-08-28T11:33:24Z" level=info msg="starting workers..."
{"level":"info","ts":"2025-08-28T11:33:24Z","msg":"Starting
...
omit some logs
...
time="2025-08-28T11:33:26Z" level=info msg="install strategy successful" csv=packageserver id=MIaOI namespace=olm phase=Installing strategy=deployment
I0828 11:33:26.756838 1 event.go:377] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"olm", Name:"packageserver", UID:"b97245d8-effb-4a9d-85a5-feff233bcc04", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"1123", FieldPath:""}): type: 'Normal' reason: 'InstallSucceeded' install strategy completed with no errors
time="2025-08-28T11:37:10Z" level=info msg="creating cluster role: olm.og.operator-sdk-og.admin-m10n7ysXEwwVPNpqphb9tOTsakP6LzLkR9gIP owned by operator group: default/operator-sdk-og"
time="2025-08-28T11:37:10Z" level=info msg="creating cluster role: olm.og.operator-sdk-og.edit-1IkXlP72QDB6hfV9RwiPxPgrTNF3GsTflLAyP5 owned by operator group: default/operator-sdk-og"
time="2025-08-28T11:37:10Z" level=info msg="creating cluster role: olm.og.operator-sdk-og.view-bTNXKYw3dBirwUlCNUySYd1jFTw3FK6RWCBEBE owned by operator group: default/operator-sdk-og"
{"level":"error","ts":"2025-08-28T11:37:18Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"airlock-microgateway.v4.7.0","namespace":"default"},"namespace":"default","name":"airlock-microgateway.v4.7.0","reconcileID":"c2e3dff2-2168-4145-8a69-99180d6bee71","error":"resource name may not be empty","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202"}
{"level":"error","ts":"2025-08-28T11:37:18Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"airlock-microgateway.v4.7.0","namespace":"default"},"namespace":"default","name":"airlock-microgateway.v4.7.0","reconcileID":"15e190be-4c15-4ea2-9f54-0e29cfe56c42","error":"resource name may not be empty","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202"}
{"level":"error","ts":"2025-08-28T11:37:18Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"airlock-microgateway.v4.7.0","namespace":"default"},"namespace":"default","name":"airlock-microgateway.v4.7.0","reconcileID":"9392ba0f-c9bb-4b48-80b0-d08a55a94167","error":"Deployment.apps \"airlock-microgateway-operator\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202"}
{"level":"error","ts":"2025-08-28T11:37:18Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"airlock-microgateway.v4.7.0","namespace":"default"},"namespace":"default","name":"airlock-microgateway.v4.7.0","reconcileID":"a233196f-67a6-458f-825d-c57b9fb6446f","error":"Deployment.apps \"airlock-microgateway-operator\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202"}
time="2025-08-28T11:37:18Z" level=warning msg="error adding operatorgroup annotations" csv=airlock-microgateway.v4.7.0 error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" id=bcxTR namespace=default opgroup=operator-sdk-og phase=
time="2025-08-28T11:37:18Z" level=info msg="operatorgroup incorrect" csv=airlock-microgateway.v4.7.0 error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" id=bcxTR namespace=default phase=
time="2025-08-28T11:37:18Z" level=info msg="not in operatorgroup namespace" csv=airlock-microgateway.v4.7.0 id=Ylg73 namespace=default phase=
E0828 11:37:18.721407 1 queueinformer_operator.go:312] "Unhandled Error" err="sync \"default/airlock-microgateway.v4.7.0\" failed: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
time="2025-08-28T11:37:18Z" level=info msg="operatorgroup incorrect" csv=airlock-microgateway.v4.7.0 error="<nil>" id=0bkNU namespace=default phase=
time="2025-08-28T11:37:18Z" level=info msg="not in operatorgroup namespace" csv=airlock-microgateway.v4.7.0 id=fNND1 namespace=default phase=
{"level":"error","ts":"2025-08-28T11:37:18Z","msg":"Reconciler error","controller":"operatorcondition","controllerGroup":"operators.coreos.com","controllerKind":"OperatorCondition","OperatorCondition":{"name":"airlock-microgateway.v4.7.0","namespace":"default"},"namespace":"default","name":"airlock-microgateway.v4.7.0","reconcileID":"4b09e31c-81a1-40c6-a65d-3832b00c39fd","error":"Deployment.apps \"airlock-microgateway-operator\" not found","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:353\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202"}
time="2025-08-28T11:37:18Z" level=warning msg="error adding operatorgroup annotations" csv=airlock-microgateway.v4.7.0 error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" id=VEnTj namespace=default opgroup=operator-sdk-og phase=
time="2025-08-28T11:37:18Z" level=info msg="operatorgroup incorrect" csv=airlock-microgateway.v4.7.0 error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" id=VEnTj namespace=default phase=
time="2025-08-28T11:37:18Z" level=info msg="not in operatorgroup namespace" csv=airlock-microgateway.v4.7.0 id=TFWki namespace=default phase=
E0828 11:37:18.750684 1 queueinformer_operator.go:312] "Unhandled Error" err="sync \"default/airlock-microgateway.v4.7.0\" failed: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
time="2025-08-28T11:37:18Z" level=warning msg="error adding operatorgroup annotations" csv=airlock-microgateway.v4.7.0 error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" namespace=default operatorGroup=operator-sdk-og
time="2025-08-28T11:37:18Z" level=warning msg="failed to annotate CSVs in operatorgroup after group change" error="Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" namespace=default operatorGroup=operator-sdk-og
E0828 11:37:18.750720 1 queueinformer_operator.go:312] "Unhandled Error" err="sync \"default/operator-sdk-og\" failed: Operation cannot be fulfilled on clusterserviceversions.operators.coreos.com \"airlock-microgateway.v4.7.0\": the object has been modified; please apply your changes to the latest version and try again" logger="UnhandledError"
time="2025-08-28T11:37:18Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=airlock-microgateway.v4.7.0 error="no owned roles found" id=BsMsL namespace=default phase=
E0828 11:37:18.757327 1 queueinformer_operator.go:312] "Unhandled Error" err="sync \"default/airlock-microgateway.v4.7.0\" failed: no owned roles found" logger="UnhandledError"
time="2025-08-28T11:37:18Z" level=info msg="scheduling ClusterServiceVersion for requirement verification" csv=airlock-microgateway.v4.7.0 id=bQZBE namespace=default phase=
I0828 11:37:18.759228 1 event.go:377] Event(v1.ObjectReference{Kind:"ClusterServiceVersion", Namespace:"default", Name:"airlock-microgateway.v4.7.0", UID:"792f738a-9883-4453-8ea0-95cbbcaef92a", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"1454", FieldPath:""}): type: 'Normal' reason: 'RequirementsUnknown' requirements not yet checked
Environment
-
operator-lifecycle-manager version:
v0.33.0 -
Kubernetes version information:
$ kubectl version
Client Version: v1.32.1
Kustomize Version: v5.5.0
Server Version: v1.31.4+k3s1
- Kubernetes cluster kind:
K3d: v1.31.4+k3s1
Possible Solution
Handle error and provide a helpful message
Additional context
The root cause was some missing CRDs inside the cluster. The install plan did show the issue:
conditions:
- lastTransitionTime: "2025-08-28T11:38:17Z"
lastUpdateTime: "2025-08-28T11:38:17Z"
message: 'api-server resource not found installing ServiceMonitor airlock-microgateway-operator-leader:
GroupVersionKind monitoring.coreos.com/v1, Kind=ServiceMonitor not found on
the cluster. This API may have been deprecated and removed, see https://kubernetes.io/docs/reference/using-api/deprecation-guide/
for more information.'
reason: InstallComponentFailed
status: "False"
type: Installed
message: 'api-server resource not found installing ServiceMonitor airlock-microgateway-operator-leader:
GroupVersionKind monitoring.coreos.com/v1, Kind=ServiceMonitor not found on the
cluster. This API may have been deprecated and removed, see https://kubernetes.io/docs/reference/using-api/deprecation-guide/
for more information.'
phase: Failed