CRD can not be created sucessful in installlation of OLM v.0.24.0 version (same bug as #2778: Applying OLM CRDs fails due to last-applied-configuration annotation)

[shaojini]

Bug Report

What did you do?
A clear and concise description of the steps you took (or insert a code snippet).
root@K8s-master:# docker login
root@K8s-master:# export olm_release=v0.24.0
root@K8s-master:~# kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml
What did you expect to see?
A clear and concise description of what you expected to happen (or insert a code snippet).
Successful installation
What did you see instead? Under which circumstances?
A clear and concise description of what you expected to happen (or insert a code snippet).
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created
The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

When I switched into "operator-sdk" to install OLM, it did not show any error for the installation of OLM, however, the packageserver service is in "false":

root@K8s-master:~# operator-sdk olm install

INFO[0000] Fetching CRDs for version "latest"
INFO[0000] Fetching resources for resolved version "latest"
I0519 20:39:28.599145 64106 request.go:690] Waited for 1.01264153s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.7:6443/apis/cilium.io/v2?timeout=32s
INFO[0008] Creating CRDs and resources
INFO[0008] Creating CustomResourceDefinition "catalogsources.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "clusterserviceversions.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "installplans.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "olmconfigs.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "operatorconditions.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "operatorgroups.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "operators.operators.coreos.com"
INFO[0008] Creating CustomResourceDefinition "subscriptions.operators.coreos.com"
INFO[0009] Creating Namespace "olm"
INFO[0009] Creating Namespace "operators"
INFO[0009] Creating ServiceAccount "olm/olm-operator-serviceaccount"
INFO[0009] Creating ClusterRole "system:controller:operator-lifecycle-manager"
INFO[0009] Creating ClusterRoleBinding "olm-operator-binding-olm"
INFO[0009] Creating OLMConfig "cluster"
I0519 20:39:38.648832 64106 request.go:690] Waited for 1.447733128s due to client-side throttling, not priority and fairness, request: GET:https://192.168.0.7:6443/apis/operators.coreos.com/v1alpha2?timeout=32s
INFO[0012] Creating Deployment "olm/olm-operator"
INFO[0012] Creating Deployment "olm/catalog-operator"
INFO[0012] Creating ClusterRole "aggregate-olm-edit"
INFO[0012] Creating ClusterRole "aggregate-olm-view"
INFO[0012] Creating OperatorGroup "operators/global-operators"
INFO[0012] Creating OperatorGroup "olm/olm-operators"
INFO[0012] Creating ClusterServiceVersion "olm/packageserver"
INFO[0012] Creating CatalogSource "olm/operatorhubio-catalog"
INFO[0012] Waiting for deployment/olm-operator rollout to complete
INFO[0012] Waiting for Deployment "olm/olm-operator" to rollout: 0 of 1 updated replicas are available
INFO[0014] Deployment "olm/olm-operator" successfully rolled out
INFO[0014] Waiting for deployment/catalog-operator rollout to complete
INFO[0014] Deployment "olm/catalog-operator" successfully rolled out
INFO[0014] Waiting for deployment/packageserver rollout to complete
INFO[0014] Waiting for Deployment "olm/packageserver" to appear
INFO[0015] Waiting for Deployment "olm/packageserver" to rollout: 0 of 2 updated replicas are available
INFO[0028] Deployment "olm/packageserver" successfully rolled out
INFO[0028] Successfully installed OLM version "latest"

NAME NAMESPACE KIND STATUS
catalogsources.operators.coreos.com CustomResourceDefinition Installed
clusterserviceversions.operators.coreos.com CustomResourceDefinition Installed
installplans.operators.coreos.com CustomResourceDefinition Installed
olmconfigs.operators.coreos.com CustomResourceDefinition Installed
operatorconditions.operators.coreos.com CustomResourceDefinition Installed
operatorgroups.operators.coreos.com CustomResourceDefinition Installed
operators.operators.coreos.com CustomResourceDefinition Installed
subscriptions.operators.coreos.com CustomResourceDefinition Installed
olm Namespace Installed
operators Namespace Installed
olm-operator-serviceaccount olm ServiceAccount Installed
system:controller:operator-lifecycle-manager ClusterRole Installed
olm-operator-binding-olm ClusterRoleBinding Installed
cluster OLMConfig Installed
olm-operator olm Deployment Installed
catalog-operator olm Deployment Installed
aggregate-olm-edit ClusterRole Installed
aggregate-olm-view ClusterRole Installed
global-operators operators OperatorGroup Installed
olm-operators olm OperatorGroup Installed
packageserver olm ClusterServiceVersion Installed
operatorhubio-catalog olm CatalogSource Installed

However,

root@K8s-master:~# kubectl get apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com

E0519 21:20:02.374158 88834 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.375416 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.377921 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.380392 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.405270 88834 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.427846 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.433485 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:20:02.436921 88834 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
NAME SERVICE AVAILABLE AGE
v1.packages.operators.coreos.com olm/packageserver-service False (FailedDiscoveryCheck) 40m

The "packages.operators.coreos.com" is not functioning:

root@K8s-master:~# kubectl get crd | grep operators.coreos.com

E0519 21:28:58.773330 94129 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:28:58.774865 94129 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:28:58.777895 94129 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
E0519 21:28:58.781929 94129 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
catalogsources.operators.coreos.com 2023-05-19T17:39:35Z
clusterserviceversions.operators.coreos.com 2023-05-19T17:39:35Z
installplans.operators.coreos.com 2023-05-19T17:39:35Z
olmconfigs.operators.coreos.com 2023-05-19T17:39:35Z
operatorconditions.operators.coreos.com 2023-05-19T17:39:35Z
operatorgroups.operators.coreos.com 2023-05-19T17:39:35Z
operators.operators.coreos.com 2023-05-19T17:39:35Z
subscriptions.operators.coreos.com 2023-05-19T17:39:35Z

Environment

• operator-lifecycle-manager version: v0.24.0

• Kubernetes version information:

root@K8s-master:~# kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:40:17Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.4", GitCommit:"f89670c3aa4059d6999cb42e23ccb4f0b9a03979", GitTreeState:"clean", BuildDate:"2023-04-12T12:05:35Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  root@K8s-master:~# kubectl cluster-info
  E0519 21:39:38.159210 100380 memcache.go:287] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
  E0519 21:39:38.166851 100380 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
  E0519 21:39:38.169478 100380 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
  E0519 21:39:38.172656 100380 memcache.go:121] couldn't get resource list for packages.operators.coreos.com/v1: the server is currently unable to handle the request
  Kubernetes control plane is running at https://192.168.0.7:6443
  CoreDNS is running at https://192.168.0.7:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Possible Solution

Additional context
Add any other context about the problem here.

[grokspawn]

Hi @shaojini, thanks for the issue.
The underlying problem is expressed in the message:

The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

which is displayed when the kubectl apply fails. Unfortunately, it leaves the CRDs incompletely installed, and worse it creates the situation where attempting to repeat the install operation won't help.

The underlying condition is the kubectl.kubernetes.io/last-applied-configuration annotation which kubectl creates whenever it performs a kubectl apply operation. This duplicates much of the CRDs annotations and so it makes it impossible to use CRs which are much more than half the max value of 262144 bytes.

The easiest resolution is manual, cleaning up old resources and using kubectl create instead of kubectl apply (which will not create the problematic annotation). Since you are working w/in a kind cluster, the easiest approach is to do

kind delete clusters kind
kind create cluster
export olm_release=v0.24.0
kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml

I'll try to get this bug in the agenda for discussion since we've evidently passed the threshold where users start encountering this and it's not great UX.

Could you update this issue with the source of your instructions that got you started here? It may be that we need to update them as well.

Thanks!

[grokspawn]

Follow-up: I stuck this on the community meeting on 23rd May: agenda.

[shaojini]

Hi @shaojini, thanks for the issue. The underlying problem is expressed in the message:

The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

which is displayed when the kubectl apply fails. Unfortunately, it leaves the CRDs incompletely installed, and worse it creates the situation where attempting to repeat the install operation won't help.

The underlying condition is the kubectl.kubernetes.io/last-applied-configuration annotation which kubectl creates whenever it performs a kubectl apply operation. This duplicates much of the CRDs annotations and so it makes it impossible to use CRs which are much more than half the max value of 262144 bytes.

The easiest resolution is manual, cleaning up old resources and using kubectl create instead of kubectl apply (which will not create the problematic annotation). Since you are working w/in a kind cluster, the easiest approach is to do

kind delete clusters kind
kind create cluster
export olm_release=v0.24.0
kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml

I'll try to get this bug in the agenda for discussion since we've evidently passed the threshold where users start encountering this and it's not great UX.

Could you update this issue with the source of your instructions that got you started here? It may be that we need to update them as well.

Thanks!

Thanks @grokspawn reply.

I have removed all resources used OLM, removed the cluster totally, removed old version and reinstalled new version of kubeadm, kubectl, kubelet, created a new cluster.

After installing the "cert-manager" via
root@K8s-master:# kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.2/cert-manager.yaml
root@K8s-master:# kubectl --namespace cert-manager wait --for condition=ready pod -l app.kubernetes.io/instance=cert-manager

I have following results according to your instructions:

root@K8s-master:# export olm_release=v0.24.0
root@K8s-master:# kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml
customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com created
customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created

root@K8s-master:~# kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/olm.yaml
namespace/olm created
namespace/operators created
serviceaccount/olm-operator-serviceaccount created
clusterrole.rbac.authorization.k8s.io/system:controller:operator-lifecycle-manager created
clusterrolebinding.rbac.authorization.k8s.io/olm-operator-binding-olm created
olmconfig.operators.coreos.com/cluster created
deployment.apps/olm-operator created
deployment.apps/catalog-operator created
clusterrole.rbac.authorization.k8s.io/aggregate-olm-edit created
clusterrole.rbac.authorization.k8s.io/aggregate-olm-view created
operatorgroup.operators.coreos.com/global-operators created
operatorgroup.operators.coreos.com/olm-operators created
clusterserviceversion.operators.coreos.com/packageserver created
catalogsource.operators.coreos.com/operatorhubio-catalog created

It seems the OLM has been installed successfully, however, it is not:

root@K8s-master:~# kubectl get apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com
NAME SERVICE AVAILABLE AGE
v1.packages.operators.coreos.com olm/packageserver-service False (FailedDiscoveryCheck) 89s

root@K8s-master:~# kubectl get crds
NAME CREATED AT
catalogsources.operators.coreos.com 2023-05-22T09:19:42Z
certificaterequests.cert-manager.io 2023-05-22T09:17:11Z
certificates.cert-manager.io 2023-05-22T09:17:11Z
challenges.acme.cert-manager.io 2023-05-22T09:17:11Z
ciliumclusterwidenetworkpolicies.cilium.io 2023-05-22T09:02:56Z
ciliumendpoints.cilium.io 2023-05-22T09:02:54Z
ciliumexternalworkloads.cilium.io 2023-05-22T09:02:54Z
ciliumidentities.cilium.io 2023-05-22T09:02:54Z
ciliumloadbalancerippools.cilium.io 2023-05-22T09:02:54Z
ciliumnetworkpolicies.cilium.io 2023-05-22T09:02:55Z
ciliumnodeconfigs.cilium.io 2023-05-22T09:02:54Z
ciliumnodes.cilium.io 2023-05-22T09:02:54Z
clusterissuers.cert-manager.io 2023-05-22T09:17:11Z
clusterserviceversions.operators.coreos.com 2023-05-22T09:19:42Z
installplans.operators.coreos.com 2023-05-22T09:19:42Z
issuers.cert-manager.io 2023-05-22T09:17:11Z
olmconfigs.operators.coreos.com 2023-05-22T09:19:42Z
operatorconditions.operators.coreos.com 2023-05-22T09:19:42Z
operatorgroups.operators.coreos.com 2023-05-22T09:19:42Z
operators.operators.coreos.com 2023-05-22T09:19:42Z
orders.acme.cert-manager.io 2023-05-22T09:17:11Z
subscriptions.operators.coreos.com 2023-05-22T09:19:42Z
traces.gadget.kinvolk.io 2023-05-22T09:11:10Z

The packetserver service is fail in installing. What is wrong? Thanks.

[shaojini]

Hi @shaojini, thanks for the issue. The underlying problem is expressed in the message:

The CustomResourceDefinition "clusterserviceversions.operators.coreos.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

which is displayed when the kubectl apply fails. Unfortunately, it leaves the CRDs incompletely installed, and worse it creates the situation where attempting to repeat the install operation won't help.
The underlying condition is the kubectl.kubernetes.io/last-applied-configuration annotation which kubectl creates whenever it performs a kubectl apply operation. This duplicates much of the CRDs annotations and so it makes it impossible to use CRs which are much more than half the max value of 262144 bytes.
The easiest resolution is manual, cleaning up old resources and using kubectl create instead of kubectl apply (which will not create the problematic annotation). Since you are working w/in a kind cluster, the easiest approach is to do

kind delete clusters kind
kind create cluster
export olm_release=v0.24.0
kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml

I'll try to get this bug in the agenda for discussion since we've evidently passed the threshold where users start encountering this and it's not great UX.
Could you update this issue with the source of your instructions that got you started here? It may be that we need to update them as well.
Thanks!

Thanks @grokspawn reply.

I have removed all resources used OLM, removed the cluster totally, removed old version and reinstalled new version of kubeadm, kubectl, kubelet, created a new cluster.

After installing the "cert-manager" via root@K8s-master:# kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.2/cert-manager.yaml root@K8s-master:# kubectl --namespace cert-manager wait --for condition=ready pod -l app.kubernetes.io/instance=cert-manager

I have following results according to your instructions:

root@K8s-master:# export olm_release=v0.24.0 root@K8s-master:# kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/crds.yaml customresourcedefinition.apiextensions.k8s.io/catalogsources.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/clusterserviceversions.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/installplans.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/olmconfigs.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/operatorconditions.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/operatorgroups.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/operators.operators.coreos.com created customresourcedefinition.apiextensions.k8s.io/subscriptions.operators.coreos.com created

root@K8s-master:~# kubectl create -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/${olm_release}/olm.yaml namespace/olm created namespace/operators created serviceaccount/olm-operator-serviceaccount created clusterrole.rbac.authorization.k8s.io/system:controller:operator-lifecycle-manager created clusterrolebinding.rbac.authorization.k8s.io/olm-operator-binding-olm created olmconfig.operators.coreos.com/cluster created deployment.apps/olm-operator created deployment.apps/catalog-operator created clusterrole.rbac.authorization.k8s.io/aggregate-olm-edit created clusterrole.rbac.authorization.k8s.io/aggregate-olm-view created operatorgroup.operators.coreos.com/global-operators created operatorgroup.operators.coreos.com/olm-operators created clusterserviceversion.operators.coreos.com/packageserver created catalogsource.operators.coreos.com/operatorhubio-catalog created

It seems the OLM has been installed successfully, however, it is not:

root@K8s-master:~# kubectl get apiservices.apiregistration.k8s.io v1.packages.operators.coreos.com NAME SERVICE AVAILABLE AGE v1.packages.operators.coreos.com olm/packageserver-service False (FailedDiscoveryCheck) 89s

root@K8s-master:~# kubectl get crds NAME CREATED AT catalogsources.operators.coreos.com 2023-05-22T09:19:42Z certificaterequests.cert-manager.io 2023-05-22T09:17:11Z certificates.cert-manager.io 2023-05-22T09:17:11Z challenges.acme.cert-manager.io 2023-05-22T09:17:11Z ciliumclusterwidenetworkpolicies.cilium.io 2023-05-22T09:02:56Z ciliumendpoints.cilium.io 2023-05-22T09:02:54Z ciliumexternalworkloads.cilium.io 2023-05-22T09:02:54Z ciliumidentities.cilium.io 2023-05-22T09:02:54Z ciliumloadbalancerippools.cilium.io 2023-05-22T09:02:54Z ciliumnetworkpolicies.cilium.io 2023-05-22T09:02:55Z ciliumnodeconfigs.cilium.io 2023-05-22T09:02:54Z ciliumnodes.cilium.io 2023-05-22T09:02:54Z clusterissuers.cert-manager.io 2023-05-22T09:17:11Z clusterserviceversions.operators.coreos.com 2023-05-22T09:19:42Z installplans.operators.coreos.com 2023-05-22T09:19:42Z issuers.cert-manager.io 2023-05-22T09:17:11Z olmconfigs.operators.coreos.com 2023-05-22T09:19:42Z operatorconditions.operators.coreos.com 2023-05-22T09:19:42Z operatorgroups.operators.coreos.com 2023-05-22T09:19:42Z operators.operators.coreos.com 2023-05-22T09:19:42Z orders.acme.cert-manager.io 2023-05-22T09:17:11Z subscriptions.operators.coreos.com 2023-05-22T09:19:42Z traces.gadget.kinvolk.io 2023-05-22T09:11:10Z

The packetserver service is fail in installing. What is wrong? Thanks.

The pods of packageserver have been created:
root@K8s-master:~# kubectl get pod -n olm
NAME READY STATUS RESTARTS AGE
catalog-operator-594448c75b-6xfzj 1/1 Running 0 113m
olm-operator-5589585cd-k7rz5 1/1 Running 0 113m
operatorhubio-catalog-ljns9 1/1 Running 0 113m
packageserver-5758fb5f7-cjp9h 1/1 Running 0 113m
packageserver-5758fb5f7-hf8qc 1/1 Running 0 113m

However, CRD of the packageserver had not been created successfully.

[grokspawn]

Hi @shaojini. This is why I asked where your instruction source was. That is just one step towards installing OLM. In order to get the ability to query the packagemanifests api, you have to install it via

kubectl apply -f https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/olm.yaml

If you delete/create your kind cluster and follow the quickstart at https://olm.operatorframework.io/docs/getting-started/ then it includes the CRDs, APIservices, default catalog sources to be fully ready. It's a much smoother user experience.

[awgreene]

Created #2969 to document how to perform an upgrade.

[shaojini]

Hi @grokspawn . I really use the link given by you to install the OLM. Now the main problem is the "apiServices" can not be installed successfully. I try to find the problem with the following command:

root@K8s-master:~# kubectl describe csv packageserver -n olm

Name: packageserver
Namespace: olm
Labels: olm.version=v0.24.0
Annotations: olm.operatorGroup: olm-operators
olm.operatorNamespace: olm
olm.targetNamespaces: olm
API Version: operators.coreos.com/v1alpha1
Kind: ClusterServiceVersion

.........

                Image:              quay.io/operator-framework/olm@sha256:f9ea8cef95ac9b31021401d4863711a5eec904536b449724e0f00357548a31e7
            Image Pull Policy:  Always

.........

Events:
Type Reason Age From Message

---

Normal RequirementsUnknown 27m operator-lifecycle-manager requirements not yet checked
Normal InstallWaiting 17m (x4 over 27m) operator-lifecycle-manager apiServices not installed
Normal AllRequirementsMet 12m (x5 over 27m) operator-lifecycle-manager all requirements found, attempting install
Warning InstallCheckFailed 12m (x6 over 22m) operator-lifecycle-manager install timeout
Normal InstallSucceeded 7m40s (x9 over 27m) operator-lifecycle-manager waiting for install components to report healthy
Normal NeedsReinstall 2m40s (x7 over 22m) operator-lifecycle-manager apiServices not installed

The result shows that it seems there is an image to be download from docker hub. The main problem might be the Docker Hub are limiting to pull images for the public Decker registry currently (I have met the same problem for creating a pot when pulling images from Docker hub). In order to resolve the problem of image pulling from the docker hub, it needs to provide the credential for pulling with k8s pod specification. I don't know how the problem can be resolved for the apiServices installing.

[acelinkio]

This can be partially avoided if you force the kubectl commands to use server-side rendering which avoid adding the last known configuration field inside of the metadata.

However, for those attempting to apply these manifests via GitOps--you will face kubernetes-sigs/structured-merge-diff#130 where the manifests will never align. The manifests in this repository are missing the ContainerPort protocol and caused ArgoCD to barf.

There really needs to be more thought into the ease of use of this project. There is no supported way for deploying this in a declarative manner. No helm chart support. No way to deploy in a GitOps method.

ARM installation works when you use their shell script.... but still required to do a manual edit to one of the deployments.

[fgiloux]

Hi @acelinkit

Thanks for pointing out GitOps considerations. That's a really good point.
I think that we should add the port protocol to the manifests. That said it is possible to configure ArgoCD and I believe other GitOps tools to ignore specific fields

replace can also be used as an alternative to SSA.

@shaojini I think that your initial request has been addressed. The slack channel may be a better place to help you with troubleshooting the apiservice issue. The images are pulled from quay.io not dockerhub. Looking at the logs of the olm-operator may help.

[fgiloux]

@acelinkit checking the latest version the port protocol has been added to the manifests, also here, last month. It should be part of the next release

[acelinkio]

Hey @fgiloux. Here is some additional context on my remarks.

I attempted to apply the helm chart directly inside this repo from ArgoCD using the v0.24.0 tag. The manifests missing port issue came up after doing server-side apply and would not apply as the manifests were rejected. I could have probably forced somerhing to eventually work, but I gave up because charts are not supported or the direction this project aims to support. #829

So I followed the docs.The base Readme https://github.com/operator-framework/operator-lifecycle-manager/tree/master#installation points to installation docs https://github.com/operator-framework/operator-lifecycle-manager/blob/master/doc/install/install.md. the installation docs then defer to the release page for installing a specific release. https://github.com/operator-framework/operator-lifecycle-manager/releases. The release notes have you download and run https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.24.0/install.sh

At that point I could go deconstruct the shell script to identify what it's doing. Then write a declarative way for what it's doing. So I just run the shell script and move on.

When I go to install an operator I faced another OLM problem and opened an issue. #2971. After applying a workaround my experience with operators was not great. ArgoCD operator does not have a ARM set of images. MetalLB operator had ARM images but I couldn't get it to work.

Just trying to share my user experience. Other hobbiests I have talked to outright avoid OLM entirely.

[fgiloux]

@acelinkit thanks for taking the time and providing this valuable feedback.

[gdoteof]

Just trying to share my user experience. Other hobbiests I have talked to outright avoid OLM entirely.

I am so many hours into realizing that this is a bug and not my noobery. i am also on ARM and have found that operators themselves work well enough but any attempts at using OLM are a nightmare of endless architecture related errors.