OLM: update to v0.19.1 broke installation

[pohly]

Found on release-1.0 branch during make test_e2e TEST_E2E_FOCUS=olm.API TEST_E2E_ARGS=-ginkgo.failFast:

+ /nvme/gopath/src/github.com/intel/pmem-csi/_work/bin/operator-sdk run bundle --namespace default --timeout 5m 172.17.42.1:5001/pmem-csi-bundle:v1.0.0 --skip-tls
          time="2021-11-12T08:05:39+01:00" level=info msg="Successfully created registry pod: 172-17-42-1-5001-pmem-csi-bundle-v1-0-0"
          time="2021-11-12T08:05:39+01:00" level=info msg="Created CatalogSource: pmem-csi-operator-catalog"
          time="2021-11-12T08:05:39+01:00" level=info msg="OperatorGroup \"operator-sdk-og\" created"
          time="2021-11-12T08:05:39+01:00" level=info msg="Created Subscription: pmem-csi-operator-v1-0-0-sub"
          time="2021-11-12T08:10:32+01:00" level=fatal msg="Failed to run bundle: install plan is not available for the subscription pmem-csi-operator-v1-0-0-sub: timed out waiting for the condition\n"

It works when installing OLM v0.18.3.

There are similar issues on the master branch:

/mnt/workspace/pmem-csi-periodic/_work/bin/operator-sdk run bundle --namespace default --timeout 5m 172.17.0.1:5000/pmem-csi-bundle:v100.0.0 --skip-tls
    time="2021-11-11T02:22:42Z" level=fatal msg="Failed to run bundle: create catalog: error creating catalog source: catalogsources.operators.coreos.com \"pmem-csi-operator-catalog\" already exists\n"

[pohly]

As a workaround I am going to fix the OLM version to 0.18.3, but this needs to be investigated.

[pohly]

More diagnostics, now with our devel branch:

$ kubectl describe --all-namespaces subscriptions
Name:         pmem-csi-operator-v100-0-0-sub
Namespace:    default
Labels:       operators.coreos.com/pmem-csi-operator.default=
Annotations:  <none>
API Version:  operators.coreos.com/v1alpha1
Kind:         Subscription
Metadata:
  Creation Timestamp:  2021-11-17T13:21:35Z
  Generation:          1
  Managed Fields:
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:catalogHealth:
        f:conditions:
        f:lastUpdated:
    Manager:      catalog
    Operation:    Update
    Subresource:  status
    Time:         2021-11-17T13:21:35Z
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:operators.coreos.com/pmem-csi-operator.default:
    Manager:      olm
    Operation:    Update
    Time:         2021-11-17T13:21:35Z
    API Version:  operators.coreos.com/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:channel:
        f:installPlanApproval:
        f:name:
        f:source:
        f:sourceNamespace:
        f:startingCSV:
    Manager:         operator-sdk
    Operation:       Update
    Time:            2021-11-17T13:21:35Z
  Resource Version:  21120
  UID:               d96c7834-6a74-4b6a-9f91-28e8841103f6
Spec:
  Channel:                alpha
  Install Plan Approval:  Manual
  Name:                   pmem-csi-operator
  Source:                 pmem-csi-operator-catalog
  Source Namespace:       default
  Starting CSV:           pmem-csi-operator.v100.0.0
Status:
  Catalog Health:
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              pmem-csi-operator-catalog
      Namespace:         default
      Resource Version:  21117
      UID:               f7422ba4-9ffb-43c0-b4dc-0a530a6c847f
    Healthy:             true
    Last Updated:        2021-11-17T13:21:36Z
    Catalog Source Ref:
      API Version:       operators.coreos.com/v1alpha1
      Kind:              CatalogSource
      Name:              operatorhubio-catalog
      Namespace:         olm
      Resource Version:  17519
      UID:               a8fb807c-b7b3-425f-bba6-4ef890b5c060
    Healthy:             true
    Last Updated:        2021-11-17T13:21:36Z
  Conditions:
    Last Transition Time:  2021-11-17T13:21:36Z
    Message:               all available catalogsources are healthy
    Reason:                AllCatalogSourcesHealthy
    Status:                False
    Type:                  CatalogSourcesUnhealthy
  Last Updated:            2021-11-17T13:21:36Z
Events:                    <none>

Note the odd "AllCatalogSourcesHealthy: False". This pod here might be responsible for it (not sure) and reports an error:

$ kubectl logs -n olm pods/catalog-operator-84976fd7df-x5tmr
...
E1117 13:21:35.798270       1 queueinformer_operator.go:290] sync {"update" "default/pmem-csi-operator-v100-0-0-sub"} failed: Operation cannot be fulfilled on subscriptions.operators.coreos.com "pmem-csi-operator-v100-0-0-sub": the object has been modified; please apply your changes to the latest version and try again
time="2021-11-17T13:21:35Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-17T13:21:35Z" level=info msg=syncing event=update reconciling="*v1alpha1.Subscription" selflink=
time="2021-11-17T13:21:36Z" level=warning msg="an error was encountered during reconciliation" error="Operation cannot be fulfilled on subscriptions.operators.coreos.com \"pmem-csi-operator-v100-0-0-sub\": the object has been modified; please apply your changes to the latest version and try again" event=update reconciling="*v1alpha1.Subscription" selflink=
...

This repeats a few times but then not anymore. Deleting that pod doesn't help, the recreated one has the same problem.