make OCS exits successfully but leaves an increasing number of pods in Terminating state

[mcornea]

make OCS exits successfully but leaves a bunch of pods in Terminating state:

[kni@rhhi-node-worker-0 install-scripts]$ make OCS
set -e; pushd OCS; make; popd
~/install-scripts/OCS ~/install-scripts
make[1]: Entering directory '/home/kni/install-scripts/OCS'
./deploy-ocs.sh
Loading kubeconfig from /home/kni/install-scripts/OpenShift/ocp/auth/kubeconfig
Using osd_devices /dev/sdb,/dev/sdc,/dev/sdd,/dev/sde of size 8
namespace/local-storage created
operatorgroup.operators.coreos.com/local-operator-group created
catalogsource.operators.coreos.com/local-storage-manifests created
catalogsource.operators.coreos.com/ocs-catalogsource created
namespace/openshift-storage created
operatorgroup.operators.coreos.com/openshift-storage-operatorgroup created
subscription.operators.coreos.com/ocs-subscription created
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
pod/ocs-operator-6f874fc946-s86l2 condition met
pod/rook-ceph-operator-85d5fc66fb-zkvmg condition met
pod/noobaa-operator-68f5f64cfd-8vnps condition met
pod/local-storage-operator-bcfd5765f-mw5zs condition met
clusterrole.rbac.authorization.k8s.io/cluster-admin added: "ocs-operator"
clusterrole.rbac.authorization.k8s.io/cluster-admin added: "local-storage-operator"
storageclass.storage.k8s.io/mon-sc created
Warning: Permanently added '192.168.123.124' (ECDSA) to the list of known hosts.
persistentvolume/hostpath0000 created
Warning: Permanently added '192.168.123.126' (ECDSA) to the list of known hosts.
persistentvolume/hostpath0001 created
Warning: Permanently added '192.168.123.111' (ECDSA) to the list of known hosts.
persistentvolume/hostpath0002 created
localvolume.local.storage.openshift.io/osd-disks created
node/rhhi-node-master-0 labeled
node/rhhi-node-master-1 labeled
node/rhhi-node-master-2 labeled
storagecluster.ocs.openshift.io/openshift-storage created
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
error: no matching resources found
pod/rook-ceph-mgr-a-6c8ff5b7d7-wvft9 condition met
storageclass.storage.k8s.io/openshift-storage-ceph-rbd patched
Waiting for the OSD jobs to be run...
error: no matching resources found
error: no matching resources found
job.batch/rook-ceph-osd-prepare-osd-0-f5fs9 condition met
job.batch/rook-ceph-osd-prepare-osd-1-fnh8t condition met
job.batch/rook-ceph-osd-prepare-osd-10-tgk66 condition met
job.batch/rook-ceph-osd-prepare-osd-11-6vvvr condition met
job.batch/rook-ceph-osd-prepare-osd-2-ksxr6 condition met
job.batch/rook-ceph-osd-prepare-osd-3-vfxqn condition met
job.batch/rook-ceph-osd-prepare-osd-4-4w5fg condition met
job.batch/rook-ceph-osd-prepare-osd-5-hmlb4 condition met
job.batch/rook-ceph-osd-prepare-osd-6-n4lcw condition met
job.batch/rook-ceph-osd-prepare-osd-7-w7t6p condition met
job.batch/rook-ceph-osd-prepare-osd-8-fxz4g condition met
job.batch/rook-ceph-osd-prepare-osd-9-rlndb condition met
make[1]: Leaving directory '/home/kni/install-scripts/OCS'
~/install-scripts
[kni@rhhi-node-worker-0 install-scripts]$ echo $?
0
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods
NAME                                                              READY   STATUS                 RESTARTS   AGE
csi-cephfsplugin-cxqj7                                            3/3     Running                0          7m11s
csi-cephfsplugin-provisioner-587c9ccf8d-4d49c                     4/4     Running                0          7m11s
csi-cephfsplugin-provisioner-587c9ccf8d-hr8g2                     4/4     Running                0          7m12s
csi-cephfsplugin-t9prc                                            3/3     Running                0          7m12s
csi-cephfsplugin-vdldp                                            3/3     Running                0          7m11s
csi-rbdplugin-7j72q                                               3/3     Running                0          7m12s
csi-rbdplugin-cdqvf                                               3/3     Running                0          7m12s
csi-rbdplugin-dgkws                                               3/3     Running                0          7m12s
csi-rbdplugin-provisioner-766d79f78-cf8dq                         5/5     Running                0          7m12s
csi-rbdplugin-provisioner-766d79f78-xlrdb                         5/5     Running                0          7m12s
local-storage-operator-bcfd5765f-mw5zs                            1/1     Running                0          8m29s
noobaa-core-0                                                     0/2     Pending                0          7m10s
noobaa-operator-68f5f64cfd-8vnps                                  1/1     Running                0          7m54s
ocs-operator-6f874fc946-s86l2                                     1/1     Running                0          7m54s
osd-disks-local-diskmaker-54frz                                   1/1     Running                0          7m12s
osd-disks-local-diskmaker-p9n6t                                   1/1     Running                0          7m12s
osd-disks-local-diskmaker-rrfgz                                   1/1     Running                0          7m12s
osd-disks-local-provisioner-28r2m                                 1/1     Running                0          7m12s
osd-disks-local-provisioner-5rc2d                                 1/1     Running                0          7m12s
osd-disks-local-provisioner-pp664                                 1/1     Running                0          7m12s
rook-ceph-drain-canary-rhhi-node-master-0-545fff7849-brw8c        0/1     Terminating            0          40s
rook-ceph-drain-canary-rhhi-node-master-0-5467b5d76c-2625s        0/1     Terminating            0          84s
rook-ceph-drain-canary-rhhi-node-master-0-5467b5d76c-7d4p7        0/1     Terminating            0          24s
rook-ceph-drain-canary-rhhi-node-master-0-547b7cdc8c-4xksm        0/1     Terminating            0          6s
rook-ceph-drain-canary-rhhi-node-master-0-547b7cdc8c-tbzq2        0/1     Terminating            0          68s
rook-ceph-drain-canary-rhhi-node-master-0-548b594bdc-qs8gt        0/1     Terminating            0          85s
rook-ceph-drain-canary-rhhi-node-master-0-54f6b57665-xk57h        0/1     Terminating            0          25s
rook-ceph-drain-canary-rhhi-node-master-0-55444d4c75-jtn2h        0/1     Terminating            0          3s
rook-ceph-drain-canary-rhhi-node-master-0-55696d9b99-xgvq8        0/1     Terminating            0          83s
rook-ceph-drain-canary-rhhi-node-master-0-55dbc4b4d9-64ldb        0/1     Terminating            0          48s
rook-ceph-drain-canary-rhhi-node-master-0-55dbc4b4d9-7kdq6        0/1     Terminating            0          5s
rook-ceph-drain-canary-rhhi-node-master-0-55dbc4b4d9-9jvhf        0/1     Terminating            0          66s
rook-ceph-drain-canary-rhhi-node-master-0-564ddf559c-txpq4        0/1     Terminating            0          28s
rook-ceph-drain-canary-rhhi-node-master-0-56558844c9-jdflt        0/1     Terminating            0          80s
rook-ceph-drain-canary-rhhi-node-master-0-5697797599-8sgkn        0/1     Terminating            0          59s
rook-ceph-drain-canary-rhhi-node-master-0-5697797599-cw8cn        0/1     Terminating            0          18s
rook-ceph-drain-canary-rhhi-node-master-0-569d496bb5-jzchr        0/1     Terminating            0          79s
rook-ceph-drain-canary-rhhi-node-master-0-56c85d77f9-sdnjr        0/1     Terminating            0          37s
rook-ceph-drain-canary-rhhi-node-master-0-56f497bb65-bkkhv        0/1     Terminating            0          58s
rook-ceph-drain-canary-rhhi-node-master-0-56f497bb65-fdf4r        0/1     Terminating            0          71s
rook-ceph-drain-canary-rhhi-node-master-0-56f497bb65-hq8vq        0/1     Terminating            0          101s
rook-ceph-drain-canary-rhhi-node-master-0-5755f67d59-cnjx2        0/1     Terminating            0          22s
rook-ceph-drain-canary-rhhi-node-master-0-57c6fd4799-n98kg        0/1     Terminating            0          82s
rook-ceph-drain-canary-rhhi-node-master-0-57d9f4f4cf-66zqh        0/1     Terminating            0          14s
rook-ceph-drain-canary-rhhi-node-master-0-5867f67b7c-zd4f2        0/1     Terminating            0          46s
rook-ceph-drain-canary-rhhi-node-master-0-588448db7c-8bcgz        0/1     Terminating            0          59s
rook-ceph-drain-canary-rhhi-node-master-0-589cc84d89-pkp82        0/1     Terminating            0          92s
rook-ceph-drain-canary-rhhi-node-master-0-58cd764d97-9bgrl        0/1     Terminating            0          43s
rook-ceph-drain-canary-rhhi-node-master-0-58cdccbb49-869g8        0/1     Terminating            0          30s
rook-ceph-drain-canary-rhhi-node-master-0-5b5c7f55c-m29gn         0/1     Terminating            0          34s
rook-ceph-drain-canary-rhhi-node-master-0-5b856d68f7-7dr42        0/1     Terminating            0          65s
rook-ceph-drain-canary-rhhi-node-master-0-5bd98c9b49-c456l        0/1     Terminating            0          74s
rook-ceph-drain-canary-rhhi-node-master-0-5bd98c9b49-nn8ms        0/1     Terminating            0          13s
rook-ceph-drain-canary-rhhi-node-master-0-5bffd8d879-9m56r        0/1     Terminating            0          7s
rook-ceph-drain-canary-rhhi-node-master-0-5c4c4d68d9-v52xz        0/1     Terminating            0          90s
rook-ceph-drain-canary-rhhi-node-master-0-5ccf5c756c-cf2xk        0/1     Terminating            0          98s
rook-ceph-drain-canary-rhhi-node-master-0-5d58bdcdcc-kfdpm        0/1     Terminating            0          108s
rook-ceph-drain-canary-rhhi-node-master-0-5ddcc86db5-crjwj        0/1     Terminating            0          60s
rook-ceph-drain-canary-rhhi-node-master-0-655db9dc45-hrx84        0/1     Terminating            0          86s
rook-ceph-drain-canary-rhhi-node-master-0-66649467fc-f4kkn        0/1     Terminating            0          76s
rook-ceph-drain-canary-rhhi-node-master-0-667c686879-4mpfp        0/1     Terminating            0          64s
rook-ceph-drain-canary-rhhi-node-master-0-675957fb97-62p6k        0/1     Terminating            0          29s
rook-ceph-drain-canary-rhhi-node-master-0-675957fb97-ds7jc        0/1     Terminating            0          56s
rook-ceph-drain-canary-rhhi-node-master-0-675957fb97-nf66q        0/1     Terminating            0          19s
rook-ceph-drain-canary-rhhi-node-master-0-675957fb97-zktjj        0/1     Terminating            0          94s
rook-ceph-drain-canary-rhhi-node-master-0-679b466cc5-ssrss        0/1     Terminating            0          97s
rook-ceph-drain-canary-rhhi-node-master-0-679c96bfdf-2sthr        0/1     Terminating            0          4s
rook-ceph-drain-canary-rhhi-node-master-0-68487d575f-46v6l        0/1     Terminating            0          55s
rook-ceph-drain-canary-rhhi-node-master-0-68487d575f-tsmbq        0/1     Terminating            0          10s
rook-ceph-drain-canary-rhhi-node-master-0-68d484fc69-9nmcx        0/1     Terminating            0          72s
rook-ceph-drain-canary-rhhi-node-master-0-6c474cdc59-q28sk        0/1     Terminating            0          45s
rook-ceph-drain-canary-rhhi-node-master-0-6c4cf86d75-h6g8f        0/1     Terminating            0          88s
rook-ceph-drain-canary-rhhi-node-master-0-6c6bf54775-66mgs        0/1     Terminating            0          54s
rook-ceph-drain-canary-rhhi-node-master-0-6c7c998985-f4sk2        0/1     Terminating            0          49s
rook-ceph-drain-canary-rhhi-node-master-0-6d747f78f5-chdp7        0/1     Terminating            0          34s
rook-ceph-drain-canary-rhhi-node-master-0-6d76b85977-4zpbg        0/1     Terminating            0          17s
rook-ceph-drain-canary-rhhi-node-master-0-6d76b85977-v2pnn        0/1     Terminating            0          67s
rook-ceph-drain-canary-rhhi-node-master-0-6f8969dfdf-nmjxm        0/1     Terminating            0          63s
rook-ceph-drain-canary-rhhi-node-master-0-6f9b798495-v85f8        0/1     Terminating            0          107s
rook-ceph-drain-canary-rhhi-node-master-0-6fc4f64c59-nkmf9        0/1     Terminating            0          2m8s
rook-ceph-drain-canary-rhhi-node-master-0-6fcf679c9c-zgh42        0/1     Terminating            0          16s
rook-ceph-drain-canary-rhhi-node-master-0-6fd68547c7-rhzlq        0/1     Terminating            0          42s
rook-ceph-drain-canary-rhhi-node-master-0-7567d9c697-sm6sq        0/1     Terminating            0          21s
rook-ceph-drain-canary-rhhi-node-master-0-7669b784ff-s6b98        0/1     Terminating            0          91s
rook-ceph-drain-canary-rhhi-node-master-0-76d6c5f995-nsnjm        0/1     Terminating            0          39s
rook-ceph-drain-canary-rhhi-node-master-0-76dd9c5985-46d7w        0/1     Terminating            0          2m12s
rook-ceph-drain-canary-rhhi-node-master-0-77b669dd7c-gsbq6        0/1     Terminating            0          75s
rook-ceph-drain-canary-rhhi-node-master-0-77cfff6d7f-fwr4n        0/1     Terminating            0          93s
rook-ceph-drain-canary-rhhi-node-master-0-77f68c4695-kccg9        0/1     Terminating            0          81s
rook-ceph-drain-canary-rhhi-node-master-0-78c95cd687-k5nnw        0/1     Terminating            0          69s
rook-ceph-drain-canary-rhhi-node-master-0-79995c64b5-qr798        0/1     Pending                0          1s
rook-ceph-drain-canary-rhhi-node-master-0-7d599c4c5f-g6rj2        0/1     Terminating            0          27s
rook-ceph-drain-canary-rhhi-node-master-0-7db7469c45-d5sfq        0/1     Terminating            0          33s
rook-ceph-drain-canary-rhhi-node-master-0-8458484475-5hcp6        0/1     Pending                0          2s
rook-ceph-drain-canary-rhhi-node-master-0-8464c75bf7-r428v        0/1     Terminating            0          61s
rook-ceph-drain-canary-rhhi-node-master-0-8499b74df7-ghkdz        0/1     Terminating            0          26s
rook-ceph-drain-canary-rhhi-node-master-0-84cf9d7c5-svwhx         0/1     Terminating            0          89s
rook-ceph-drain-canary-rhhi-node-master-0-8594b845ff-xqw9h        0/1     Terminating            0          77s
rook-ceph-drain-canary-rhhi-node-master-0-85dd46d97-5qxwp         0/1     Terminating            0          41s
rook-ceph-drain-canary-rhhi-node-master-0-85f54b699-d5qbf         0/1     Terminating            0          17s
rook-ceph-drain-canary-rhhi-node-master-0-867c4658cc-kj6k6        0/1     Terminating            0          50s
rook-ceph-drain-canary-rhhi-node-master-0-867f54f985-z7fxw        0/1     Terminating            0          51s
rook-ceph-drain-canary-rhhi-node-master-0-869d57c9ff-n2knx        0/1     Terminating            0          6s
rook-ceph-drain-canary-rhhi-node-master-0-86df7d86b5-mdtzx        0/1     Terminating            0          9s
rook-ceph-drain-canary-rhhi-node-master-0-86df7d86b5-mzwvh        0/1     Terminating            0          47s
rook-ceph-drain-canary-rhhi-node-master-0-8b57c9fd7-h6mhf         0/1     Terminating            0          32s
rook-ceph-drain-canary-rhhi-node-master-0-8f56fb96c-7w64p         0/1     Terminating            0          20s
rook-ceph-drain-canary-rhhi-node-master-0-954c8f767-77gld         0/1     Terminating            0          11s
rook-ceph-drain-canary-rhhi-node-master-0-bc66c55c-kpdp4          0/1     Terminating            0          36s
rook-ceph-drain-canary-rhhi-node-master-0-d685ddd65-pbl9d         0/1     Terminating            0          53s
rook-ceph-drain-canary-rhhi-node-master-0-d8dd88cd7-nvtgb         0/1     Terminating            0          57s
rook-ceph-drain-canary-rhhi-node-master-0-f464845d5-95g9h         0/1     Terminating            0          73s
rook-ceph-drain-canary-rhhi-node-master-0-f464845d5-crdf8         0/1     Terminating            0          2m9s
rook-ceph-drain-canary-rhhi-node-master-0-f88ddbd8c-nqfv5         0/1     Terminating            0          12s
rook-ceph-drain-canary-rhhi-node-master-0-f8b99cf5c-qsxg6         0/1     Terminating            0          27s
rook-ceph-drain-canary-rhhi-node-master-0-fbcc6fb65-gk5vh         0/1     Terminating            0          35s
rook-ceph-drain-canary-rhhi-node-master-1-5444fdf94d-xzjgs        0/1     Terminating            0          8s
rook-ceph-drain-canary-rhhi-node-master-1-5446d54bd8-4fczv        0/1     Terminating            0          17s
rook-ceph-drain-canary-rhhi-node-master-1-5446d54bd8-8ltpx        0/1     Terminating            0          43s
rook-ceph-drain-canary-rhhi-node-master-1-54574697b8-cwn79        0/1     Terminating            0          36s
rook-ceph-drain-canary-rhhi-node-master-1-54574697b8-g9zqn        0/1     Terminating            0          26s
rook-ceph-drain-canary-rhhi-node-master-1-54574697b8-h8kbg        0/1     Terminating            0          3s
rook-ceph-drain-canary-rhhi-node-master-1-54956bdb94-t89l9        0/1     Terminating            0          60s
rook-ceph-drain-canary-rhhi-node-master-1-54cdd4954-vczl4         0/1     Terminating            0          2s
rook-ceph-drain-canary-rhhi-node-master-1-54cdd4954-vn4gc         0/1     Terminating            0          39s
rook-ceph-drain-canary-rhhi-node-master-1-554f8ccc9b-qk8bp        0/1     Terminating            0          25s
rook-ceph-drain-canary-rhhi-node-master-1-555db9df4b-qqtkn        0/1     Terminating            0          13s
rook-ceph-drain-canary-rhhi-node-master-1-55df4dff54-gff7d        0/1     Terminating            0          37s
rook-ceph-drain-canary-rhhi-node-master-1-568dbf5f9d-xzlwb        0/1     Terminating            0          30s
rook-ceph-drain-canary-rhhi-node-master-1-57474cb888-4hvsw        0/1     Terminating            0          46s
rook-ceph-drain-canary-rhhi-node-master-1-57584fb58b-ns84p        0/1     Terminating            0          47s
rook-ceph-drain-canary-rhhi-node-master-1-57584fb58b-p7g5v        0/1     Terminating            0          58s
rook-ceph-drain-canary-rhhi-node-master-1-5769dc866b-s4s7c        0/1     Terminating            0          35s
rook-ceph-drain-canary-rhhi-node-master-1-5897f4bf4b-cct2d        0/1     Terminating            0          42s
rook-ceph-drain-canary-rhhi-node-master-1-58bc4fb7c6-xwxzm        0/1     Terminating            0          53s
rook-ceph-drain-canary-rhhi-node-master-1-595774f4c8-tl5bb        0/1     Terminating            0          24s
rook-ceph-drain-canary-rhhi-node-master-1-59867bd7dd-qmfnq        0/1     Terminating            0          6s
rook-ceph-drain-canary-rhhi-node-master-1-599ff4c784-thrr2        0/1     Terminating            0          77s
rook-ceph-drain-canary-rhhi-node-master-1-5b5564f856-r55hs        0/1     Terminating            0          74s
rook-ceph-drain-canary-rhhi-node-master-1-5b64bcb64-m7ppw         1/1     Terminating            0          2m19s
rook-ceph-drain-canary-rhhi-node-master-1-5bb5f8686b-qct5t        0/1     Terminating            0          63s
rook-ceph-drain-canary-rhhi-node-master-1-5bfdcb5cb6-gfhpm        0/1     Terminating            0          56s
rook-ceph-drain-canary-rhhi-node-master-1-5c599b559b-lqx8z        0/1     Terminating            0          5s
rook-ceph-drain-canary-rhhi-node-master-1-5c7d569df4-tr48v        0/1     Terminating            0          28s
rook-ceph-drain-canary-rhhi-node-master-1-5d85669754-k4k4j        0/1     Terminating            0          48s
rook-ceph-drain-canary-rhhi-node-master-1-5db98fc5bb-hlkvz        0/1     Terminating            0          77s
rook-ceph-drain-canary-rhhi-node-master-1-5db98fc5bb-stfxv        0/1     Terminating            0          25s
rook-ceph-drain-canary-rhhi-node-master-1-64744b79b6-pxp79        0/1     Terminating            0          32s
rook-ceph-drain-canary-rhhi-node-master-1-66444f4f54-nxdxl        0/1     Terminating            0          54s
rook-ceph-drain-canary-rhhi-node-master-1-665c6649c4-d4x2c        0/1     Terminating            0          75s
rook-ceph-drain-canary-rhhi-node-master-1-66fcf7f7d-6258q         0/1     Terminating            0          61s
rook-ceph-drain-canary-rhhi-node-master-1-6749cf4d6-6qqgc         0/1     Terminating            0          72s
rook-ceph-drain-canary-rhhi-node-master-1-67957654c8-hr2g2        0/1     Terminating            0          62s
rook-ceph-drain-canary-rhhi-node-master-1-67957654c8-wfv7k        0/1     Terminating            0          83s
rook-ceph-drain-canary-rhhi-node-master-1-67b94cf47b-p8v4z        0/1     Terminating            0          4s
rook-ceph-drain-canary-rhhi-node-master-1-67d986b45d-p88jz        0/1     Terminating            0          12s
rook-ceph-drain-canary-rhhi-node-master-1-6866f7db58-rckgm        0/1     Terminating            0          24s
rook-ceph-drain-canary-rhhi-node-master-1-69647bb4-rh84q          0/1     Terminating            0          59s
rook-ceph-drain-canary-rhhi-node-master-1-69d798f456-mf2rq        0/1     Terminating            0          56s
rook-ceph-drain-canary-rhhi-node-master-1-6b4cf59784-75w6l        0/1     Terminating            0          45s
rook-ceph-drain-canary-rhhi-node-master-1-6b9dd6fcdb-s4cvv        0/1     Terminating            0          33s
rook-ceph-drain-canary-rhhi-node-master-1-6c55cb46cb-4f7w8        0/1     Terminating            0          67s
rook-ceph-drain-canary-rhhi-node-master-1-6c55cb46cb-925v4        0/1     Terminating            0          21s
rook-ceph-drain-canary-rhhi-node-master-1-6f5d989864-t64g8        0/1     Terminating            0          56s
rook-ceph-drain-canary-rhhi-node-master-1-6f5d989864-zz925        0/1     Terminating            0          18s
rook-ceph-drain-canary-rhhi-node-master-1-6f7959bd54-7k7kj        0/1     Terminating            0          33s
rook-ceph-drain-canary-rhhi-node-master-1-6f7959bd54-t6d4g        0/1     Terminating            0          50s
rook-ceph-drain-canary-rhhi-node-master-1-746f7ff846-xkcxg        0/1     Terminating            0          22s
rook-ceph-drain-canary-rhhi-node-master-1-74867bbfd4-cz46h        0/1     Terminating            0          19s
rook-ceph-drain-canary-rhhi-node-master-1-74867bbfd4-s26h9        0/1     Terminating            0          11s
rook-ceph-drain-canary-rhhi-node-master-1-74cd78f58b-twc9d        0/1     Terminating            0          6s
rook-ceph-drain-canary-rhhi-node-master-1-75f58cbbbb-mwlbg        0/1     Terminating            0          34s
rook-ceph-drain-canary-rhhi-node-master-1-764b88ffd4-jlz9p        0/1     Terminating            0          50s
rook-ceph-drain-canary-rhhi-node-master-1-7698b47c4-pxslh         0/1     Terminating            0          51s
rook-ceph-drain-canary-rhhi-node-master-1-775d95c966-pk4db        0/1     Terminating            0          18s
rook-ceph-drain-canary-rhhi-node-master-1-7779884786-4hv4v        0/1     Terminating            0          10s
rook-ceph-drain-canary-rhhi-node-master-1-7779884786-fn445        0/1     Terminating            0          40s
rook-ceph-drain-canary-rhhi-node-master-1-779447d6d6-6kmkt        0/1     Terminating            0          46s
rook-ceph-drain-canary-rhhi-node-master-1-7965658dcb-r2x7m        0/1     Terminating            0          55s
rook-ceph-drain-canary-rhhi-node-master-1-79c448db6-j7x7c         0/1     Terminating            0          69s
rook-ceph-drain-canary-rhhi-node-master-1-7b499987c6-jjtsn        0/1     Terminating            0          53s
rook-ceph-drain-canary-rhhi-node-master-1-7bf484466b-fsqnn        0/1     Terminating            0          27s
rook-ceph-drain-canary-rhhi-node-master-1-7cf44cf898-45dcd        0/1     Terminating            0          41s
rook-ceph-drain-canary-rhhi-node-master-1-7d56775dd6-dxwsb        0/1     Terminating            0          83s
rook-ceph-drain-canary-rhhi-node-master-1-7d5f8d4466-bd56q        0/1     Terminating            0          16s
rook-ceph-drain-canary-rhhi-node-master-1-7f5c9df898-2nnct        0/1     Terminating            0          9s
rook-ceph-drain-canary-rhhi-node-master-1-7f9dd94c46-6sx5l        0/1     Terminating            0          35s
rook-ceph-drain-canary-rhhi-node-master-1-7f9dd94c46-fdtqh        0/1     Terminating            0          14s
rook-ceph-drain-canary-rhhi-node-master-1-7fb6955454-dcgjl        0/1     Pending                0          1s
rook-ceph-drain-canary-rhhi-node-master-1-7fc4899cdd-sb7r5        0/1     Terminating            0          81s
rook-ceph-drain-canary-rhhi-node-master-1-7fc88f4f9b-9n7s8        0/1     Terminating            0          29s
rook-ceph-drain-canary-rhhi-node-master-1-846cd5f558-xmmnw        0/1     Terminating            0          71s
rook-ceph-drain-canary-rhhi-node-master-1-846d99b9f4-h4hnl        0/1     Terminating            0          82s
rook-ceph-drain-canary-rhhi-node-master-1-84ccdccf68-tw8rn        0/1     Terminating            0          20s
rook-ceph-drain-canary-rhhi-node-master-1-9985d4f4b-j8kpn         0/1     Terminating            0          49s
rook-ceph-drain-canary-rhhi-node-master-1-9df57b8d6-nbbjx         0/1     Terminating            0          20s
rook-ceph-drain-canary-rhhi-node-master-1-bcff5c744-426wq         0/1     Terminating            0          60s
rook-ceph-drain-canary-rhhi-node-master-1-c695545bb-qbbp5         0/1     Terminating            0          39s
rook-ceph-drain-canary-rhhi-node-master-1-c9f959dd8-tdq46         0/1     Terminating            0          45s
rook-ceph-drain-canary-rhhi-node-master-1-cf547849d-9krxt         0/1     Terminating            0          73s
rook-ceph-drain-canary-rhhi-node-master-1-d88dcc68-mn2kx          0/1     Terminating            0          80s
rook-ceph-drain-canary-rhhi-node-master-1-f775ccbf6-gknk5         0/1     Terminating            0          48s
rook-ceph-drain-canary-rhhi-node-master-1-f78db8fdb-78pgl         0/1     Terminating            0          57s
rook-ceph-drain-canary-rhhi-node-master-1-f8999bbcd-7578w         0/1     Terminating            0          34s
rook-ceph-drain-canary-rhhi-node-master-1-f8f759784-jpjxh         1/1     Running                0          2m18s
rook-ceph-drain-canary-rhhi-node-master-2-54cc4d59b7-462pk        0/1     Terminating            0          94s
rook-ceph-drain-canary-rhhi-node-master-2-558859766f-jrccn        0/1     Terminating            0          24s
rook-ceph-drain-canary-rhhi-node-master-2-55cd8c455c-bbbjz        0/1     Terminating            0          110s
rook-ceph-drain-canary-rhhi-node-master-2-57556cf8bf-97gfs        0/1     Terminating            0          13s
rook-ceph-drain-canary-rhhi-node-master-2-597b745549-89vsl        0/1     Terminating            0          12s
rook-ceph-drain-canary-rhhi-node-master-2-5bf8799445-ffwqk        0/1     Terminating            0          51s
rook-ceph-drain-canary-rhhi-node-master-2-5c9cb58575-75ssd        0/1     Terminating            0          41s
rook-ceph-drain-canary-rhhi-node-master-2-5f9b956789-fvslc        0/1     Pending                0          1s
rook-ceph-drain-canary-rhhi-node-master-2-5fd46c885f-v7xr9        0/1     Terminating            0          42s
rook-ceph-drain-canary-rhhi-node-master-2-6586dcd95-c7qrm         0/1     Terminating            0          18s
rook-ceph-drain-canary-rhhi-node-master-2-667fb8b5c5-zfrk9        0/1     Terminating            0          93s
rook-ceph-drain-canary-rhhi-node-master-2-66bfd8c5bf-tmsz9        0/1     Terminating            0          14s
rook-ceph-drain-canary-rhhi-node-master-2-66f455b449-krx24        0/1     Terminating            0          21s
rook-ceph-drain-canary-rhhi-node-master-2-6759b6fcc5-crnxl        0/1     Terminating            0          42s
rook-ceph-drain-canary-rhhi-node-master-2-676444d58c-l5s2s        0/1     Terminating            0          15s
rook-ceph-drain-canary-rhhi-node-master-2-67ccdd4bbc-65d8c        0/1     Terminating            0          20s
rook-ceph-drain-canary-rhhi-node-master-2-67f646f9fc-b5bgg        0/1     Terminating            0          103s
rook-ceph-drain-canary-rhhi-node-master-2-67ff84b689-8s2fr        0/1     Terminating            0          28s
rook-ceph-drain-canary-rhhi-node-master-2-6bfff7c94c-vgft5        0/1     Terminating            0          47s
rook-ceph-drain-canary-rhhi-node-master-2-6c47cd485f-rdpvx        0/1     Terminating            0          30s
rook-ceph-drain-canary-rhhi-node-master-2-6c976b8997-cv5g5        0/1     Terminating            0          9s
rook-ceph-drain-canary-rhhi-node-master-2-6cfc46b89-lqwlz         0/1     Terminating            0          83s
rook-ceph-drain-canary-rhhi-node-master-2-6f9ff5c847-ndr4f        0/1     Terminating            0          49s
rook-ceph-drain-canary-rhhi-node-master-2-6fbb8bd94c-vkjcs        0/1     Terminating            0          29s
rook-ceph-drain-canary-rhhi-node-master-2-745f8f86ff-rzlx4        0/1     Terminating            0          7s
rook-ceph-drain-canary-rhhi-node-master-2-74889996d7-jkc9z        0/1     Terminating            0          89s
rook-ceph-drain-canary-rhhi-node-master-2-74c468d45-2wtr7         0/1     Terminating            0          90s
rook-ceph-drain-canary-rhhi-node-master-2-757d84f9f9-p9tv2        0/1     Terminating            0          32s
rook-ceph-drain-canary-rhhi-node-master-2-757d84f9f9-tv5wz        0/1     Terminating            0          71s
rook-ceph-drain-canary-rhhi-node-master-2-76b458d97f-tghm4        0/1     Terminating            0          34s
rook-ceph-drain-canary-rhhi-node-master-2-779b9855cf-26fmx        0/1     Terminating            0          46s
rook-ceph-drain-canary-rhhi-node-master-2-779b9855cf-lfjz5        0/1     Terminating            0          35s
rook-ceph-drain-canary-rhhi-node-master-2-785c5d989c-pb8jq        0/1     Terminating            0          56s
rook-ceph-drain-canary-rhhi-node-master-2-7954bf77f7-7v547        0/1     Terminating            0          85s
rook-ceph-drain-canary-rhhi-node-master-2-7956cb6959-p7mmh        0/1     Pending                0          0s
rook-ceph-drain-canary-rhhi-node-master-2-79d9bfd575-8vx6f        0/1     Terminating            0          107s
rook-ceph-drain-canary-rhhi-node-master-2-79f764d869-m2nhh        0/1     Terminating            0          40s
rook-ceph-drain-canary-rhhi-node-master-2-7b56f6c5c5-7bqt9        0/1     Terminating            0          95s
rook-ceph-drain-canary-rhhi-node-master-2-7b587cc78c-h85t2        0/1     Terminating            0          54s
rook-ceph-drain-canary-rhhi-node-master-2-7b69dc8d6f-grpsl        0/1     Terminating            0          26s
rook-ceph-drain-canary-rhhi-node-master-2-7c785cd8ff-cdzpb        0/1     Terminating            0          50s
rook-ceph-drain-canary-rhhi-node-master-2-7c94d754c-f94lv         0/1     Terminating            0          45s
rook-ceph-drain-canary-rhhi-node-master-2-7d6bb45b7-2hd26         0/1     Terminating            0          37s
rook-ceph-drain-canary-rhhi-node-master-2-7dcd598b5f-r6w2d        0/1     Terminating            0          5s
rook-ceph-drain-canary-rhhi-node-master-2-7f48f46c7f-kjhbc        0/1     Terminating            0          22s
rook-ceph-drain-canary-rhhi-node-master-2-7f87c96847-54dtn        0/1     Terminating            0          11s
rook-ceph-drain-canary-rhhi-node-master-2-8487db94f5-nf9d6        0/1     Terminating            0          25s
rook-ceph-drain-canary-rhhi-node-master-2-84d68646c-jscgf         0/1     Terminating            0          57s
rook-ceph-drain-canary-rhhi-node-master-2-857c7b445f-lmmjl        0/1     Terminating            0          19s
rook-ceph-drain-canary-rhhi-node-master-2-864479f9b7-pn9vm        0/1     Terminating            0          17s
rook-ceph-drain-canary-rhhi-node-master-2-8656b684d7-7bxgw        0/1     Terminating            0          33s
rook-ceph-drain-canary-rhhi-node-master-2-874b4c8df-cj5m6         0/1     Terminating            0          58s
rook-ceph-drain-canary-rhhi-node-master-2-8c896db79-5jnld         0/1     Terminating            0          27s
rook-ceph-drain-canary-rhhi-node-master-2-8dd6cb577-kfrfv         0/1     Terminating            0          48s
rook-ceph-drain-canary-rhhi-node-master-2-8df46487-mr5mc          0/1     Terminating            0          56s
rook-ceph-drain-canary-rhhi-node-master-2-9997c45f9-844x4         0/1     Terminating            0          10s
rook-ceph-drain-canary-rhhi-node-master-2-b5cd658b9-gfzj5         0/1     Terminating            0          3s
rook-ceph-drain-canary-rhhi-node-master-2-bb885c7c7-xqgkf         0/1     Terminating            0          4s
rook-ceph-drain-canary-rhhi-node-master-2-bf8957b87-v7s5f         0/1     Terminating            0          36s
rook-ceph-drain-canary-rhhi-node-master-2-c4d764c99-xnnjq         0/1     Terminating            0          97s
rook-ceph-drain-canary-rhhi-node-master-2-c67c7bbf5-znjkl         0/1     Terminating            0          43s
rook-ceph-drain-canary-rhhi-node-master-2-c6f6579c5-m87rl         0/1     Terminating            0          75s
rook-ceph-drain-canary-rhhi-node-master-2-cddc89b7-wbp8q          0/1     Terminating            0          55s
rook-ceph-drain-canary-rhhi-node-master-2-d4bb7ffdf-v2r5m         0/1     Terminating            0          92s
rook-ceph-drain-canary-rhhi-node-master-2-d7457b6ff-cprgt         0/1     Terminating            0          6s
rook-ceph-drain-canary-rhhi-node-master-2-df97c58cc-59m4p         0/1     Terminating            0          39s
rook-ceph-drain-canary-rhhi-node-master-2-f99fb4dc9-xwfdh         0/1     Terminating            0          53s
rook-ceph-drain-canary-rhhi-node-master-2-ff959c4c-sclsc          0/1     Terminating            0          113s
rook-ceph-mds-openshift-storage-cephfilesystem-a-74d4b6564lzhm9   1/1     Running                0          2m2s
rook-ceph-mds-openshift-storage-cephfilesystem-b-5786d78b9bd92c   1/1     Running                0          2m2s
rook-ceph-mgr-a-6c8ff5b7d7-wvft9                                  1/1     Running                0          3m40s
rook-ceph-mon-a-8479c9d474-lp4v8                                  1/1     Running                0          5m18s
rook-ceph-mon-b-6d45647947-ksgs4                                  1/1     Running                0          4m29s
rook-ceph-mon-c-7f655b56c6-8g68q                                  1/1     Running                0          4m6s
rook-ceph-operator-85d5fc66fb-zkvmg                               1/1     Running                0          7m54s
rook-ceph-osd-0-86475849ff-8kkg6                                  1/1     Running                0          2m34s
rook-ceph-osd-1-75b5f785cf-hn2kc                                  1/1     Running                0          2m37s
rook-ceph-osd-10-56d8df6d8f-4rxpv                                 1/1     Running                0          2m28s
rook-ceph-osd-11-67d5598748-xwzhn                                 1/1     Running                0          2m18s
rook-ceph-osd-2-d7bff954d-hpzsm                                   1/1     Running                0          2m29s
rook-ceph-osd-3-66b7db9c75-nk2bs                                  1/1     Running                0          2m34s
rook-ceph-osd-4-74556667f9-qv6ls                                  1/1     Running                0          2m34s
rook-ceph-osd-5-97c7647ff-8nlqc                                   1/1     Running                0          2m32s
rook-ceph-osd-6-7764dbf554-dldt8                                  1/1     Running                0          2m33s
rook-ceph-osd-7-65ccf47786-j84zw                                  1/1     Running                0          2m32s
rook-ceph-osd-8-7c5bb788cf-q9pbl                                  1/1     Running                0          2m24s
rook-ceph-osd-9-77fb4f8789-r48bg                                  1/1     Running                0          2m29s
rook-ceph-osd-prepare-osd-0-f5fs9-zzgww                           0/1     Completed              0          3m17s
rook-ceph-osd-prepare-osd-1-fnh8t-bhp2w                           0/1     Completed              0          3m16s
rook-ceph-osd-prepare-osd-10-tgk66-d69jb                          0/1     Completed              0          3m11s
rook-ceph-osd-prepare-osd-11-6vvvr-78thc                          0/1     Completed              0          3m11s
rook-ceph-osd-prepare-osd-2-ksxr6-5khjs                           0/1     Completed              0          3m16s
rook-ceph-osd-prepare-osd-3-vfxqn-rwl2t                           0/1     Completed              0          3m15s
rook-ceph-osd-prepare-osd-4-4w5fg-zh8mj                           0/1     Completed              0          3m15s
rook-ceph-osd-prepare-osd-5-hmlb4-6vbwg                           0/1     Completed              0          3m14s
rook-ceph-osd-prepare-osd-6-n4lcw-q2jzp                           0/1     Completed              0          3m13s
rook-ceph-osd-prepare-osd-7-w7t6p-hlqxd                           0/1     Completed              0          3m13s
rook-ceph-osd-prepare-osd-8-fxz4g-drhp2                           0/1     Completed              0          3m13s
rook-ceph-osd-prepare-osd-9-rlndb-sztwb                           0/1     Completed              0          3m12s
rook-ceph-tools-747987675d-ttqgv                                  0/1     CreateContainerError   0          7m11s

[mcornea]

[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
313
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
312
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
313
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
314
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
316
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
318
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
317
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
319
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
319
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
319
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
320
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
320
[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage get pods | grep Terminating | wc -l
340

[mcornea]

[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage describe pods/rook-ceph-drain-canary-rhhi-node-master-0-545fff7849-bgrzv
Name:                      rook-ceph-drain-canary-rhhi-node-master-0-545fff7849-bgrzv
Namespace:                 openshift-storage
Priority:                  0
PriorityClassName:         <none>
Node:                      rhhi-node-master-0/
Labels:                    app=rook-ceph-drain-canary
                           kubernetes.io/hostname=rhhi-node-master-0
                           node_name=rhhi-node-master-0
                           pod-template-hash=545fff7849
Annotations:               openshift.io/scc: restricted
Status:                    Terminating (lasts 75s)
Termination Grace Period:  30s
IP:                        
Controlled By:             ReplicaSet/rook-ceph-drain-canary-rhhi-node-master-0-545fff7849
Containers:
  busybox:
    Image:      busybox
    Port:       <none>
    Host Port:  <none>
    Command:
      bin/sh
    Args:
      -c
      sleep infinity
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zgjmn (ro)
Conditions:
  Type           Status
  PodScheduled   True 
Volumes:
  default-token-zgjmn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zgjmn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/hostname=rhhi-node-master-0
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
                 node.ocs.openshift.io/storage=true:NoSchedule
Events:
  Type     Reason                  Age   From                         Message
  ----     ------                  ----  ----                         -------
  Normal   Scheduled               106s  default-scheduler            Successfully assigned openshift-storage/rook-ceph-drain-canary-rhhi-node-master-0-545fff7849-bgrzv to rhhi-node-master-0
  Warning  FailedCreatePodSandBox  92s   kubelet, rhhi-node-master-0  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_rook-ceph-drain-canary-rhhi-node-master-0-545fff7849-bgrzv_openshift-storage_86c8ee36-df0e-11e9-88fb-525400322df2_0(e73f15d4f5b3957e00c13d9140454c1551d618f482b4134df68e8be01b723905): Multus: Err adding pod to network "openshift-sdn": cannot set "openshift-sdn" ifname to "eth0": no netns: failed to Statfs "/proc/111588/ns/net": no such file or directory

[mcornea]

rook-ceph-tools-747987675d-ttqgv pod is in CreateContainerError state:

[kni@rhhi-node-worker-0 install-scripts]$ oc -n openshift-storage describe pods/rook-ceph-tools-747987675d-ttqgv
Name:               rook-ceph-tools-747987675d-ttqgv
Namespace:          openshift-storage
Priority:           0
PriorityClassName:  <none>
Node:               rhhi-node-master-2/192.168.123.111
Start Time:         Tue, 24 Sep 2019 16:52:25 -0400
Labels:             app=rook-ceph-tools
                    pod-template-hash=747987675d
Annotations:        k8s.v1.cni.cncf.io/networks-status:
                      [{
                          "name": "openshift-sdn",
                          "interface": "eth0",
                          "ips": [
                              "10.129.0.78"
                          ],
                          "default": true,
                          "dns": {}
                      }]
                    openshift.io/scc: rook-ceph
Status:             Pending
IP:                 10.129.0.78
Controlled By:      ReplicaSet/rook-ceph-tools-747987675d
Containers:
  rook-ceph-tools:
    Container ID:  
    Image:         ceph/ceph:v14.2.4-20190917
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /tini
    Args:
      -g
      --
      /usr/local/bin/toolbox.sh
    State:          Waiting
      Reason:       CreateContainerError
    Ready:          False
    Restart Count:  0
    Environment:
      ROOK_ADMIN_SECRET:  <set to the key 'admin-secret' in secret 'rook-ceph-mon'>  Optional: false
    Mounts:
      /dev from dev (rw)
      /etc/rook from mon-endpoint-volume (rw)
      /lib/modules from libmodules (rw)
      /sys/bus from sysbus (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-zgjmn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  dev:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  
  sysbus:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/bus
    HostPathType:  
  libmodules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  mon-endpoint-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      rook-ceph-mon-endpoints
    Optional:  false
  default-token-zgjmn:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-zgjmn
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason       Age                From                         Message
  ----     ------       ----               ----                         -------
  Normal   Scheduled    13m                default-scheduler            Successfully assigned openshift-storage/rook-ceph-tools-747987675d-ttqgv to rhhi-node-master-2
  Warning  FailedMount  13m (x7 over 13m)  kubelet, rhhi-node-master-2  MountVolume.SetUp failed for volume "mon-endpoint-volume" : configmaps "rook-ceph-mon-endpoints" not found
  Normal   Pulling      12m                kubelet, rhhi-node-master-2  Pulling image "ceph/ceph:v14.2.4-20190917"
  Normal   Pulled       12m                kubelet, rhhi-node-master-2  Successfully pulled image "ceph/ceph:v14.2.4-20190917"
  Warning  Failed       11m (x8 over 12m)  kubelet, rhhi-node-master-2  Error: container create failed: container_linux.go:345: starting container process caused "exec: \"/tini\": stat /tini: no such file or directory"
  Normal   Pulled       4m (x38 over 12m)  kubelet, rhhi-node-master-2  Container image "ceph/ceph:v14.2.4-20190917" already present on machine