Route Laucher not found on PPC64LE

[sarveshtamba]

I am following the steps at:-
https://github.com/fabric8-launcher/launcher-operator

Facing the following error at step 4:-

[root@p006vm32 launcher-operator]# oc get route launcher --template={{.spec.host}}
Error from server (NotFound): routes.route.openshift.io "launcher" not found

Investigated the same and found the following:-

[root@p006vm32 launcher-operator]# oc get all
NAME                                     READY     STATUS    RESTARTS   AGE
pod/launcher-operator-7997df99b5-8hdrw   0/1       Error     0          12s

NAME                                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/launcher-operator   1         1         1            0           16s

NAME                                           DESIRED   CURRENT   READY     AGE
replicaset.apps/launcher-operator-7997df99b5   1         1         0         15s
[root@p006vm32 launcher-operator]# oc get all
NAME                                     READY     STATUS             RESTARTS   AGE
pod/launcher-operator-7997df99b5-8hdrw   0/1       CrashLoopBackOff   1          16s

NAME                                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/launcher-operator   1         1         1            0           20s

NAME                                           DESIRED   CURRENT   READY     AGE
replicaset.apps/launcher-operator-7997df99b5   1         1         0         19s

[root@p006vm32 launcher-operator]# oc logs launcher-operator-7997df99b5-8hdrw
standard_init_linux.go:178: exec user process caused "exec format error"

[root@p006vm32 launcher-operator]# oc describe pod/launcher-operator-7997df99b5-8hdrw
Name:               launcher-operator-7997df99b5-8hdrw
Namespace:          launcher-operator
Priority:           0
PriorityClassName:  <none>
Node:               p006vm32/129.40.81.32
Start Time:         Thu, 29 Aug 2019 06:23:01 -0400
Labels:             name=launcher-operator
                    pod-template-hash=3553895561
Annotations:        openshift.io/scc=anyuid
Status:             Running
IP:                 10.128.0.135
Controlled By:      ReplicaSet/launcher-operator-7997df99b5
Containers:
  launcher-operator:
    Container ID:  docker://a2656ec83e7eb608b198ba5aca0c1c50e2921a873650060c35eef684be83b23f
    Image:         fabric8/launcher-operator:latest
    Image ID:      docker-pullable://docker.io/fabric8/launcher-operator@sha256:31142ae9b981abc900e165b6b8aed62ba718c1ff1292291254c6c2bec6fb2864
    Port:          60000/TCP
    Host Port:     0/TCP
    Command:
      launcher-operator
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 29 Aug 2019 06:34:51 -0400
      Finished:     Thu, 29 Aug 2019 06:34:51 -0400
    Ready:          False
    Restart Count:  7
    Readiness:      exec [stat /tmp/operator-sdk-ready] delay=4s timeout=1s period=10s #success=1 #failure=1
    Environment:
      WATCH_NAMESPACE:  launcher-operator (v1:metadata.namespace)
      POD_NAME:         launcher-operator-7997df99b5-8hdrw (v1:metadata.name)
      OPERATOR_NAME:    launcher-operator
      TEMPLATE_PATH:    /usr/local/bin/templates
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from launcher-operator-token-x6p42 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  launcher-operator-token-x6p42:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  launcher-operator-token-x6p42
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     <none>
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  12m                default-scheduler  Successfully assigned launcher-operator/launcher-operator-7997df99b5-8hdrw to p006vm32
  Normal   Created    11m (x4 over 12m)  kubelet, p006vm32  Created container
  Normal   Started    11m (x4 over 12m)  kubelet, p006vm32  Started container
  Normal   Pulling    10m (x5 over 12m)  kubelet, p006vm32  pulling image "fabric8/launcher-operator:latest"
  Normal   Pulled     10m (x5 over 12m)  kubelet, p006vm32  Successfully pulled image "fabric8/launcher-operator:latest"
  Warning  BackOff    2m (x45 over 12m)  kubelet, p006vm32  Back-off restarting failed container
[root@p006vm32 launcher-operator]#

[sarveshtamba]

Hi, any help on this one would be greatly appreciated. Thanks in advance.

[sarveshtamba]

@ia3andy are you the right person for this issue?
For fabric8/launcher-operator deployment I am facing "CrashLoopBackOff" for the pods.

In the file "deploy/operator.yaml" the image used is as below:-
image: fabric8/launcher-operator:latest
However, the latest fabric8/launcher-operator image has ARCHITECTURE=amd64 and OS=linux(https://hub.docker.com/r/fabric8/launcher-operator/tags), so we might need to build the image for ppc64le first and then continue the deployment using the built image.

Can you please confirm if this is the correct understanding?

[sarveshtamba]

I tried to build the image for ppc64le using steps in Develop section and it got built successfully with the operator-sdk build command.

[root@p006vm32 launcher-operator]# operator-sdk build fabric8/launcher-operator:9.1.2
INFO[0029] Building OCI image fabric8/launcher-operator:9.1.2
Sending build context to Docker daemon 183.2 MB
Step 1/5 : FROM alpine:3.8
Trying to pull repository registry.redhat.io/alpine ...
Trying to pull repository docker.io/library/alpine ...
3.8: Pulling from docker.io/library/alpine
0c6396bb25a4: Pull complete
Digest: sha256:04696b491e0cc3c58a75bace8941c14c924b9f313b03ce5029ebbc040ed9dcd9
Status: Downloaded newer image for docker.io/alpine:3.8
 ---> 18052731e50a
Step 2/5 : RUN apk upgrade --update --no-cache
 ---> Running in 846ac8e4d77c

fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/ppc64le/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/ppc64le/APKINDEX.tar.gz
(1/2) Upgrading musl (1.1.19-r10 -> 1.1.19-r11)
(2/2) Upgrading musl-utils (1.1.19-r10 -> 1.1.19-r11)
Executing busybox-1.28.4-r3.trigger
OK: 5 MiB in 13 packages
 ---> 7216824410f3
Removing intermediate container 846ac8e4d77c
Step 3/5 : USER nobody
 ---> Running in 5381649be74c
 ---> c8052cd77c4e
Removing intermediate container 5381649be74c
Step 4/5 : ADD templates/*.yaml /usr/local/bin/templates/
 ---> 509ce5fd7e1e
Removing intermediate container a7e734a74387
Step 5/5 : ADD build/_output/bin/launcher-operator /usr/local/bin/launcher-operator
 ---> 32c8c0c50ded
Removing intermediate container fce538cd38f4
Successfully built 32c8c0c50ded
INFO[0058] Operator build complete.
[root@p006vm32 launcher-operator]# docker images | grep fabric8
fabric8/launcher-operator                                                      9.1.2                                                              32c8c0c50ded        40 seconds ago      48.7 MB
[root@p006vm32 launcher-operator]# oc get all
NAME                               AGE
task.tekton.dev/echo-hello-world   5d

NAME                                           SUCCEEDED   REASON           STARTTIME   COMPLETIONTIME
taskrun.tekton.dev/echo-hello-world-task-run   False       TaskRunTimeout   5d          5d
[root@p006vm32 launcher-operator]# vi ./deploy/operator.yaml
[root@p006vm32 launcher-operator]# docker inspect fabric8/launcher-operator:9.1.2
[
    {
        "Id": "sha256:32c8c0c50dedc9a1b6dd46356c16d72f07995ad7ff517757e2095cfe0b5cc2be",
        "RepoTags": [
            "fabric8/launcher-operator:9.1.2"
        ],
        "RepoDigests": [],
        "Parent": "sha256:509ce5fd7e1e0d458d358237b2f7669db7d7ca580b23707011b6235f918a159e",
        "Comment": "",
        "Created": "2019-09-17T08:50:00.142287463Z",
        "Container": "fce538cd38f476125626cf5c72694dd2a3f596033c9b1b064bad731f18595f18",
        "ContainerConfig": {
            "Hostname": "846ac8e4d77c",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh",
                "-c",
                "#(nop) ADD file:30fe9b1b4f59428969888e33d2331e7501243691f08027c5152eee27b36e8eae in /usr/local/bin/launcher-operator "
            ],
            "ArgsEscaped": true,
            "Image": "sha256:509ce5fd7e1e0d458d358237b2f7669db7d7ca580b23707011b6235f918a159e",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": [],
            "Labels": {}
        },
        "DockerVersion": "1.13.1",
        "Author": "",
        "Config": {
            "Hostname": "846ac8e4d77c",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "/bin/sh"
            ],
            "ArgsEscaped": true,
            "Image": "sha256:509ce5fd7e1e0d458d358237b2f7669db7d7ca580b23707011b6235f918a159e",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": [],
            "Labels": {}
        },
        "Architecture": "ppc64le",
        "Os": "linux",
        "Size": 48669181,
        "VirtualSize": 48669181,
        "GraphDriver": {
            "Name": "overlay2",
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/98126c7461e113722f9ddb7937416b715ce5d339ef46cfb071ae2970dab871c1/diff:/var/lib/docker/overlay2/a37e26a1d02dabfc9e400bedd39bda8296bdaaf4fec6270f6146c6af88131891/diff:/var/lib/docker/overlay2/ad6c31ab9142e321fd07ca21dd0979a643ca776adc041be509916e23a126fef4/diff",
                "MergedDir": "/var/lib/docker/overlay2/003c5dcf1bfc0a20781157a21fe66c3d81022760749671c4f053d984685c493d/merged",
                "UpperDir": "/var/lib/docker/overlay2/003c5dcf1bfc0a20781157a21fe66c3d81022760749671c4f053d984685c493d/diff",
                "WorkDir": "/var/lib/docker/overlay2/003c5dcf1bfc0a20781157a21fe66c3d81022760749671c4f053d984685c493d/work"
            }
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:c90acc73c11e8a38a35855cd566e904cf7b13d4a3958e6ada2e85529e9fadba4",
                "sha256:70cfd04ab5777e8625ad2543d3870e6610ad36a89313bf3bd2faa986663015fd",
                "sha256:931bb96b156355889a233194eb0b20383e54ab1d57841329d5c8161afeb70a5e",
                "sha256:39e9463790e6b8bce6c766c35d196a6fcaa2f93d2d8fcced4f15789bd9265680"
            ]
        }
    }
]
[root@p006vm32 launcher-operator]# oc create -R -f ./deploy
customresourcedefinition.apiextensions.k8s.io/launchers.launcher.fabric8.io created
deployment.apps/launcher-operator created
role.rbac.authorization.k8s.io/launcher-operator created
rolebinding.rbac.authorization.k8s.io/launcher-operator created
serviceaccount/launcher-operator created
[root@p006vm32 launcher-operator]# oc get all
NAME                                     READY     STATUS             RESTARTS   AGE
pod/launcher-operator-65564cd544-xmppd   0/1       CrashLoopBackOff   2          43s

NAME                                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/launcher-operator   1         1         1            0           44s

NAME                                           DESIRED   CURRENT   READY     AGE
replicaset.apps/launcher-operator-65564cd544   1         1         0         44s

NAME                               AGE
task.tekton.dev/echo-hello-world   5d

NAME                                           SUCCEEDED   REASON           STARTTIME   COMPLETIONTIME
taskrun.tekton.dev/echo-hello-world-task-run   False       TaskRunTimeout   5d          5d
[root@p006vm32 launcher-operator]# kubectl describe pod launcher-operator-65564cd544-xmppd
Name:               launcher-operator-65564cd544-xmppd
Namespace:          launcher-operator
Priority:           0
PriorityClassName:  <none>
Node:               p006vm32/129.40.81.32
Start Time:         Tue, 17 Sep 2019 05:02:21 -0400
Labels:             name=launcher-operator
                    pod-template-hash=2112078100
Annotations:        openshift.io/scc=anyuid
Status:             Running
IP:                 10.128.1.79
Controlled By:      ReplicaSet/launcher-operator-65564cd544
Containers:
  launcher-operator:
    Container ID:  docker://76415e9d2424fe4e3d9522c32c082f058177bb46af88a04e06c4b230e2f59512
    Image:         fabric8/launcher-operator:9.1.2
    Image ID:      docker://sha256:32c8c0c50dedc9a1b6dd46356c16d72f07995ad7ff517757e2095cfe0b5cc2be
    Port:          60000/TCP
    Host Port:     0/TCP
    Command:
      launcher-operator
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 17 Sep 2019 05:02:47 -0400
      Finished:     Tue, 17 Sep 2019 05:02:47 -0400
    Ready:          False
    Restart Count:  2
    Readiness:      exec [stat /tmp/operator-sdk-ready] delay=4s timeout=1s period=10s #success=1 #failure=1
    Environment:
      WATCH_NAMESPACE:  launcher-operator (v1:metadata.namespace)
      POD_NAME:         launcher-operator-65564cd544-xmppd (v1:metadata.name)
      OPERATOR_NAME:    launcher-operator
      TEMPLATE_PATH:    /usr/local/bin/templates
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from launcher-operator-token-jb27l (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  launcher-operator-token-jb27l:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  launcher-operator-token-jb27l
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  node-role.kubernetes.io/compute=true
Tolerations:     <none>
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  55s                default-scheduler  Successfully assigned launcher-operator/launcher-operator-65564cd544-xmppd to p006vm32
  Normal   Started    29s (x3 over 49s)  kubelet, p006vm32  Started container
  Warning  BackOff    15s (x4 over 46s)  kubelet, p006vm32  Back-off restarting failed container
  Normal   Pulled     2s (x4 over 51s)   kubelet, p006vm32  Container image "fabric8/launcher-operator:9.1.2" already present on machine
  Normal   Created    2s (x4 over 50s)   kubelet, p006vm32  Created container

However, the pod is still in "CrashLoopBackOff". On investigating found that the build/_output/bin/launcher-operator gets generated for x86_64 while the build/_output/bin/launcher-operator-local is for ppc64le.

[root@p006vm32 launcher-operator]# file build/_output/bin/launcher-operator*
build/_output/bin/launcher-operator:       ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
build/_output/bin/launcher-operator-local: ELF 64-bit LSB executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=2d72ea8c8d958d1865d7c45ea3a180cbad9c0ec1, not stripped

Investigating further on this.

[sarveshtamba]

• Resolved the issue wherein 'launcher-operator' binary used in the image gets generated for x86_64 by default, by setting GOARCH variable for ppc64le arch as below:-
  export GOARCH=ppc64le

• Cleaned up and built the image successfully with the operator-sdk build command, post which the Launcher operator got deployed successfully.

[root@p006vm32 launcher-operator]# oc get all -n launcher-operator
NAME                                     READY     STATUS    RESTARTS   AGE
pod/launcher-operator-65564cd544-swdh7   1/1       Running   0          2h

NAME                                DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/launcher-operator   1         1         1            1           2h

NAME                                           DESIRED   CURRENT   READY     AGE
replicaset.apps/launcher-operator-65564cd544   1         1         1         2h

• Currently working on step to Install the Launcher (via the installed operator). Facing issues with the example launcher yaml which makes use of image: fabric8/launcher-backend:latest, which for now is available for x86_64.