How to use shell in extension container

[b1r3k]

I'm trying to create extension for [netclient](https://github.com/gravitl/netclient). I have taken network/tailscale extension as template which resulted in following yamls:

Here is pkg.yaml:

name: netmaker-client
variant: alpine
shell: /bin/sh
dependencies:
  - stage: base
    runtime: true
    from: /
    to: /
install:
  - bash
  - musl-dev
  - libmnl
  - gcompat
  - openresolv
  - openrc
steps:
  - env:
      GOPATH: /go
      CGO_ENABLED: 0
      GOOS: linux
  - cachePaths:
      - /.cache/go-build
      - /go/pkg
    sources:
      - url: https://github.com/gravitl/netclient/archive/refs/tags/v{{ .NETMAKER_VERSION }}.tar.gz
        destination: netmaker-client.tar.gz
        sha256: e86c25bca0dda02eb5207a64d7251017faeb9c00a45c4300237956aa5a6a7815
        sha512: c4649e7ac34c31d18c2bf0a3dd0e59b9d8fc4f521e2095c496090bfe3aef929e85e19f4b5592933b8b28c052a38f1e6124d8f01d75547e9d58d1d0173b3e6346
    prepare:
      - |
        sed -i 's#$VERSION#{{ .VERSION }}#' /pkg/manifest.yaml

      - |
        mkdir netmaker-client dist
        tar -xzvf netmaker-client.tar.gz --strip-components=1 -C netmaker-client
    build:
      - |
        export PATH=${PATH}:${TOOLCHAIN}/go/bin

        go build \
          -C netmaker-client \
          -o ../dist/netclient \
        	-ldflags "-s -w \
                      -X netmaker-client.com/version.shortStamp={{ .NETMAKER_VERSION }} \
         			  -X netmaker-client.com/version.longStamp={{ .NETMAKER_VERSION }}-TalosLinux" .
    install:
      - |
        mkdir -p /rootfs/usr/local/lib/containers/netmaker-client/usr/local/bin/
        mkdir -p /rootfs/run/openrc
        mkdir -p /rootfs/usr/local/lib/containers/netmaker-client/etc/netclient
        cp -pr dist/netclient /rootfs/usr/local/lib/containers/netmaker-client/usr/local/bin
        cp -pr /toolchain/bin/bash /rootfs/usr/local/lib/containers/netmaker-client/usr/local/bin
      - |
        cp /pkg/netclient.sh /rootfs/usr/local/lib/containers/netmaker-client/usr/local/bin
        chmod 0755 /rootfs/usr/local/lib/containers/netmaker-client/usr/local/bin/netclient.sh
      - |
        mkdir -p /rootfs/usr/local/etc/containers
        cp /pkg/netclient.yaml /rootfs/usr/local/etc/containers/
    test:
      - |
        mkdir -p /extensions-validator-rootfs
        cp -r /rootfs/ /extensions-validator-rootfs/rootfs
        cp /pkg/manifest.yaml /extensions-validator-rootfs/manifest.yaml
        /extensions-validator validate --rootfs=/extensions-validator-rootfs --pkg-name="${PKG_NAME}"
finalize:
  - from: /rootfs
    to: /rootfs
  - from: /pkg/manifest.yaml
    to: /

netclient.yaml

name: netmaker-client
depends:
  - service: cri
  - network:
    - addresses
    - connectivity
    - etcfiles
  - configuration: true
container:
  entrypoint: /usr/local/bin/bash
  args:
  - /usr/local/bin/netclient.sh
  environment:
  - PATH=/bin:/sbin:/usr/local/bin
  security:
    writeableRootfs: true
    writeableSysfs: true
  mounts:
  # libs
  - source: /lib
    destination: /lib
    type: bind
    options:
      - bind
      - ro
  # more libs
  - source: /usr/lib
    destination: /usr/lib
    type: bind
    options:
      - bind
      - ro
  ## Required for tailscale. Ip addr and other commands
  - source: /sbin
    destination: /sbin
    type: bind
    options:
      - bind
      - ro
  ## Tailscale needs to write to this to create the interfaces
  - source: /dev/net/tun
    destination: /dev/net/tun
    type: bind
    options:
      - bind
      - rw
  ## netclient writes logs to file
  - source: /var/log
    destination: /var/log
    type: bind
    options:
      - bind
      - rw
restart: always
logToConsole: true

Everything seems to work: build, upgrade etc. except extension startup:

$ talosctl -n 192.168.0.187 upgrade --image docker.io/ljachym/talos-metal-installer:v1.9.2@sha256:83894a4c9375a465e1780d9482d528f3e4df22654f68e65ba71424a838dc75e9
WARNING: 192.168.0.187: server version 1.9.2 is older than client version 1.9.3
watching nodes: [192.168.0.187]
    * 192.168.0.187: post check passed
isolated@13cf64a34911:~$ talosctl -n 192.168.0.187 service
NODE            SERVICE               STATE     HEALTH   LAST CHANGE   LAST EVENT
192.168.0.187   apid                  Running   OK       17s ago       Health check successful
192.168.0.187   auditd                Running   OK       32s ago       Health check successful
192.168.0.187   containerd            Running   OK       32s ago       Health check successful
192.168.0.187   cri                   Running   OK       23s ago       Health check successful
192.168.0.187   dashboard             Running   ?        25s ago       Process Process(["/sbin/dashboard"]) started with PID 1998
192.168.0.187   ext-netmaker-client   Waiting   ?        1s ago        Error running Containerd(ext-netmaker-client), going to restart forever: task "ext-netmaker-client" failed: exit code 255
192.168.0.187   kubelet               Running   OK       20s ago       Health check successful
192.168.0.187   machined              Running   OK       32s ago       Health check successful
192.168.0.187   syslogd               Running   OK       31s ago       Health check successful
192.168.0.187   udevd                 Running   OK       26s ago       Health check successful
isolated@13cf64a34911:~$ talosctl -n 192.168.0.187 logs ext-netmaker-client
192.168.0.187: exec /usr/local/bin/bash: no such file or directory
192.168.0.187: exec /usr/local/bin/bash: no such file or directory
192.168.0.187: exec /usr/local/bin/bash: no such file or directory
192.168.0.187: exec /usr/local/bin/bash: no such file or directory
192.168.0.187: exec /usr/local/bin/bash: no such file or directory
192.168.0.187: exec /usr/local/bin/bash: no such file or directory

So it seems that /usr/local/bin/bash is not available inside extension container although I'm copying it during install phase of pkg.yaml. Can anybody point me to solution here? How to use bash inside of extension container properly? Tailscale does not have this issue because they have [containerboot command - tailscale.com/cmd/containerboot - Go Packages](https://pkg.go.dev/tailscale.com@v1.40.1/cmd/containerboot) which unfortunately is not the case for netclient, see their dockerfile: [netclient/Dockerfile at develop · gravitl/netclient](https://github.com/gravitl/netclient/blob/develop/Dockerfile) and bash wrapper used to launch it: https://github.com/gravitl/netclient/blob/develop/scripts/netclient.sh

Here is code for adding netclient extension: https://github.com/b1r3k/extensions/tree/adds-netmaker-netclient

[smira]

One option is to rework skipping bash.

But your issue is that you copy bash binary, but libraries (libc, libmusl in particular) is not there.

Your container image should work on its own if you copy out the root of it out. In general, if you have a container image, you can run it as is if you copy it to the expected path.

[smira]

In other words, if you COPY alpine image to /usr/local/... (expected path of the extension service rootfs), it should just work out of the box.

Same way you can test your image without Talos by copying it back to / and doing docker run.

[b1r3k]

@smira I appreciate prompt reply, got me a while to get back to this problem..

so I tried to do that as you can see in this commit: b1r3k@6e6fe29 Basically I'm trying to leverage alpine building system to build bash from sources. Once that works I'll switch flags to get static binary.

but for now basic build step defined in network/netmaker-client/pkg.yaml fails:

 => ERROR netmaker-client:build-1                                                                                 0.1s
------
 > netmaker-client:build-1:
0.059 /bin/bash: /usr/bin/abuild: /bin/ash: bad interpreter: Symbolic link loop
------
ERROR: failed to solve: process "/bin/bash -c set -eou pipefail\nabuild fetch" did not complete successfully: exit code: 126
make[1]: *** [Makefile:170: target-netmaker-client] Error 1
make[1]: Leaving directory '/home/lukasz/workspace/open-source/siderolabs-extensions'
make: *** [Makefile:176: docker-netmaker-client] Error 2

Apparently, abuild wants to use ash but this leads to Symbolic link loop. Ok, so it seems I'm not operating on vanilla alpine image but on some kind of mod. Can you point me to right resources about it? I'd like to understand why I'm getting Symbolic link loop otherwise I'm wandering in the dark here..

[smira]

I don't have any exact idea, we never tried to build with aports.

Do you actually need bash or you can get away with busybox's sh?