runner deployment is not working with any docker image except summerwind/actions-runner:latest

[williamsunctp]

Hello Team,

I am not able to run RunnerDeployment with any docker image other than the original summerwind/actions-runner:latest. on AWS EKS 1.22.

When the deployment with the following configuration, it works

  env:
    - name: DISABLE_RUNNER_UPDATE # Disables automatic runner updates
      value: "true"
  image: summerwind/actions-runner:latest

But it does not have aws command, kubectl command etc. And it does not recognize the GITHUB_TOKEN to pull the private github repo.

docker build --build-arg github_token=$GITHUB_TOKEN -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .

So I plan to add some more package to the original summerwind/actions-runner:latest image.
but none of them are working. Even docker image built with one line docker file

FROM summerwind/actions-runner:latest

It is still not working. Can you please point me in the right direction?

Thank you very much!

[williamsunctp]

Add some more context. GitHub action workflows are working fine under VM self-hosted runner. I would like to try the actions-runner-controller in EKS so I do not have to setup VM runner in the future. The same issue happens with https://github.com/actions/actions-runner-controller/releases/download/v0.26.0/actions-runner-controller.yaml and https://github.com/actions/actions-runner-controller/releases/download/v0.22.0/actions-runner-controller.yaml. Thank you very much in advance for your help!

[kevholmes]

Hello Team,

I am not able to run RunnerDeployment with any docker image other than the original summerwind/actions-runner:latest. on AWS EKS 1.22.

When the deployment with the following configuration, it works

  env:
    - name: DISABLE_RUNNER_UPDATE # Disables automatic runner updates
      value: "true"
  image: summerwind/actions-runner:latest

But it does not have aws command, kubectl command etc. And it does not recognize the GITHUB_TOKEN to pull the private github repo.

The ARC team intentionally keeps their base runner images as scaled-back as possible, which is a good thing. I would suggest setting up your own repo + Action to build these Docker images you'll be using on a regular cadence, test them in some basic way, and then push them to your registry of choice if the image works as expected. This will also eliminate issues with Dockerhub rate limiting concerns as you will not be having your runners pull :latest every time for the summerwind images.

The only token you should need to provide to ARC is the one giving it permissions to manage runners depending on how you scope things in your setup (enterprise-wide, org-wide, or individual repo.) During the kickoff process for a runner, the ARC stack will be given an ephemeral token scoped to the repo your Action workflow will run within and that is handed off to the runner to then utilize when pulling down code and possibly pushing changes back in if you allow that kind of thing.

docker build --build-arg github_token=$GITHUB_TOKEN -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .

So I plan to add some more package to the original summerwind/actions-runner:latest image. but none of them are working. Even docker image built with one line docker file

FROM summerwind/actions-runner:latest

It is still not working. Can you please point me in the right direction?

This just sounds like an environment error if you cannot even pull in the summerwind image? Does your organization block dockerhub access via network firewall rules or policies?

Thank you very much!

[williamsun-hha]

Thank you very much for your help, Kevin!

I am able to pull the summerwind image and run with it. k8s runs ok with it but the image does not have the package I want.

So I try to build a custom image from summerwind/actions-runner:latest with one line in the docker file
FROM summerwind/actions-runner:latest

The new docker image is not run. I can see k8s was able to pull the new docker image successfully from the log. but it is errored out with following error

exec /bin/bash: exec format error

When I clone the actions-runner-controller repo

and try to build the docker image

docker build . -t action.orig

I got this error

Thank you very much for your help and guidance.

" => ERROR [builder 5/5] RUN --mount=target=. --mount=type=cache,mode=0777,target=${GOCACHE} export GOOS=${TARGETOS} GOARCH=${TARGETARCH} GOARM=${TARGE 0.1s
"

full message below.

[+] Building 1.9s (13/16)
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 2.18kB 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 157B 0.0s
=> [internal] load metadata for gcr.io/distroless/static:nonroot 0.4s
=> [internal] load metadata for docker.io/library/golang:1.19.4 1.7s
=> [auth] library/golang:pull token for registry-1.docker.io 0.0s
=> CACHED [stage-1 1/5] FROM gcr.io/distroless/static:nonroot@sha256:1fa522fe6cfe020d50341f1ca561c099487bd44f8eb98d25d1920b07e05e40be 0.0s
=> [builder 1/5] FROM docker.io/library/golang:1.19.4@sha256:660f138b4477001d65324a51fa158c1b868651b44e43f0953bf062e9f38b72f3 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 12.73kB 0.0s
=> CACHED [internal] settings cache mount permissions 0.0s
=> CACHED [builder 2/5] WORKDIR /workspace 0.0s
=> CACHED [builder 3/5] COPY go.mod go.sum ./ 0.0s
=> CACHED [builder 4/5] RUN go mod download 0.0s
=> ERROR [builder 5/5] RUN --mount=target=. --mount=type=cache,mode=0777,target=${GOCACHE} export GOOS=${TARGETOS} GOARCH=${TARGETARCH} GOARM=${TARGE 0.1s

[builder 5/5] RUN --mount=target=. --mount=type=cache,mode=0777,target=${GOCACHE} export GOOS=${TARGETOS} GOARCH=${TARGETARCH} GOARM=${TARGETVARIANT#v} && go build -trimpath -ldflags="-s -w -X 'github.com/actions/actions-runner-controller/build.Version=${VERSION}'" -o /out/manager main.go && go build -trimpath -ldflags="-s -w" -o /out/github-webhook-server ./cmd/githubwebhookserver && go build -trimpath -ldflags="-s -w" -o /out/actions-metrics-server ./cmd/actionsmetricsserver:
#13 0.065 runc run failed: unable to start container process: error during container init: error mounting "/var/lib/docker/overlay2/o0cusirz4cbplwbaxxnam69pf/merged/cache" to rootfs at "/workspace/${GOCACHE}": mkdir /var/lib/docker/buildkit/executor/xi2q1vhkf3rpts7mmeenaeacn/rootfs/workspace/${GOCACHE}: read-only file system

---

executor failed running [/bin/sh -c export GOOS=${TARGETOS} GOARCH=${TARGETARCH} GOARM=${TARGETVARIANT#v} && go build -trimpath -ldflags="-s -w -X 'github.com/actions/actions-runner-controller/build.Version=${VERSION}'" -o /out/manager main.go && go build -trimpath -ldflags="-s -w" -o /out/github-webhook-server ./cmd/githubwebhookserver && go build -trimpath -ldflags="-s -w" -o /out/actions-metrics-server ./cmd/actionsmetricsserver]: exit code: 1

[williamsun-hha]

Also, when I pull the docker form docker hub

docker pull summerwind/actions-runner:latest

and pushed the image to ECR,

The image does not work either.

Thanks

Events:
Type Reason Age From Message

---

Normal Scheduled 29s default-scheduler Successfully assigned actions-runner-system/example-runnerdeploy-d2l6z-qg6dv to ip-10-3-11-69.ec2.internal
Normal Pulling 29s kubelet Pulling image "xxx.dkr.ecr.us-east-1.amazonaws.com/actions-runner:summerwind"
Normal Pulled 29s kubelet Successfully pulled image "xxx.dkr.ecr.us-east-1.amazonaws.com/actions-runner:summerwind" in 227.903171ms
Normal Created 29s kubelet Created container runner
Normal Started 28s kubelet Started container runner
Normal Pulled 28s kubelet Container image "docker:dind" already present on machine
Normal Created 28s kubelet Created container docker
Normal Started 28s kubelet Started container docker
Normal Killing 24s kubelet Stopping container docker

[kevholmes]

Thanks for the info @williamsun-hha. I think part of the issue here is that if we try to build a new image using something like what you mentioned it will not work:

FROM summerwind/actions-runner:latest

...

If you are going to be building your own custom image for runners it will likely require a different approach.

My approach was to vendor the https://github.com/actions/actions-runner-controller/tree/master/runner part of this project into a private repo as /runner-${releaseSemver}. This repo with runner configs/deployments has an Action that will then build the ARC runner dockerfiles from source daily (to get the latest updates/fixes) and we have made some small modifications to add in any tooling we'd like in our base image. You could then use gorm or something to test the image's basic functionality (adding in kubectl for instance) before pushing/updating tags for the runner images in your RunnerDeployment resources.

[williamsun-hha]

Thank you very much for your help, Kevin!

I just used an EKS 1.23 cluster, everything is working fine now. All of the process are working fine now.

Thanks