Pods getting Evicted during a GitHub action run

[orbi-ci-bot]

Randomly I see some of the pods getting evicted(happens to atleast 2 pods per day). I have set resource limits and Priority class as well but still they were getting evicted.

Updated the runners to v2.296.2-ubuntu-20.04 version two days ago to see if it fixes the issue.

Currently the runnerdeployment file looks like this:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: runner-deploy
  namespace: actions-runner

spec:
  replicas: 4
  template:
    spec:
      organization: eka-care
      image: ekacare/actions-runner:v7
      labels:
        - runner-server
      resources:
      limits:
        cpu: "1.0"
        memory: "4Gi"
      requests:
        cpu: "1.0"
        memory: "4Gi" 

I had removed priority class after the update since it made zero sense having it anyway. Tried increasing the memory and CPU as well which also didnt fix the issue. Need help as we have to re-run our Workflows which are causing a big issue in some GitHub CD's.

[orbi-ci-bot]

The custom image is installing docker-compose and aws cli.

FROM summerwind/actions-runner:v2.296.2-ubuntu-20.04

RUN sudo curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
RUN sudo chmod +x /usr/local/bin/docker-compose

RUN sudo curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN sudo unzip awscliv2.zip
RUN sudo ./aws/install

[orbi-ci-bot]

bump

[orbi-ci-bot]

I am still facing this issue. Was looking into this and found that my pods are using more memory than required.

[root@ip-10-70-111-97 ~]# kubectl top pods -A | grep runner
actions-runner actions-runner-controller-bc6885676-4xvzz 2m 35Mi
actions-runner runner-deploy-tw8bh-c7whv 2m 2089Mi
actions-runner runner-deploy-tw8bh-mr828 2m 7536Mi
actions-runner runner-deploy-tw8bh-vgsbx 2m 1446Mi

The config file limits it to 6 GB(modified it last week actually from 4GB listed above) and yet the runner is running at 7.5 GB of memory.

This maybe due to a memory leak maybe? I ran the following command inside the runner pod itself and got the below response.

[root@ip-10-70-111-97 ~]# kubectl exec -it runner-deploy-tw8bh-mr828 -n actions-runner sh
Defaulting container name to runner.
Use 'kubectl describe pod/runner-deploy-tw8bh-mr828 -n actions-runner' to see all of the containers in this pod.
$ cat /sys/fs/cgroup/memory/memory.usage_in_bytes
2258173952

Which is around 2 GB compared to the 7.5GB shown by kubectl top pods.

When the pods hits around 10GB, the pod is evicted. Also, No jobs were running when getting that output.

Right now. I am running a cronjob to delete the pods and recreate the pods every 8 hours. But I would appreciate if there is any other fix we can include.

Looking for any help I can get