How to use Service Containers

[dstockhammer]

I'm trying to use a service container in a job that runs on a self hosted runner. See the simplified workflow yaml below; this works on the cloud runner (ubuntu-latest):

jobs:
  build-and-test:
    runs-on: self-hosted
    services:
      mysql:
        image: mysql:8
        env:
          MYSQL_ROOT_PASSWORD: ${{ env.MYSQL_ROOT_PASSWORD }}
          MYSQL_DATABASE: ${{ env.MYSQL_DATABASE }}
          MYSQL_USER: ${{ env.MYSQL_USER }}
          MYSQL_PASSWORD: ${{ env.MYSQL_PASSWORD }}
    steps:
    - name: Wait for MySQL to be ready
      run: >-
        timeout 60s sh -c 'until ssh mysql -p 3306 2>&1
        | grep -q "invalid format"; do sleep 1; done'

Sadly on the self-hosted runner the connection fails. However it looks like the service container actually starts correctly; here's the simplified output of the Initialize containers step:

2023-07-13T12:56:05.1437555Z A job started hook has been configured by the self-hosted runner administrator
2023-07-13T12:56:05.1733374Z ##[group]Run '/etc/arc/hooks/job-started.sh'
2023-07-13T12:56:05.1778786Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-07-13T12:56:05.1779232Z ##[endgroup]
2023-07-13T12:56:05.2190877Z �[0;37m2023-07-13 12:56:05.212  DEBUG --- Running ARC Job Started Hooks�[0m
2023-07-13T12:56:05.2191643Z �[0;37m2023-07-13 12:56:05.215  DEBUG --- Running hook: /etc/arc/hooks/job-started.d/update-status�[0m
2023-07-13T12:56:05.2730799Z ##[group]Checking docker version
2023-07-13T12:56:05.2765424Z ##[command]/usr/local/bin/docker version --format '{{.Server.APIVersion}}'
2023-07-13T12:56:05.3386510Z '1.43'
2023-07-13T12:56:05.3407178Z Docker daemon API version: '1.43'
2023-07-13T12:56:05.3408077Z ##[command]/usr/local/bin/docker version --format '{{.Client.APIVersion}}'
2023-07-13T12:56:05.3808651Z '1.41'
2023-07-13T12:56:05.3843015Z Docker client API version: '1.41'
2023-07-13T12:56:05.3862418Z ##[endgroup]
2023-07-13T12:56:05.3869313Z ##[group]Clean up resources from previous jobs
2023-07-13T12:56:05.3881678Z ##[command]/usr/local/bin/docker ps --all --quiet --no-trunc --filter "label=60e226"
2023-07-13T12:56:05.4180852Z ##[command]/usr/local/bin/docker network prune --force --filter "label=60e226"
2023-07-13T12:56:05.4554375Z ##[endgroup]
2023-07-13T12:56:05.4555586Z ##[group]Create local container network
2023-07-13T12:56:05.4589677Z ##[command]/usr/local/bin/docker network create --label 60e226 github_network_b32e3391cbb44109907af320e7cf1790
2023-07-13T12:56:05.5590782Z 966f17831b8fe521ae7b736e6d2e707645e94cf0f1f31c2246db6a5e39d3ed32
2023-07-13T12:56:05.5706144Z ##[endgroup]
2023-07-13T12:56:05.5878683Z ##[group]Starting mysql service container
2023-07-13T12:56:05.5908207Z ##[command]/usr/local/bin/docker pull mysql:8
2023-07-13T12:56:06.9471413Z 8: Pulling from library/mysql
..
2023-07-13T12:56:18.0286992Z e5527cf2eb5f: Pull complete
2023-07-13T12:56:18.0489781Z Digest: sha256:232936eb036d444045da2b87a90d48241c60b68b376caf509051cb6cffea6fdc
2023-07-13T12:56:18.0498803Z Status: Downloaded newer image for mysql:8
2023-07-13T12:56:18.0568346Z docker.io/library/mysql:8
2023-07-13T12:56:18.0779727Z ##[command]/usr/local/bin/docker create --name 74027845f5c4449dbae4b51b3e1ea416_mysql8_444d01 --label 60e226 --network github_network_b32e3391cbb44109907af320e7cf1790 --network-alias mysql -p 3306:3306  -e "MYSQL_ROOT_PASSWORD=x" -e "MYSQL_DATABASE=x" -e "MYSQL_USER=x" -e "MYSQL_PASSWORD=x" -e GITHUB_ACTIONS=true -e CI=true mysql:8
2023-07-13T12:56:18.6021529Z 5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf
2023-07-13T12:56:18.6074980Z ##[command]/usr/local/bin/docker start 5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf
2023-07-13T12:56:19.3124805Z 5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf
2023-07-13T12:56:19.3156056Z ##[command]/usr/local/bin/docker ps --all --filter id=5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
2023-07-13T12:56:19.3560133Z 5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf Up Less than a second
2023-07-13T12:56:19.3600863Z ##[command]/usr/local/bin/docker port 5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf
2023-07-13T12:56:19.3845366Z 3306/tcp -> 0.0.0.0:3306
2023-07-13T12:56:19.3846027Z 3306/tcp -> :::3306
2023-07-13T12:56:19.4036121Z ##[endgroup]
2023-07-13T12:56:19.4073955Z ##[group]Waiting for all services to be ready
2023-07-13T12:56:19.4128748Z ##[command]/usr/local/bin/docker inspect --format="{{if .Config.Healthcheck}}{{print .State.Health.Status}}{{end}}" 5f509d328104f08093af0232b430beea90ac46ddb9319e1e6b88749211b819cf
2023-07-13T12:56:19.4425375Z mysql service is healthy.
2023-07-13T12:56:19.4425988Z ##[endgroup]

I don't see any errors other than the mysql service not being available at mysql, localhost or 127.0.0.1 (tried all 3 variants, with and without explicit port mapping in the service).

Am I doing something wrong? Do I need to configure something for service containers to work? I couldn't find anything in the docs.
I'm running latest everything; actions-runner-controller/actions-runner-controller helm chart version 0.23.3 with images summerwind/actions-runner-controller:v0.27.4 and summerwind/actions-runner:latest (which is v2.306.0-ubuntu-20.04-e0a7e14 at the time of writing).

Thanks!