ARC Runner Scale Sets on Azure

[simpson]

We recently built a new Github workflow that is broken down to two buckets of work

1. Terraform an AKS cluster on Azure
2. Pour ARC and build a Runner Scale Set

After a large bit of back and forth testing to work out the kinks, everything is running on AKS. We have it set to a minimum of 1 runner and a max of 20, so after finishing deployment of the whole workflow we have the Runner Set online and one runner is online and idle. The issue we are facing is that when we push a workflow that has 20+ builds, ARC works beautifully and quickly scales up to 20 runners, however, our AKS cluster never spins up additional nodes to back that workload so it basically crashes and no builds complete. Does anyone have any advice on how to get AKS to scale up more nodes to balance the workload?

Github Workflow

jobs:
  Terraform:
    name: Terraform AKS Cluster
    runs-on: ubuntu-latest

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - name: Set up Terraform
        uses: hashicorp/setup-terraform@v1
        with:
          terraform_version: 1.4.6
          cli_config_credentials_token: ${{ secrets.asdf }}

      - name: Terraform Init
        run: terraform -chdir=tf-files init

      - name: Terraform Validate
        run: terraform -chdir=tf-files validate

      - name: Terraform Plan
        if: github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch'
        run: TF_VAR_client_id=${{ secrets.asdf }} TF_VAR_subscription_id=${{ secrets.asdf }} TF_VAR_tenant_id=${{ secrets.asdf }} TF_VAR_client_secret=${{ secrets.asdf }} TF_VAR_resource_group_name=${{ github.event.inputs.resource_group_name }} TF_VAR_aks_cluster_name=${{ github.event.inputs.aks_cluster_name }} TF_VAR_location=${{ github.event.inputs.location }} terraform -chdir=tf-files plan -out=tfplan -input=false -no-color

      - name: Terraform Apply
        if: (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'workflow_dispatch'
        run: |
          TF_VAR_client_id=${{ secrets.AZURE_CLIENT_ID }} TF_VAR_subscription_id=${{ secrets.AZURE_SUBSCRIPTION_ID }} TF_VAR_tenant_id=${{ secrets.AZURE_TENANT_ID }} TF_VAR_client_secret=${{ secrets.AZURE_CLIENT_SECRET }} TF_VAR_resource_group_name=${{ github.event.inputs.resource_group_name }} TF_VAR_aks_cluster_name=${{ github.event.inputs.aks_cluster_name }} TF_VAR_location=${{ github.event.inputs.location }} terraform -chdir=tf-files apply -auto-approve
          echo "::set-output name=resource_group_name::$(terraform output -raw resource_group_name)"
          echo "::set-output name=cluster_name::$(terraform output -raw cluster_name)"

  Kubernetes:
    name: Deploy Kubernetes
    runs-on: ubuntu-latest
    needs: terraform

    steps:
      - name: Checkout repository
        uses: actions/checkout@v2

      - uses: Azure/login@v1
        with:
          creds: '{"clientId":"${{ secrets.AZURE_CLIENT_ID }}","clientSecret":"${{ secrets.asdf }}","subscriptionId":"${{ secrets.AZURE_SUBSCRIPTION_ID }}","tenantId":"${{ secrets.asdf }}"}'

      - uses: azure/aks-set-context@v3
        with:
          resource-group: ${{ github.event.inputs.resource_group_name }}
          cluster-name: ${{ github.event.inputs.aks_cluster_name }}

      - name: Install Helm
        run: |
          curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
          chmod 700 get_helm.sh
          ./get_helm.sh

      - name: Install kubectl
        run: |
          curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
          echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
          sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
          kubectl version --client
          kubectl cluster-info

      - name: Install actions-runner-controller
        run: |
          NAMESPACE="gh-aks-cluster"
          helm install arc \
              --namespace "${NAMESPACE}" \
              --create-namespace \
              oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

      - name: Install autoscaling runner set
        run: |
          INSTALLATION_NAME="gh-aks-runners"
          NAMESPACE="arc-runners"
          helm install "${INSTALLATION_NAME}" \
              -f ./manifest/values.yaml \
              --set githubConfigSecret.github_token="${{ secrets.asdf }}" \
              --namespace "${NAMESPACE}" \
              --create-namespace \
              oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

      - name: Check namespace and pods
        run: |
          helm list -A
          kubectl get pods -n gh-aks-cluster

Terraform for AKS cluster

terraform {
  required_version = ">=1.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~>3.0"
    }
  }

  cloud {
    organization = "lads"

    workspaces {
      name = "asdf"
    }
  }
}

provider "azurerm" {
  features {}
  subscription_id = var.subscription_id
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
}

resource "azurerm_resource_group" "rg" {
  name     = var.resource_group_name
  location = var.location
}

resource "azurerm_virtual_network" "vnet" {
  name                = "vnet"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  address_space       = ["10.0.0.0/16"]
}

resource "azurerm_subnet" "subnet" {
  name                 = "subnet"
  resource_group_name  = azurerm_resource_group.rg.name
  virtual_network_name = azurerm_virtual_network.vnet.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_log_analytics_workspace" "log-workspace" {
  name                = "DefaultWorkspace"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  sku                 = "PerGB2018"
}

resource "azurerm_kubernetes_cluster" "arc-cluster" {
  name                = var.aks_cluster_name
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  node_resource_group = var.node_resource_group
  dns_prefix          = "${var.aks_cluster_name}-dns"

  identity {
    type = "SystemAssigned"
  }

  auto_scaler_profile {
    balance_similar_node_groups       = false
    empty_bulk_delete_max             = "10"
    expander                          = "random"
    max_graceful_termination_sec      = "600"
    max_node_provisioning_time        = "15m"
    max_unready_nodes                 = 3 
    max_unready_percentage            = 45
    new_pod_scale_up_delay            = "0s"
    scale_down_delay_after_add        = "10m"
    scale_down_delay_after_delete     = "10s"
    scale_down_delay_after_failure    = "3m"
    scale_down_unneeded               = "5m"
    scale_down_unready                = "10m"
    scale_down_utilization_threshold  = "0.5"
    scan_interval                     = "5s"
    skip_nodes_with_local_storage     = false
    skip_nodes_with_system_pods       = false
  }

  default_node_pool {
    name                  = "runners"
    vm_size               = "Standard_DS2_v2"
    os_disk_size_gb       = 32
    os_sku                = "Ubuntu"
    os_disk_type          = "Managed"
    scale_down_mode       = "Delete"
    max_pods              = "40"
    vnet_subnet_id        = azurerm_subnet.subnet.id
    type                  = "VirtualMachineScaleSets"
    enable_auto_scaling   = true
    min_count             = 1
    max_count             = 100
  }

  network_profile {
    network_plugin      = "kubenet"
    load_balancer_sku   = "standard"
    outbound_type       = "loadBalancer"
    pod_cidr            = "10.244.0.0/16"
    service_cidr        = "10.2.0.0/16"
    dns_service_ip      = "10.2.0.10"
    docker_bridge_cidr  = "172.17.0.1/16"
  }
}

[lorencarvalho]

By default AKS will run up to 110 pods per node, so unless you specify some resource request/limits AKS presumes you don't want more nodes.

[simpson]

I was able to get it working by adding this in values.yaml

  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        resources:
          requests:
            memory: "12G"
          limits:
            memory: "16G"

[lorencarvalho]

@simpson be advised that by setting a memory limit if your runner tries to allocate more than that amount (16G) the runner process may be OOM killed, inadvertently causing the pod to be terminated and any active workflow executions to fail. Just something to be mindful of.

A good way to check on the state of the autoscaler, and to get any clues about why it might not be working right, take a look at the cluster-autoscaler configmap:

kubectl get configmap -n kube-system cluster-autoscaler-status -o yaml