Skip to content

Commit 267ca50

Browse files
KeenutsKeenuts
authored
[CI] Update documents around the GCP runners (#367)
This PR expands the documentation around GCP and the K8 cluster running linux/windows runners.
1 parent 87df822 commit 267ca50

File tree

5 files changed

+346
-23
lines changed

5 files changed

+346
-23
lines changed

‎premerge/README.md

Lines changed: 5 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -5,27 +5,9 @@ resources used to run the premerge checks. Currently, only Google employees
55
with access to the GCP project where these checks are hosted are able to apply
66
changes. Pull requests from anyone are still welcome.
77

8-
## Setup
8+
## Index
99

10-
- install terraform (https://developer.hashicorp.com/terraform/install?product_intent=terraform)
11-
- get the GCP tokens: `gcloud auth application-default login`
12-
- initialize terraform: `terraform init`
13-
14-
To apply any changes to the cluster:
15-
- setup the cluster: `terraform apply`
16-
- terraform will list the list of proposed changes.
17-
- enter 'yes' when prompted.
18-
19-
## Setting the cluster up for the first time
20-
21-
```
22-
terraform apply -target google_container_node_pool.llvm_premerge_linux_service
23-
terraform apply -target google_container_node_pool.llvm_premerge_linux
24-
terraform apply -target google_container_node_pool.llvm_premerge_windows
25-
terraform apply
26-
```
27-
28-
Setting the cluster up for the first time is more involved as there are certain
29-
resources where terraform is unable to handle explicit dependencies. This means
30-
that we have to set up the GKE cluster before we setup any of the Kubernetes
31-
resources as otherwise the Terraform Kubernetes provider will error out.
10+
- [Architecture overview](architecture.md)
11+
- [Cluster management](cluster-management.md)
12+
- [Monitoring](monitoring.md)
13+
- [Past issues](issues.md)

‎premerge/architecture.md

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# LLVM Premerge infra - GCP runners
2+
3+
This document describes how the GCP based presubmit infra is working, and
4+
explains common maintenance actions.
5+
6+
## Overview
7+
8+
Presubmit tests are using GitHub workflows. Executing GitHub workflows can be
9+
done in two ways:
10+
- using GitHub provided runners.
11+
- using self-hosted runners.
12+
13+
GitHub provided runners are not very powerful, and have limitations, but they
14+
are **FREE**.
15+
Self hosted runners are self-hosted, meaning they can be large virtual
16+
machines running on GCP, very powerful, but **expensive**.
17+
18+
To balance cost/performance, we keep both types.
19+
- simple jobs like `clang-format` shall run on GitHub runners.
20+
- building & testing LLVM shall be done on self-hosted runners.
21+
22+
LLVM has several flavor of self-hosted runners:
23+
- libcxx runners.
24+
- MacOS runners for HLSL managed by Microsoft.
25+
- GCP windows/linux runners managed by Google.
26+
27+
This document only focuses on Google's GCP hosted runners.
28+
29+
Choosing on which runner a workflow runs is done in the workflow definition:
30+
31+
```
32+
jobs:
33+
my_job_name:
34+
# Runs on expensive GCP VMs.
35+
runs-on: llvm-premerge-linux-runners
36+
```
37+
38+
Our self hosted runners come in two flavors:
39+
- Linux
40+
- Windows
41+
42+
## GCP runners - Architecture overview
43+
44+
Our runners are hosted on a GCP Kubernetes cluster, and use the [Action Runner Controller (ARC)](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller).
45+
The cluster has 3 pools:
46+
- llvm-premerge-linux
47+
- llvm-premerge-linux-service
48+
- llvm-premerge-windows
49+
50+
**llvm-premerge-linux-service** is a fixed pool, only used to host the
51+
services required to manage the premerge infra (controller, listeners,
52+
monitoring). Today, this pool has three `e2-highcpu-4` machine.
53+
54+
**llvm-premerge-linux** is a auto-scaling pool with large `n2-standard-64`
55+
VMs. This pool runs the Linux workflows.
56+
57+
**llvm-premerge-windows** is a auto-scaling pool with large `n2-standard-64`
58+
VMs. Similar to the Linux pool, but this time it runs Windows workflows.
59+
60+
### Service pool: llvm-premerge-linux-service
61+
62+
This pool runs all the services managing the presubmit infra.
63+
- Action Runner Controller
64+
- 1 listener for the Linux runners.
65+
- 1 listener for the windows runners.
66+
- Grafana Alloy to gather metrics.
67+
- metrics container.
68+
69+
The Action Runner Controller listens on the LLVM repository job queue.
70+
Individual jobs are then handled by the listeners.
71+
72+
How a job is run:
73+
- The controller informs GitHub the self-hosted runner set is live.
74+
- A PR is uploaded on GitHub
75+
- The listener finds a Linux job to run.
76+
- The listener creates a new runner pod to be scheduled by Kubernetes.
77+
- Kubernetes adds one instance to the Linux pool to schedule new pod.
78+
- The runner starts executing on the new node.
79+
- Once finished, the runner dies, meaning the pod dies.
80+
- If the instance is not reused in the next 10 minutes, the autoscaler
81+
will turn down the instance, freeing resources.
82+
83+
### Worker pools : llvm-premerge-linux, llvm-premerge-windows
84+
85+
To make sure each runner pod is scheduled on the correct pool (linux or
86+
windows, avoiding the service pool), we use labels and taints.
87+
88+
The other constraints we define are the resource requirements. Without
89+
information, Kubernetes is allowed to schedule multiple pods on the instance.
90+
So if we do not enforce limits, the controller could schedule 2 runners on
91+
the same instance, forcing containers to share resources.
92+
93+
Those bits are configures in the
94+
[linux runner configuration](linux_runners_values.yaml) and
95+
[windows runner configuration](windows_runners_values.yaml).
96+

‎premerge/cluster-management.md

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Cluster configuration
2+
3+
The cluster is managed using Terraform. The main configuration is
4+
[main.tf](main.tf).
5+
6+
---
7+
NOTE: As of today, only Googlers can administrate the cluster.
8+
---
9+
10+
Terraform is a tool to automate infrastructure deployment. Basic usage is to
11+
change this configuration and to call `terraform apply` make the required
12+
changes.
13+
Terraform won't recreate the whole cluster from scratch every time, instead
14+
it tries to only apply the new changes. To do so, **Terraform needs a state**.
15+
16+
**If you apply changes without this state, you might break the cluster.**
17+
18+
The current configuration stores its state into a GCP bucket.
19+
20+
21+
## Accessing Google Cloud Console
22+
23+
This web interface is the easiest way to get a quick look at the infra.
24+
25+
---
26+
IMPORTANT: cluster state is managed with terraform. Please DO NOT change
27+
shapes/scaling, and other settings using the cloud console. Any change not
28+
done through terraform will be at best overridden by terraform, and in the
29+
worst case cause an inconsistent state.
30+
---
31+
32+
The main part you want too look into is `Menu > Kubernetes Engine > Clusters`.
33+
34+
Currently, we have 3 clusters:
35+
- `llvm-premerge-checks`: the cluster hosting BuildKite Linux runners.
36+
- `windows-cluster`: the cluster hosting BuildKite Windows runners.
37+
- `llvm-premerge-prototype`: the cluster for those GCP hoster runners.
38+
39+
Yes, it's called `prototype`, but that's the production cluster.
40+
We should rename it at some point.
41+
42+
To add a VM to the cluster, the VM has to come from a `pool`. A `pool` is
43+
a group of nodes withing a cluster that all have the same configuration.
44+
45+
For example:
46+
A pool can say it contains at most 10 nodes, each using the `c2d-highcpu-32`
47+
configuration (32 cores, 64GB ram).
48+
In addition, a pool can `autoscale` [docs](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler).
49+
50+
If you click on `llvm-premerge-prototype`, and go to the `Nodes` tab, you
51+
will see 3 node pools:
52+
- llvm-premerge-linux
53+
- llvm-premerge-linux-service
54+
- llvm-premerge-windows
55+
56+
Definitions for each pool is in [Architecture overview](architecture.md).
57+
58+
If you click on a pool, example `llvm-premerge-linux`, you will see one
59+
instance group, and maybe several nodes.
60+
61+
Each created node must be attached to an instance group, which is used to
62+
manage a group of instances. Because we use automated autoscale, and we have
63+
a basic cluster, we have a single instance group per pool.
64+
65+
Then, we have the nodes. If you are looking at the panel during off hours,
66+
you might see no nodes at all: when no presubmit is running, no VM is on.
67+
If you are looking at the panel at peak time, you should see 4 instances.
68+
(Today, autoscale is capped at 4 instances).
69+
70+
If you click on a node, you'll see the CPU usage, memory usage, and can access
71+
the logs for each instance.
72+
73+
As long as you don't click on actions like `Cordon`, `Edit`, `Delete`, etc,
74+
navigating the GCP panel should not cause any harm. So feel free to look
75+
around to familiarize yourself with the interface.
76+
77+
## Setup
78+
79+
- install terraform (https://developer.hashicorp.com/terraform/install?product_intent=terraform)
80+
- get the GCP tokens: `gcloud auth application-default login`
81+
- initialize terraform: `terraform init`
82+
83+
To apply any changes to the cluster:
84+
- setup the cluster: `terraform apply`
85+
- terraform will list the list of proposed changes.
86+
- enter 'yes' when prompted.
87+
88+
## Setting the cluster up for the first time
89+
90+
```
91+
terraform apply -target google_container_node_pool.llvm_premerge_linux_service
92+
terraform apply -target google_container_node_pool.llvm_premerge_linux
93+
terraform apply -target google_container_node_pool.llvm_premerge_windows
94+
terraform apply
95+
```
96+
97+
Setting the cluster up for the first time is more involved as there are certain
98+
resources where terraform is unable to handle explicit dependencies. This means
99+
that we have to set up the GKE cluster before we setup any of the Kubernetes
100+
resources as otherwise the Terraform Kubernetes provider will error out.

‎premerge/issues.md

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Past Issues
2+
3+
This document lists past issues that could be of interest if you encounter
4+
issues with the cluser/presubmit.
5+
6+
## Workflows are failing: DNS resolution of github.com fails.
7+
8+
### Date: 2025-01-27
9+
10+
### Symptoms:
11+
12+
We noticed GitHub jobs were failing, and the logs showed `hostname lookup
13+
failed for github.com`.
14+
15+
### Investigation
16+
17+
Initial steps were to check the github status page. No outages.
18+
Then, we checked internal incident page. No outages.
19+
20+
Then we looked at the instance logs.
21+
When a node fails, GCP removes it, meaning the logs are not accessible from
22+
the node page anymore, but can be retrieved either in the global logs.
23+
24+
Looking at the logs, we discovered other services failing to resolve
25+
hostname, like the metrics container.
26+
27+
In Kubernetes, each cluster runs a `kube-dns` service/pod, which is used
28+
by other pods to do DNS requests.
29+
This pod was crashing.
30+
31+
Looking at the node this pod was running on showed a RAM usage close to the
32+
VM limits.
33+
At the time, the service instances were running on `e2-small` VMs, which only
34+
have 2GB of RAM.
35+
In addition, we recently added more runner nodes by adding a new Windows pool.
36+
This meant cluster size increased. This caused the cluster management services
37+
to take more resources to run, and pushed us just above the 2GB limit.
38+
39+
This causes the kube-dns service to be OOM killed, and then caused various DNS
40+
failures in the cluster.
41+
42+
### Solution
43+
44+
Change the shape of the service pool to be `e2-highcpu-4`, doubling the RAM
45+
and CPU budget. We also increased the pool size from 2 to 3.
46+
47+
## LLVM dashboard graphs are empty for presubmits
48+
49+
### Date: 2025-01-28
50+
51+
### Symptoms
52+
53+
The LLVM dashboard was showing empty graphs for the presubmit job runtime and
54+
queue time. Autoscaling graphs were still working.
55+
56+
### Investigation
57+
58+
The graphs were empty because no new metrics were received, but other GCP
59+
metrics were still showing.
60+
Our dashboard has multiple data source:
61+
- the GCP cluster.
62+
- the metrics container.
63+
64+
Because we had GCP metrics, it meant the Grafana instance was working, and
65+
the Grafana Alloy component running in the cluster was also fine.
66+
67+
It was probably the metrics container.
68+
We checked the heartbeat metric: `metrics_container_heartbeat`.
69+
This is a simple ping recorded every minutes by the container. If this
70+
metrics stops emitting, it means something is wrong with the job.
71+
This metric was still being recorded.
72+
73+
A recent change was made to add the windows version of the premerge check.
74+
This caused the job name to change, and thus changed the recorded metric
75+
names from `llvm_premerge_checks_linux_run_time` to
76+
`llvm_premerge_checks_premerge_checks_linux_run_time`.
77+
78+
### Solution
79+
80+
Change the dashboards to read the new metric name instead of the previous
81+
name, allowing new data to be shown.
82+
SLO definitions and alerts also had to be adjusted to look at the new metrics.
83+
84+
## LLVM dashboard graphs are empty for run/queue times
85+
86+
### Date: 2025-01-10
87+
88+
### Symptoms
89+
90+
The LLVM dashboard was showing empty graphs for the presubmit job runtime and
91+
queue time. Autoscaling graphs were still working.
92+
93+
### Investigation
94+
95+
Grafana was still recording GCP metrics, but no new data coming from the
96+
metrics container.
97+
98+
A quick look at the google cloud console showed the metrics container pod was
99+
crashing.
100+
Looking at the logs, we saw the script failed to connect to GitHub to get
101+
the workflow status. Reason was a bad GitHub token.
102+
103+
Because we have no admin access to the GitHub admin organization, we cannot
104+
emit LLVM owned tokens. A Googler had used its personal account to setup a
105+
PAT token. This token expired in December, causing the metrics container
106+
to fail since.
107+
108+
### Solution
109+
110+
Another Googler generated a new token, and replaced it in `Menu > Security > Secrets Manager > llvm-premerge-github-pat`.
111+
Note: this secret is in the general secret manager, not in `Kubernetes Engine > Secrets & ConfigMaps`.
112+
113+
Once the secret updated, the metrics container had to be restarted:
114+
- `Menu > Kubernetes Engine > Workflows`
115+
- select `metrics`.
116+
- click `Rolling update`
117+
- set all thresholds to `100%`, the click update.
118+
119+
This will allow GCP to delete the only metrics container pod, and recreate it
120+
using the new secret value.
121+
Because we have a single metrics container instance running, we have to but
122+
all thresholds to `100%`.
123+
124+
In addition, we added a heartbeat metric to the container, and Grafana
125+
alerting to make sure we detect this kind of failure early.

‎premerge/monitoring.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Monitoring
2+
3+
Presubmit monitoring is provided by Grafana.
4+
The dashboard link is [https://llvm.grafana.net/dashboards](https://llvm.grafana.net/dashboards).
5+
6+
Grafana pulls its data from 2 sources: the GCP Kubernetes cluster & GitHub.
7+
Grafana instance access is restricted, but there is a publicly visible dashboard:
8+
- [Public dashboard](https://llvm.grafana.net/public-dashboards/21c6e0a7cdd14651a90e118df46be4cc)
9+
10+
## GCP monitoring
11+
12+
Cluster metrics are gathered through Grafana alloy.
13+
This service is deployed using Helm, as described [HERE](main.tf)
14+
15+
## Github monitoring
16+
17+
Github CI queue and job status is fetched using a custom script which pushes
18+
metrics to grafana.
19+
The script itself lives in the llvm-project repository: [LINK](https://github.com/llvm/llvm-project/blob/main/.ci/metrics/metrics.py).
20+
The deployment configuration if in the [metrics_deployment file](metrics_deployment.yaml).

0 commit comments

Comments
 (0)