Skip to content

Commit 12bb419

Browse files
sanderginnsanderginn
authored
Executors: add native k8s executors (#279)
* Move dind executor chart into new subdirectory * Add native executor charts * Cleanup from move * Add TODO * configures native executors * move registry into executor chart * conditionally mount docker config * move rbac into main chart * remove subdir * Fix test * yml -> yaml` * some more tests * rename dir native->k8s * set image tag * move rbac value into executor block * remove image name from tag * docs * update schema validation file * cleanup dind * gen docs * add readme with brief chart descriptions * slight change to validation script * bump rgistry disk size and add to dind values * gen docs * rip out private registry for k8s impl * rename rolebinding file\ * clarify dind in docs * change default tag of dind image * gen docs * clarify frontend url * gen docs * Specify implementation in chart names * Add separate step for executor helm chart packaging * Add separate step for executor helm chart packaging in insiders release * small doc tweak * Add changelog for executors * Fix SA name in role binding * Actually use value for dockerAddHostGateway * Set secret as env var on deployment * Use storageclass value in PVC * update docs
1 parent 7cdad50 commit 12bb419

34 files changed

+949
-170
lines changed

‎.github/workflows/gcs_chart_publish_insiders.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,11 @@ jobs:
5353
sed -i 's/appVersion:.*/appVersion: insiders/g' charts/*/Chart.yaml
5454
sed -i '/^version:/ s/"$/-insiders.${{ steps.metadata.outputs.shortSHA }}"/' charts/*/Chart.yaml
5555
56-
- name: Package helm charts
57-
run: for i in charts/*; do helm package -u $i; done
56+
- name: Package Sourcegraph helm charts
57+
run: for i in charts/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
58+
59+
- name: Package Sourcegraph executor helm charts
60+
run: for i in charts/sourcegraph-executor/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
5861

5962
- name: Publish chart if it doesn't already exist in the repository
6063
run: |

‎.github/workflows/gcs_chart_publish_release.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -44,8 +44,11 @@ jobs:
4444
- name: Set up helm repo
4545
run: helm repo add sg gs://${{ env.BUCKET_NAME }}/${{ env.BUCKET_PATH }}
4646

47-
- name: Package helm charts
48-
run: for i in charts/*; do helm package -u $i; done
47+
- name: Package Sourcegraph helm charts
48+
run: for i in charts/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
49+
50+
- name: Package Sourcegraph executor helm charts
51+
run: for i in charts/sourcegraph-executor/*; do [ -f "$i/Chart.yaml" ] && helm package -u $i; done
4952

5053
- name: Publish chart if it doesn't already exist in the repository
5154
run: |

‎charts/sourcegraph-executor/CHANGELOG.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Changelog
2+
3+
<!-- START CHANGELOG -->
4+
## Unreleased
5+
6+
* Added new chart `sourcegraph-executor-k8s` to deploy Sourcegraph executors that use Kubernetes jobs.
7+
* **BREAKING:** Renamed `sourcegraph-executor` chart to `sourcegraph-executor-dind` to indicate these are Docker in Docker executors. To update to newer versions of this chart, ensure the new Chart name is used.

‎charts/sourcegraph-executor/README.md‎

Lines changed: 12 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -1,102 +1,16 @@
1-
<!--
2-
DO NOT EDIT README.md directly.
3-
README.md is automatically generated from README.md.gotmpl
4-
-->
1+
# Sourcegraph executor helm charts
52

6-
# Sourcegraph Exexutor Helm Chart
3+
This directory contains two Helm charts for deploying executors on Kubernetes. [Read the docs](doc/admin/executors/deploy_executors_kubernetes.md) for more information on executors on Kubernetes.
74

8-
This chart contains two deployments, Sourcegraph Executors and a private Docker Registry. It is a supplemental chart for the parent [sourcegraph/sourcegraph] Helm Chart if you wish to deploy executors
5+
## Native Kubernetes (k8s)
6+
This chart deploys executors that interact with the Kubernetes API to handle jobs. This is the preferred deployment option.
7+
Your cluster will need to allow configuration of the following RBAC rules:
98

10-
Use cases:
9+
| API Groups | Resources | Verbs | Reason |
10+
|------------|--------------------|---------------------------|-------------------------------------------------------------------------------------------|
11+
| `batch` | `jobs` | `create`, `delete` | Executors create Job pods to run processes. Once Jobs are completed, they are cleaned up. |
12+
| | `pods`, `pods/log` | `get`, `list`, `watch` | Executors need to look up and steam logs from the Job Pods. |
1113

12-
- Deploy Sourcegraph Executors on Kubernetes
13-
14-
## Requirements
15-
16-
* [Helm 3 CLI](https://helm.sh/docs/intro/install/)
17-
* Kubernetes 1.19 or greater
18-
19-
## Installation
20-
21-
Add the Sourcegraph charts repo to Helm:
22-
23-
```sh
24-
helm repo add sourcegraph https://helm.sourcegraph.com/release
25-
```
26-
27-
## Usage
28-
29-
> The chart has to be installed in the same namespace as the parent [sourcegraph/sourcegraph] chart
30-
31-
## Rendering manifests for kubectl deployment
32-
33-
Manifests rendered using the `helm template` command can be used for direct deployment using `kubectl`.
34-
35-
## Configuration Options
36-
37-
Reference the table below for available configuration parameters and consult [executor] documentation.
38-
39-
In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values
40-
41-
- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
42-
- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
43-
- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
44-
- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
45-
- `executor.env` - consult `values.yaml`
46-
47-
- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
48-
- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
49-
- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
50-
- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
51-
- `privateDockerRegistry.env` - consult `values.yaml` file
52-
53-
| Key | Type | Default | Description |
54-
|-----|------|---------|-------------|
55-
| dind.image.registry | string | `"index.docker.io"` | |
56-
| dind.image.repository | string | `"docker"` | |
57-
| dind.image.tag | string | `"20.10.22-dind"` | |
58-
| executor.enabled | bool | `true` | |
59-
| executor.env.EXECUTOR_FRONTEND_PASSWORD | object | `{"value":""}` | The shared secret configured in the Sourcegraph instance site config under executors.accessToken. Required. |
60-
| executor.env.EXECUTOR_FRONTEND_URL | object | `{"value":""}` | The external URL of the Sourcegraph instance. Required. |
61-
| executor.env.EXECUTOR_QUEUE_NAME | object | `{"value":""}` | The name of the queue to pull jobs from to. Possible values: batches and codeintel. Required. |
62-
| executor.image.defaultTag | string | `"5.0.6@sha256:f78a0ba3d84941ec3577f19075e935bec34a9bfaa9dcb85908142fff017e9d6d"` | |
63-
| executor.image.name | string | `"executor"` | |
64-
| privateDockerRegistry.image.registry | string | `"index.docker.io"` | |
65-
| privateDockerRegistry.image.repository | string | `"docker/regisry"` | |
66-
| privateDockerRegistry.image.tag | int | `2` | |
67-
| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) |
68-
| sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag |
69-
| sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy |
70-
| sourcegraph.image.repository | string | `"index.docker.io/sourcegraph"` | Global docker image registry or prefix |
71-
| sourcegraph.image.useGlobalTagAsDefault | bool | `false` | When set to true, sourcegraph.image.defaultTag is used as the default defaultTag for all services, instead of service-specific default defaultTags |
72-
| sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials |
73-
| sourcegraph.labels | object | `{}` | Add a global label to all resources |
74-
| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation |
75-
| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) |
76-
| sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods |
77-
| sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods |
78-
| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) |
79-
| storageClass.allowedTopologies | object | `{}` | Persistent volumes topology configuration, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies) |
80-
| storageClass.create | bool | `false` | Enable creation of storageClass. Defaults to Google Cloud Platform. Disable if you have your own existing storage class |
81-
| storageClass.name | string | `"sourcegraph"` | Name of the storageClass. Use to customize to the existing storage class name |
82-
| storageClass.parameters | object | `{}` | Extra parameters of storageClass, consult your cloud provider persistent storage documentation |
83-
| storageClass.provisioner | string | `"kubernetes.io/gce-pd"` | Name of the storageClass provisioner, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner) and consult your cloud provider persistent storage documentation |
84-
| storageClass.type | string | `"pd-ssd"` | Value of `type` key in storageClass `parameters`, consult your cloud provider persistent storage documentation |
85-
86-
## Troubleshooting
87-
88-
See a list of running executor pods
89-
90-
```sh
91-
kubectl get pods -l app=executor
92-
```
93-
94-
Check logs of the executor container
95-
96-
```sh
97-
kubectl logs -l app=executor -c executor
98-
```
99-
100-
[sourcegraph/sourcegraph]: ../sourcegraph/
101-
[sourcegraph/sourcegraph-executor]: ./
102-
[executor]: https://docs.sourcegraph.com/admin/executors
14+
## Docker in Docker (dind)
15+
This chart deploys executors that deploy a [Docker in Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) sidecar with each executor pod to avoid accessing the host container runtime directly. This method requires privileged access to a container runtime daemon in order to operate correctly.
16+
If you have security concerns, consider deploying via [a non-Kubernetes method](https://docs.sourcegraph.com/admin/executors).

charts/sourcegraph-executor/.helmignore renamed to charts/sourcegraph-executor/dind/.helmignore

File renamed without changes.

‎charts/sourcegraph-executor/dind/Chart.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
apiVersion: v2
2+
name: sourcegraph-executor-dind
3+
description: Chart for installing Sourcegraph executors using Docker in Docker (dind)
4+
icon: https://sourcegraph.com/favicon.ico
5+
type: application
6+
7+
# Chart version, separate from Sourcegraph
8+
version: "5.0.2"
9+
10+
# Version of Sourcegraph release
11+
appVersion: "5.0.2"

‎charts/sourcegraph-executor/dind/README.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
<!--
2+
DO NOT EDIT README.md directly.
3+
README.md is automatically generated from README.md.gotmpl
4+
-->
5+
6+
# Sourcegraph Exexutor Helm Chart
7+
8+
This chart contains two deployments, Sourcegraph Executors and a private Docker Registry. It is a supplemental chart for the parent [sourcegraph/sourcegraph] Helm Chart if you wish to deploy executors
9+
10+
Use cases:
11+
12+
- Deploy Sourcegraph Executors on Kubernetes
13+
14+
## Requirements
15+
16+
* [Helm 3 CLI](https://helm.sh/docs/intro/install/)
17+
* Kubernetes 1.19 or greater
18+
19+
## Installation
20+
21+
Add the Sourcegraph charts repo to Helm:
22+
23+
```sh
24+
helm repo add sourcegraph https://helm.sourcegraph.com/release
25+
```
26+
27+
## Usage
28+
29+
> The chart has to be installed in the same namespace as the parent [sourcegraph/sourcegraph] chart
30+
31+
## Rendering manifests for kubectl deployment
32+
33+
Manifests rendered using the `helm template` command can be used for direct deployment using `kubectl`.
34+
35+
## Configuration Options
36+
37+
Reference the table below for available configuration parameters and consult [executor] documentation.
38+
39+
In addition to the documented values, the `executor` and `private-docker-registry` services also supports the following values
40+
41+
- `executor.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
42+
- `executor.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
43+
- `executor.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
44+
- `executor.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
45+
- `executor.env` - consult `values.yaml`
46+
47+
- `privateDockerRegistry.affinity` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity)
48+
- `privateDockerRegistry.nodeSelector` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)
49+
- `privateDockerRegistry.tolerations` - [learn more](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
50+
- `privateDockerRegistry.podSecurityContext` - [learn more](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod)
51+
- `privateDockerRegistry.env` - consult `values.yaml` file
52+
53+
| Key | Type | Default | Description |
54+
|-----|------|---------|-------------|
55+
| dind.image.registry | string | `"index.docker.io"` | |
56+
| dind.image.repository | string | `"docker"` | |
57+
| dind.image.tag | string | `"20.10.22-dind"` | |
58+
| executor.enabled | bool | `true` | |
59+
| executor.env.EXECUTOR_FRONTEND_PASSWORD | object | `{"value":""}` | The shared secret configured in the Sourcegraph instance site config under executors.accessToken. Required. |
60+
| executor.env.EXECUTOR_FRONTEND_URL | object | `{"value":""}` | The external URL of the Sourcegraph instance. Required. |
61+
| executor.env.EXECUTOR_QUEUE_NAME | object | `{"value":""}` | The name of the queue to pull jobs from to. Possible values: batches and codeintel. **Either this or EXECUTOR_QUEUE_NAMES is required.** |
62+
| executor.env.EXECUTOR_QUEUE_NAMES | object | `{"value":""}` | The comma-separated list of names of multiple queues to pull jobs from to. Possible values: batches and codeintel. **Either this or EXECUTOR_QUEUE_NAME is required.** |
63+
| executor.image.defaultTag | string | `"5.1_230340_2023-06-23_5.0-93d39d620e83@sha256:172770133661d4d148327d2cac87c051a6409871ee6f6e28ce3495b60f883ad5"` | |
64+
| executor.image.name | string | `"executor"` | |
65+
| executor.replicaCount | int | `1` | |
66+
| privateDockerRegistry.enabled | bool | `true` | Whether to deploy the private registry. Only one registry is needed when deploying multiple executors. More information: https://docs.sourcegraph.com/admin/executors/deploy_executors#using-private-registries |
67+
| privateDockerRegistry.image.registry | string | `"index.docker.io"` | |
68+
| privateDockerRegistry.image.repository | string | `"docker/regisry"` | |
69+
| privateDockerRegistry.image.tag | int | `2` | |
70+
| privateDockerRegistry.storageSize | string | `"10Gi"` | |
71+
| sourcegraph.affinity | object | `{}` | Affinity, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity) |
72+
| sourcegraph.image.defaultTag | string | `"{{ .Chart.AppVersion }}"` | Global docker image tag |
73+
| sourcegraph.image.pullPolicy | string | `"IfNotPresent"` | Global docker image pull policy |
74+
| sourcegraph.image.repository | string | `"index.docker.io/sourcegraph"` | Global docker image registry or prefix |
75+
| sourcegraph.image.useGlobalTagAsDefault | bool | `false` | When set to true, sourcegraph.image.defaultTag is used as the default defaultTag for all services, instead of service-specific default defaultTags |
76+
| sourcegraph.imagePullSecrets | list | `[]` | Mount named secrets containing docker credentials |
77+
| sourcegraph.labels | object | `{}` | Add a global label to all resources |
78+
| sourcegraph.nameOverride | string | `""` | Set a custom name for the app.kubernetes.io/name annotation |
79+
| sourcegraph.nodeSelector | object | `{}` | NodeSelector, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#nodeselector) |
80+
| sourcegraph.podAnnotations | object | `{}` | Add extra annotations to attach to all pods |
81+
| sourcegraph.podLabels | object | `{}` | Add extra labels to attach to all pods |
82+
| sourcegraph.tolerations | list | `[]` | Tolerations, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/) |
83+
| storageClass.allowedTopologies | object | `{}` | Persistent volumes topology configuration, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#allowed-topologies) |
84+
| storageClass.create | bool | `false` | Enable creation of storageClass. Defaults to Google Cloud Platform. Disable if you have your own existing storage class |
85+
| storageClass.name | string | `"sourcegraph"` | Name of the storageClass. Use to customize to the existing storage class name |
86+
| storageClass.parameters | object | `{}` | Extra parameters of storageClass, consult your cloud provider persistent storage documentation |
87+
| storageClass.provisioner | string | `"kubernetes.io/gce-pd"` | Name of the storageClass provisioner, learn more from the [Kubernetes documentation](https://kubernetes.io/docs/concepts/storage/storage-classes/#provisioner) and consult your cloud provider persistent storage documentation |
88+
| storageClass.type | string | `"pd-ssd"` | Value of `type` key in storageClass `parameters`, consult your cloud provider persistent storage documentation |
89+
90+
## Troubleshooting
91+
92+
See a list of running executor pods
93+
94+
```sh
95+
kubectl get pods -l app=executor
96+
```
97+
98+
Check logs of the executor container
99+
100+
```sh
101+
kubectl logs -l app=executor -c executor
102+
```
103+
104+
[sourcegraph/sourcegraph]: ../sourcegraph/
105+
[sourcegraph/sourcegraph-executor]: ./
106+
[executor]: https://docs.sourcegraph.com/admin/executors

charts/sourcegraph-executor/README.md.gotmpl renamed to charts/sourcegraph-executor/dind/README.md.gotmpl

File renamed without changes.

0 commit comments

Comments
 (0)