Skip to content

Commit ee08164

Browse files
author
Ware, Joseph (DLSLtd,RAL,LSCI)
committed
re-order and expand docs
1 parent ab22281 commit ee08164

File tree

1 file changed

+87
-47
lines changed

1 file changed

+87
-47
lines changed

‎docs/how-to/debug-in-cluster.md‎

Lines changed: 87 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,99 @@
1-
# Debug a container within a cluster
1+
# Debugging containers
22

3-
The container build also publishes a debug container for each tagged release of the container with the tag suffixed with `-debug`. This container contains the workspace and has an alternative entrypoint which allows the devcontainer to attach: if you have configured a `livenessProbe` that requires the service to have started it should be disabled. The container also installs debugpy and makes the service install editable. Any custom `command` or `args` defined for the container should be disabled.
3+
The container build also publishes a debug container for each tagged release of the container suffixed with `-debug`. This container contains an editable install of the workspace & debugpy and has an alternate entrypoint which allows the devcontainer to attach.
44

5-
With the [Kubernetes plugin for vscode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. This may require that your targeted kubeconfig is at `~/.kube/config`, rather than referenced from the environment variable `KUBECONFIG`. It may also be necessary to [add additional contextual information](https://kubernetes.io/docs/reference/kubectl/generated/kubectl_config/kubectl_config_set-context/), such as the namespace.
5+
## Using Debug image in a Helm chart
66

7-
![Location of the Kubernetes plugin in the plugin bar (screen left), with the Clusters>cluster>Workloads>Pods views expanded out to show a pod named "my-service", overlaid with a dropdown box, with "Attach Visual Studio Code" highlighted](../images/debugging-kubernetes.jpg)
8-
The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted.
7+
⚠️ If running with the Diamond filesystem mounted or as a specific user, further adjustments are required, as described [below](#using-debug-image-in-a-helm-chart-that-mounts-the-filesystem).
98

10-
After the connection to the cluster has been established, it may be necessary to open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer.
9+
To use the debug image in a Helm chart can be as simple as modifying `image.tag` value in values.yaml to the tag with `-debug`, but this may run into issues if you have defined liveness or readiness probes, a custom command or args, or if the container is running as non-root. To make capturing these edge cases easier it's recommended to define a single flag `debug.enabled` in your `values.yaml` and make the following modifications to the `Deployment|ReplicaSet|StatefulSet`:
1110

12-
Starting your service with the command in the container definition starts it on the node, with access to Kubernetes resources, however it is also now possible to run with or attach a debugger, potentially configured to autoReload code, or to start and stop the service rapidly to implement prospective changes.
11+
```helm
12+
spec:
13+
template:
14+
spec:
15+
containers:
16+
- image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{ ternary "-debug" "" .Values.debug.enabled }}"
17+
{{- if not .Values.debug.enabled }} # If your Helm chart overrides the `CMD` Containerfile instruction, it should not when in debug mode
18+
args: ["some", "example", "args"]
19+
{{- end }}
20+
{{- if not .Values.debug.enabled }} # prevent probes causing issues before attaching and starting the service
21+
{{- with .Values.livenessProbe }}
22+
livenessProbe:
23+
{{- toYaml . | nindent 12 }}
24+
{{- end }}
25+
{{- with .Values.readinessProbe }}
26+
readinessProbe:
27+
{{- toYaml . | nindent 12 }}
28+
{{- end }}
29+
{{- end }}
30+
volumeMounts:
31+
{{- if .Values.debug.enabled }}
32+
- mountPath: /home # required for VSCode to install extensions if running as non-root
33+
name: home
34+
{{- end }}
35+
{{- with .Values.volumeMounts }}
36+
{{- toYaml . | nindent 12 }}
37+
{{- end }}
38+
volumes:
39+
{{- if .Values.debug.enabled }}
40+
- name: home # mount /home as an editable volume to prevent permission issues
41+
emptyDir:
42+
sizeLimit: 500Mi
43+
{{- end }}
44+
{{- with .Values.volumes }}
45+
{{- toYaml . | nindent 8 }}
46+
{{- end }}
47+
```
1348

14-
After you are happy with the changes, commit them and release a new version of your container. Changes will otherwise not be persisted across container restarts. Your git and ssh config will be mounted inside the devcontainer while connected and for containers on github, the remote `origin` will be configured to use ssh.
49+
## Using Debug image in a Helm chart that mounts the filesystem
1550

16-
## Debugging containers that run as non-root
17-
For containers running in the Diamond Kubernetes infrastructure that run as a specific uid (e.g. if mounting the filesystem), it is required to use a sidecar container to provide name resolution from Diamond's LDAP infrastructure and to mount a home directory to house vscode plugins.
51+
Containers running in the Diamond Kubernetes infrastructure as a specific uid (e.g. when mounting the filesystem) must provide name resolution from Diamond's LDAP infrastructure: inside the cluster the VSCode server will be running as that user, but requires that the name & home directory of the user can be found. The debug image configures the name lookup service to try finding the user internally (i.e. from `/etc/passwd`) then fall back to calling LDAP through a service called `libnss-ldapd`. As containers are designed to run a single process, this service is run in a sidecar container which must mutually mount the `/var/run/nslcd` socket with the primary container.
1852

19-
A sidecar for the Debian-based Python image this template uses is published as a container from this repository, the version should match the version of the python-copier-template you are using, to ensure compatibility with the underlying container infrastructure.
53+
It therefore requires the further additions to the template modified above:
2054

21-
```yaml
22-
- name: debug-account-sync
23-
image: ghcr.io/diamondlightsource/python-copier-template/account-sync:<version>
24-
volumeMounts:
25-
# The nslcd socket will be shared between the service and the sidecar
26-
- mountPath: /var/run/nslcd
27-
name: nslcd
55+
```helm
56+
spec:
57+
template:
58+
spec:
59+
containers:
60+
- volumeMounts:
61+
{{- if .Values.debug.enabled }}
62+
- mountPath: /var/run/nslcd # socket to place query for user information
63+
name: nslcd
64+
[...]
65+
{{- if .Values.debug.enabled }}
66+
- name: debug-account-sync
67+
image: ghcr.io/diamondlightsource/account-sync-sidecar:3.0.0
68+
volumeMounts:
69+
- mountPath: /var/run/nslcd # socket to pick queries for user information
70+
name: nslcd
71+
{{- end }}
72+
volumes:
73+
{{- if .Values.debug.enabled }}
74+
- name: nslcd # mutually mounted filesystem to both containers
75+
emptyDir:
76+
sizeLimit: 5Mi
77+
[...]
2878
```
2979

30-
The following changes/additions to your `values.yaml` may be required to connect vscode when using the sidecar.
31-
It is recommended to set the `HOME` environment variable on your container to be debugged to the same value used in the volume below.
32-
33-
```yaml
34-
volumes:
35-
- name: home # Required for vscode to start and install plugins
36-
hostPath:
37-
path: /home/<fedid>
38-
- name: nslcd # Shared volume between main and sidecar container
39-
emptyDir:
40-
sizeLimit: 500Mi
41-
42-
volumeMounts:
43-
- mountPath: /home/<fedid>
44-
name: home
45-
- mountPath: /var/run/nslcd
46-
name: nslcd
47-
48-
# Disable any liveness probe, as will not start service automatically
49-
livenessProbe: null
50-
readinessProbe: null
51-
52-
# Required to mount /home/, /dls/ etc.
53-
podSecurityContext:
54-
runAsUser: <uid of fedid>
55-
runAsGroup: <gid of fedid>
56-
57-
image:
58-
tag: "<version>-debug"
80+
# Debugging in the cluster
81+
82+
With the [Kubernetes plugin for VSCode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. From the VSCode Command Palette (Ctrl+Shift+P) use the `Kubernetes: Set Kubeconfig` to configure VSCode with the server to use, then`Kubernetes: Use Namespace`.
83+
84+
```sh
85+
# To find the KUBECONFIG to use from a Diamond machine
86+
$ module load pollux
87+
...
88+
$ echo $KUBECONFIG
89+
~/.kube/config_pollux
5990
```
91+
92+
![Location of the Kubernetes plugin in the plugin bar (screen left), with the Clusters>cluster>Workloads>Pods views expanded out to show a pod named "my-service", overlaid with a dropdown box, with "Attach Visual Studio Code" highlighted](../images/debugging-kubernetes.jpg)
93+
The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted.
94+
95+
After the connection to the cluster has been established open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer.
96+
97+
Starting your service with the command in the container definition starts it on the node, with access to Kubernetes resources, however it is also now possible to run with or attach a debugger, potentially configured to autoReload code, or to start and stop the service rapidly to implement prospective changes.
98+
99+
After you are happy with the changes, commit them and release a new version of your container. Changes will otherwise not be persisted across container restarts. Your git and ssh config will be mounted inside the devcontainer while connected and for containers on github, the remote `origin` will be configured to use ssh.

0 commit comments

Comments
 (0)