Skip to content

Commit 76c187c

Browse files
DiamondJosephDiamondJoseph
authored
feat: Publish debug container image and account-sync sidecar (#251)
Closes #234 Adds the option to create a -debug image when publishing a container from a project managed by the copier template and documentation on how to use the debug image to debug within the DLS cluster infrastructure.
1 parent 5294bde commit 76c187c

File tree

7 files changed

+193
-3
lines changed

7 files changed

+193
-3
lines changed

‎copier.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -103,6 +103,15 @@ docker:
103103
Would you like to publish your project in a Docker container?
104104
You should select this if you are making a service.
105105
106+
docker_debug:
107+
type: bool
108+
when: "{{ docker }}"
109+
help: |
110+
Would you like to publish a debug image of your service?
111+
This will increase the number of published images, but may
112+
be useful if debugging the service inside of the cluster
113+
infrastructure is required.
114+
106115
docs_type:
107116
type: str
108117
help: |

‎docs/how-to/debug-in-cluster.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Debugging containers
2+
3+
If the `docker_debug` option is chosen, the container build also publishes a debug container for each tagged release of the container suffixed with `-debug`. This container contains an editable install of the workspace & debugpy and has an alternate entrypoint which allows the devcontainer to attach.
4+
5+
# Using Debug image in a Helm chart
6+
7+
⚠️ If running with the Diamond filesystem mounted or as a specific user, further adjustments are required, as described in the next section.
8+
9+
To use the debug image in a Helm chart can be as simple as modifying `image.tag` value in values.yaml to the tag with `-debug`, but this may run into issues if you have defined liveness or readiness probes, a custom command or args, or if the container is running as non-root. To make capturing these edge cases easier it's recommended to define a single flag `debug.enabled` in your `values.yaml` and make the following modifications to the `Deployment|ReplicaSet|StatefulSet`:
10+
11+
```yaml
12+
spec:
13+
template:
14+
spec:
15+
containers:
16+
- image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}{{ ternary "-debug" "" .Values.debug.enabled }}"
17+
{{- if not .Values.debug.enabled }} # If your Helm chart overrides the `CMD` Containerfile instruction, it should not when in debug mode
18+
args: ["some", "example", "args"]
19+
{{- end }}
20+
{{- if not .Values.debug.enabled }} # prevent probes causing issues before attaching and starting the service
21+
{{- with .Values.livenessProbe }}
22+
livenessProbe:
23+
{{- toYaml . | nindent 12 }}
24+
{{- end }}
25+
{{- with .Values.readinessProbe }}
26+
readinessProbe:
27+
{{- toYaml . | nindent 12 }}
28+
{{- end }}
29+
{{- end }}
30+
volumeMounts:
31+
{{- if .Values.debug.enabled }}
32+
- mountPath: /home # required for VSCode to install extensions if running as non-root
33+
name: home
34+
{{- end }}
35+
{{- with .Values.volumeMounts }}
36+
{{- toYaml . | nindent 12 }}
37+
{{- end }}
38+
volumes:
39+
{{- if .Values.debug.enabled }}
40+
- name: home # mount /home as an editable volume to prevent permission issues
41+
emptyDir:
42+
sizeLimit: 500Mi
43+
{{- end }}
44+
{{- with .Values.volumes }}
45+
{{- toYaml . | nindent 8 }}
46+
{{- end }}
47+
```
48+
49+
# Using Debug image in a Helm chart that mounts the filesystem
50+
51+
Containers running in the Diamond Kubernetes infrastructure as a specific uid (e.g. when mounting the filesystem) must provide name resolution from Diamond's LDAP infrastructure: inside the cluster the VSCode server will be running as that user, but requires that the name & home directory of the user can be found. The debug image configures the name lookup service to try finding the user internally (i.e. from `/etc/passwd`) then fall back to calling LDAP through a service called `libnss-ldapd`. As containers are designed to run a single process, this service is run in a sidecar container which must mutually mount the `/var/run/nslcd` socket with the primary container.
52+
53+
It therefore requires the further additions to the template modified above:
54+
55+
```yaml
56+
spec:
57+
template:
58+
spec:
59+
containers:
60+
- volumeMounts:
61+
{{- if .Values.debug.enabled }}
62+
- mountPath: /var/run/nslcd # socket to place query for user information
63+
name: nslcd
64+
[...]
65+
{{- if .Values.debug.enabled }}
66+
- name: debug-account-sync
67+
image: ghcr.io/diamondlightsource/account-sync-sidecar:3.0.0
68+
volumeMounts:
69+
- mountPath: /var/run/nslcd # socket to pick queries for user information
70+
name: nslcd
71+
{{- end }}
72+
volumes:
73+
{{- if .Values.debug.enabled }}
74+
- name: nslcd # mutually mounted filesystem to both containers
75+
emptyDir:
76+
sizeLimit: 5Mi
77+
[...]
78+
```
79+
80+
# Debugging in the cluster
81+
82+
With the [Kubernetes plugin for VSCode](https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools) it is then possible to attach to the container inside the cluster. From the VSCode Command Palette (Ctrl+Shift+P) use the `Kubernetes: Set Kubeconfig` to configure VSCode with the server to use, then`Kubernetes: Use Namespace`.
83+
84+
```sh
85+
# To find the KUBECONFIG to use from a Diamond machine
86+
$ module load pollux
87+
...
88+
$ echo $KUBECONFIG
89+
~/.kube/config_pollux
90+
```
91+
92+
![Location of the Kubernetes plugin in the plugin bar (screen left), with the Clusters>cluster>Workloads>Pods views expanded out to show a pod named "my-service", overlaid with a dropdown box, with "Attach Visual Studio Code" highlighted](../images/debugging-kubernetes.jpg)
93+
The Kubernetes plugin can be found in the plugin bar. Expanding the Clusters>`cluster`>Workloads>Pods views, your service should be visible. Right Click>Attach Visual Studio Code will initiate connecting to the workspace in the cluster. Select your service container from the top menu when prompted.
94+
95+
After the connection to the cluster has been established open the workspace folder by clicking the Explorer option in the plugin bar, the repository will be mounted at `/workspaces/<service name>`, equivalent to when working with a local devcontainer.
96+
97+
Starting your service with the command in the container definition starts it on the node, with access to Kubernetes resources, however it is also now possible to run with or attach a debugger, potentially configured to autoReload code, or to start and stop the service rapidly to implement prospective changes.
98+
99+
After you are happy with the changes, commit them and release a new version of your container. Changes will otherwise not be persisted across container restarts. Your git and ssh config will be mounted inside the devcontainer while connected and for containers on github, the remote `origin` will be configured to use ssh.

‎docs/images/debugging-kubernetes.jpg‎

53.8 KB
Loading

‎example-answers.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ component_lifecycle: experimental
66
description: An expanded https://github.com/DiamondLightSource/python-copier-template to illustrate how it looks with all the options enabled.
77
distribution_name: dls-python-copier-template-example
88
docker: true
9+
docker_debug: true
910
docs_type: sphinx
1011
git_platform: github.com
1112
github_org: DiamondLightSource

‎template/Dockerfile.jinja‎

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,33 @@ ENV PATH=/venv/bin:$PATH{% if docker %}
1414

1515
# The build stage installs the context into the venv
1616
FROM developer AS build
17-
COPY . /context
18-
WORKDIR /context
17+
# Requires buildkit 0.17.0
18+
COPY --chmod=o+wrX . /workspaces/{{ repo_name }}
19+
WORKDIR /workspaces/{{ repo_name }}
1920
RUN touch dev-requirements.txt && pip install -c dev-requirements.txt .
2021

22+
{% if docker_debug %}
23+
FROM build AS debug
24+
25+
{% if git_platform=="github.com" %}
26+
# Set origin to use ssh
27+
RUN git remote set-url origin [email protected]:{{github_org}}/{{repo_name}}.git
28+
{% endif %}
29+
30+
# For this pod to understand finding user information from LDAP
31+
RUN apt update
32+
RUN DEBIAN_FRONTEND=noninteractive apt install libnss-ldapd -y
33+
RUN sed -i 's/files/ldap files/g' /etc/nsswitch.conf
34+
35+
# Make editable and debuggable
36+
RUN pip install debugpy
37+
RUN pip install -e .
38+
39+
# Alternate entrypoint to allow devcontainer to attach
40+
ENTRYPOINT [ "/bin/bash", "-c", "--" ]
41+
CMD [ "while true; do sleep 30; done;" ]
42+
43+
{% endif %}
2144
# The runtime stage copies the built venv into a slim runtime container
2245
FROM python:${PYTHON_VERSION}-slim AS runtime
2346
# Add apt-get system dependecies for runtime here if needed

‎template/{% if git_platform=="github.com" %}.github{% endif %}/workflows/ci.yml.jinja‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,16 @@ jobs:
4141
permissions:
4242
contents: read
4343
packages: write
44-
{% endraw %}{% endif %}{% if sphinx %}
44+
{% endraw %}{% if docker_debug %}{% raw %}
45+
debug_container:
46+
needs: [container, test]
47+
uses: ./.github/workflows/_debug_container.yml
48+
with:
49+
publish: ${{ needs.test.result == 'success' }}
50+
permissions:
51+
contents: read
52+
packages: write
53+
{% endraw %}{% endif %}{% endif %}{% if sphinx %}
4554
docs:
4655
uses: ./.github/workflows/_docs.yml
4756

‎template/{% if git_platform=="github.com" %}.github{% endif %}/workflows/{% if docker_debug %}_debug_container.yml{% endif %}‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
on:
2+
workflow_call:
3+
inputs:
4+
publish:
5+
type: boolean
6+
description: If true, pushes image to container registry
7+
8+
jobs:
9+
build:
10+
runs-on: ubuntu-latest
11+
12+
steps:
13+
- name: Checkout
14+
uses: actions/checkout@v4
15+
with:
16+
# Need this to get version number from last tag
17+
fetch-depth: 0
18+
19+
- name: Set up Docker Buildx
20+
id: buildx
21+
uses: docker/setup-buildx-action@v3
22+
23+
- name: Log in to GitHub Docker Registry
24+
if: github.event_name != 'pull_request'
25+
uses: docker/login-action@v3
26+
with:
27+
registry: ghcr.io
28+
username: ${{ github.actor }}
29+
password: ${{ secrets.GITHUB_TOKEN }}
30+
31+
- name: Create tags for publishing debug image
32+
id: debug-meta
33+
uses: docker/metadata-action@v5
34+
with:
35+
images: ghcr.io/${{ github.repository }}
36+
tags: |
37+
type=ref,event=tag,suffix=-debug
38+
type=raw,value=latest-debug
39+
40+
- name: Build and publish debug image to container registry
41+
if: github.ref_type == 'tag'
42+
uses: docker/build-push-action@v6
43+
env:
44+
DOCKER_BUILD_RECORD_UPLOAD: false
45+
with:
46+
context: .
47+
push: true
48+
target: debug
49+
tags: ${{ steps.debug-meta.outputs.tags }}

0 commit comments

Comments
 (0)