Skip to content

Commit a85ec95

Browse files
Sebbo92Sebbo92
committed
Merge branch 'feature/draft-sebbo'
2 parents b6fc524 + 1fb9f14 commit a85ec95

14 files changed

+495
-2
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
__debug_bin

‎.vscode/launch.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"version": "0.2.0",
3+
"configurations": [
4+
{
5+
"name": "Remote debug in Kubernetes",
6+
"type": "go",
7+
"request": "attach",
8+
"mode":"remote",
9+
"remotePath": "/go/src/github.com/setlog/debug-k8s",
10+
"port": 30123,
11+
"host": "127.0.0.1",
12+
"showLog": true
13+
}
14+
]
15+
}

‎Dockerfile‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
FROM golang:1.13-alpine
2+
3+
ENV CGO_ENABLED=0
4+
ENV GOROOT=/usr/local/go
5+
ENV GOPATH=${HOME}/go
6+
ENV PATH=$PATH:${GOROOT}/bin
7+
8+
RUN apk update && apk add --no-cache \
9+
git && \
10+
go get github.com/go-delve/delve/cmd/dlv
11+
12+
WORKDIR /go/src/github.com/setlog/debug-k8s
13+
EXPOSE 30123 8090
14+
15+
ENTRYPOINT ["/go/bin/dlv", "debug", ".", "--listen=:30123", "--accept-multiclient", "--headless=true", "--api-version=2"]

‎README.md‎

Lines changed: 324 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,324 @@
1-
# debug-k8s
2-
How to debug a go-service in kubernetes
1+
# Preface
2+
3+
In a perfect world every written service will work smooth, your test coverage is on top and there are no bugs in the implementation of your API. But we all know, that we can't achieve this world, sadly. It's not unusual that there's a bug and you have to debug this problem in a production environment. We have faced this problem with our go services in our Kubernetes cluster, and we want to show you how it's possible to remote debug a go service in a Kubernetes cluster.
4+
5+
## Software Prerequisites
6+
7+
For this case we create a kubernetes cluster running locally on our system. Therefore we need the following software:
8+
9+
* [Docker Desktop](https://docs.docker.com/get-docker) (used version: 19.03.8)
10+
* [kind (Kubernetes in Docker)](https://kind.sigs.k8s.io) (used version: v0.7.0)
11+
* [Kubectl](https://kubernetes.io/de/docs/tasks/tools/install-kubectl) (used version: 1.17.2)
12+
* [Visual Studio Code](https://code.visualstudio.com/download) (used version: 1.32.3)
13+
14+
We decided to use `kind` instead of `minikube`, since it's a very good tool for testing Kubernetes locally, and we can use our docker images without a docker registry.
15+
16+
## Big Picture
17+
18+
First we will briefly explain how it works. We start by creating a new Kubernetes cluster `local-debug-k8s` on our local system.
19+
20+
* You need a docker container with [delve](https://github.com/go-delve/delve) (the go debugger) as the main process.
21+
* The debugger delve needs access to the path with the project data. This is done by mounting `$GOPATH/src` on the pod which is running in the Kubernetes cluster.
22+
* We start the delve container on port 30123 and bind this port to localhost, so that only our local debugger can communicate with delve.
23+
* To debug an API with delve, it's necessary to set up an ingress network. For this we use port 8090.
24+
25+
A picture serves to illustrate the communication:
26+
27+
![Overview](images/big-picture.png "Big Picture")
28+
29+
### Creating a Kubernetes cluster
30+
31+
`kind` unfortunately doesn't use the environment variable `GOPATH`, so we have to update this in [config.yaml](cluster/config.yaml#L21):
32+
33+
```sh
34+
sed -i.bak 's|'{GOPATH}'|'${GOPATH}'|g' cluster/config.yaml
35+
```
36+
37+
You can also open [config.yaml](cluster/config.yaml#L21) and replace `{GOPATH}` with the absolute path manually. If you already installed kind (Kubernetes in Docker) on your local system, you can create the cluster with this command:
38+
39+
```sh
40+
kind create cluster --config cluster/config.yaml --name=local-debug-k8s
41+
```
42+
43+
Ensure that port 8090 and 30123 are not used on your local system. The newly created cluster has the name `local-debug-k8s` and has been created with custom configuration ( `--config cluster/config.yaml`). The following is a brief explanation:
44+
45+
```yml
46+
kind: Cluster
47+
apiVersion: kind.x-k8s.io/v1alpha4
48+
nodes:
49+
- role: control-plane
50+
kubeadmConfigPatches: # necessary, since we are going to install an ingress network in the cluster
51+
- |
52+
kind: InitConfiguration
53+
nodeRegistration:
54+
kubeletExtraArgs:
55+
node-labels: "ingress-ready=true"
56+
authorization-mode: "AlwaysAllow"
57+
extraPortMappings:
58+
- containerPort: 80 # http endpoint of ingress runs on the port 80
59+
hostPort: 8090 # port on your host machine to call API's of the service
60+
protocol: TCP
61+
- containerPort: 30123 # node port for the delve server
62+
hostPort: 30123 # port on your host machine to communicate with the delve server
63+
protocol: TCP
64+
- role: worker
65+
extraMounts:
66+
- hostPath: {GOPATH}/src # ATTENTION: you might want to replace this path with your ${GOPATH}/src manually
67+
containerPath: /go/src # path to the project folder inside the worker node
68+
```
69+
70+
Expected result:
71+
72+
```sh
73+
Creating cluster "local-debug-k8s" ...
74+
✓ Ensuring node image (kindest/node:v1.17.0) 🖼
75+
✓ Preparing nodes 📦 📦
76+
✓ Writing configuration 📜
77+
✓ Starting control-plane 🕹️
78+
✓ Installing CNI 🔌
79+
✓ Installing StorageClass 💾
80+
✓ Joining worker nodes 🚜
81+
82+
Set kubectl context to "kind-local-debug-k8s"
83+
You can now use your cluster with:
84+
85+
kubectl cluster-info --context kind-local-debug-k8s
86+
87+
Have a nice day! 👋
88+
```
89+
90+
Activate the kube-context for `kubectl` to communicate with the new cluster:
91+
92+
```sh
93+
kubectl cluster-info --context kind-local-debug-k8s
94+
```
95+
96+
#### Install nginx-ingress
97+
98+
For both ports (8090 and 30123) to work, it is necessary to deploy an nginx controller as an ingress controller:
99+
100+
```sh
101+
kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml
102+
```
103+
104+
Source: [kind documentation](https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx>)
105+
106+
to observe the current status the following command can be executed:
107+
108+
```sh
109+
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s
110+
```
111+
112+
#### Labelling the node
113+
114+
We know that by default a kubernetes cluster will deploy a pod on a node which has enough ressources for this workload. Our docker image must be pulled on all nodes in our kubernetes cluster in order to be ready as quickly as possible. This process may take a long time. If the docker image isn't pulled on a node and a new pod will provisioned on this node, it will take more time to get ready and healthy.
115+
For our use case we will label a node in our kubernetes cluster so that always this node will be used.
116+
117+
We label a node with _debug=true_:
118+
119+
```sh
120+
kubectl label nodes local-debug-k8s-worker debug=true
121+
```
122+
123+
### Creating a docker image
124+
125+
Our service has only one endpoint `/hello` and writes just a few logs. Let's checkout the Dockerfile for delve:
126+
127+
```Dockerfile
128+
FROM golang:1.13-alpine
129+
130+
# compile gcc statically
131+
ENV CGO_ENABLED=0
132+
ENV GOROOT=/usr/local/go
133+
# this path will be mounted in deploy-service.yaml
134+
ENV GOPATH=${HOME}/go
135+
ENV PATH=$PATH:${GOROOT}/bin
136+
137+
# Install git and get the latest version of delve via go
138+
RUN apk update && apk add --no-cache \
139+
git && \
140+
go get github.com/go-delve/delve/cmd/dlv
141+
142+
# ATTENTION: you want to check, if the path to the project folder is the right one here
143+
WORKDIR /go/src/github.com/setlog/debug-k8s
144+
145+
# 30123 for delve and 8090 for API calls
146+
EXPOSE 30123 8090
147+
148+
# let's start delve as the entrypoint
149+
ENTRYPOINT ["/go/bin/dlv", "debug", ".", "--listen=:30123", "--accept-multiclient", "--headless=true", "--api-version=2"]
150+
```
151+
152+
So, let's build build our docker image from our [Dockerfile](Dockerfile):
153+
154+
```sh
155+
docker build -t setlog/debug-k8s ./Dockerfile
156+
```
157+
158+
After the build is done, we load the image `setlog/debug-k8s:latest` on the node _local-debug-k8s-worker_:
159+
160+
```sh
161+
kind load docker-image setlog/debug-k8s:latest --name=local-debug-k8s --nodes=local-debug-k8s-worker
162+
```
163+
164+
A message appears indicating that the docker image did not exist before:
165+
166+
```sh
167+
Image: "setlog/debug-k8s:latest" with ID "sha256:944baa03d49698b9ca1f22e1ce87b801a20ce5aa52ccfc648a6c82cf8708a783" not present on node "local-debug-k8s-worker"
168+
```
169+
170+
### Deploy the delve container in our cluster
171+
172+
First of all we need a persistent volume and its claim in order to mount the project path into the node:
173+
174+
```sh
175+
kubectl create -f cluster/persistent-volume.yaml
176+
```
177+
178+
The interesting part here is:
179+
180+
```yaml
181+
hostPath:
182+
path: /go/src
183+
```
184+
185+
Below is an image that shows the configurations through which our local path is mounted. In your environment it could be another path:
186+
187+
![Mounting](images/mounting.png "How to mount the project folder")
188+
189+
Check, if your persistent volume claim has been successfully created (STATUS must be Bound):
190+
191+
```sh
192+
kubectl get pvc
193+
194+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
195+
go-pvc Bound go-pv 256Mi RWO hostpath 51s
196+
```
197+
198+
Now we are ready to deploy all our services in debug mode:
199+
200+
```sh
201+
kubectl create -f cluster/deploy-service.yaml
202+
```
203+
204+
Let's go through the pod manifest:
205+
206+
* _image_ is the previously built and loaded image into the kind cluster with `kind load image...`
207+
* _imagePullPolicy_ must be set to _IfNotPresent_ because it's already loaded and we don't want Kubernetes to try it again
208+
209+
```yaml
210+
image: setlog/debug-k8s:latest
211+
imagePullPolicy: IfNotPresent
212+
```
213+
214+
* We use the persistent volume claim to mount the project path into the pod, so that `/go/src` will be linked to `${GOPATH}/src` on your local system
215+
216+
```yaml
217+
containers:
218+
- name: debug-k8s
219+
...
220+
volumeMounts:
221+
- mountPath: /go/src
222+
name: go-volume
223+
volumes:
224+
- name: go-volume
225+
persistentVolumeClaim:
226+
claimName: go-pvc
227+
```
228+
229+
* As there might be several nodes in your kubernetes cluster, we deploy the pod on the node, that is labelled with _debug=true_. The docker image _setlog/debug-k8s_ was already loaded on this node.
230+
231+
```yaml
232+
nodeSelector:
233+
debug: "true"
234+
```
235+
236+
* Service _service-debug_ has the type _NodePort_ and is mounted to the node. This port 30123 is equal to the parameter _--listen=:30123_ in the Dockerfile, which makes it possible to send debug commands to the delve server.
237+
238+
* Service _debug-k8s_ will be connected to the ingress server in the final step. It serves for exposing the API endpoints we are going to debug.
239+
240+
If you did all steps correctly, your pod should be up and running. Check it with `kubectl get pod`. You should see the output with the pod status _Running_ and two additional services _debug-k8s_ and _service-debug_:
241+
242+
```sh
243+
NAME READY STATUS RESTARTS AGE
244+
pod/debug-k8s-6d69b65cf-4fl6t 1/1 Running 0 1h
245+
246+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
247+
service/debug-k8s ClusterIP 10.96.80.193 <none> 8090/TCP 1h
248+
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
249+
service/service-debug NodePort 10.96.219.86 <none> 30123:30123/TCP 1h
250+
```
251+
252+
_Hint: create a new variable to store the pod name using `PODNAME=$(kubectl get pod -o jsonpath='{.items[0].metadata.name}')`. It can be helpful, if you repeatedly debug the pod._
253+
254+
Usually it takes a couple of seconds to start the debugging process with delve. If your paths are mounted in the proper way, you will find the file `__debug_bin` in the project path on your computer. That is an executable which has been created by delve.
255+
256+
Also, you can output logs of the pod by performing `kubectl logs $PODNAME` in order to make sure the delve API server is listening at 30123.
257+
258+
Output:
259+
260+
```sh
261+
API server listening at: [::]:30123
262+
```
263+
264+
_Hint: always wait until this log message is shown for this pod before you start the debugging process. Otherwise, the delve server is not up yet and cannot answer to the debugger._
265+
266+
### Starting the debug process via launch.json
267+
268+
Now we need a debug configuration in Visual Studio Code. This can be done in `.vscode/launch.json`:
269+
270+
```json
271+
{
272+
"version": "0.2.0",
273+
"configurations": [
274+
{
275+
"name": "Remote debug in Kubernetes",
276+
"type": "go",
277+
"request": "attach",
278+
"mode":"remote",
279+
"remotePath": "/go/src/github.com/setlog/debug-k8s",
280+
"port": 30123,
281+
"host": "127.0.0.1",
282+
"showLog": true
283+
}
284+
]
285+
}
286+
```
287+
288+
Where `remotePath` is the path to the project path inside the pod, `port` the local port to send the debug commands to, and `host` the host to send the debug commands to.
289+
290+
You can find the new configuration in Visual Studio Code here:
291+
292+
![Debug Configuration](images/debug-config.png "Where to find the debug config")
293+
294+
After starting the debug process there is a new log created by the go service:
295+
296+
```sh
297+
2020/05/28 15:38:53 I am going to start...
298+
```
299+
300+
Finally we are ready to debug the service, but we have to trigger the API functions through the ingress service. Deploy it with kubectl:
301+
302+
```sh
303+
kubectl create -f cluster/ingress.yaml
304+
```
305+
306+
And try accessing it now:
307+
308+
```sh
309+
curl http://localhost:8090/hello
310+
```
311+
312+
Which should trigger the debugger:
313+
314+
![Breakpoint](images/debug-screen.png "Breakpoint in Visual Studio Code")
315+
316+
Happy debugging!
317+
318+
### Cleaning up
319+
320+
If you don't need your kind cluster anymore, you can remove this with the following command:
321+
322+
```sh
323+
kind delete cluster --name=local-debug-k8s
324+
```

‎cluster/config.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
kind: Cluster
2+
apiVersion: kind.x-k8s.io/v1alpha4
3+
nodes:
4+
- role: control-plane
5+
kubeadmConfigPatches:
6+
- |
7+
kind: InitConfiguration # necessary, since we are going to install an ingress network in the cluster
8+
nodeRegistration:
9+
kubeletExtraArgs:
10+
node-labels: "ingress-ready=true"
11+
authorization-mode: "AlwaysAllow"
12+
extraPortMappings:
13+
- containerPort: 80 # http endpoint of ingress runs on the port 80
14+
hostPort: 8090 # port on your host machine to call API's of the service
15+
protocol: TCP
16+
- containerPort: 30123 # node port for the delve server
17+
hostPort: 30123 # port on your host machine to communicate with the delve server
18+
protocol: TCP
19+
- role: worker
20+
extraMounts:
21+
- hostPath: {GOPATH}/src # ATTENTION: you might want to change this path to your ${GOPATH}/src
22+
containerPath: /go/src # path to the project folder inside the worker node

0 commit comments

Comments
 (0)