Skip to content

Commit 63ad93a

Browse files
committed
Merge branch 'feature/draft-sebbo' of github.com:setlog/debug-k8s into feature/draft-sebbo
2 parents 4d4ca72 + f155084 commit 63ad93a

File tree

1 file changed

+83
-54
lines changed

1 file changed

+83
-54
lines changed

README.md

Lines changed: 83 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,37 @@
11
# Preface
22

3-
In a perfect world every written service will work smooth, your test coverage is on top and there are no bugs in your API. But we all know, that we can´t achieve this world, sadly. It´s not unusual that there´s a bug in an API and you have to debug this problem in a production environment. We have faced this problem with our go services in our kubernetes cluster and we want to show you how it´s possible to remote debug a go service in a kubernetes cluster.
3+
In a perfect world every written service will work smooth, your test coverage is on top and there are no bugs in your API. But we all know, that we can't achieve this world, sadly. It's not unusual that there's a bug in an API and you have to debug this problem in a production environment. We have faced this problem with our go services in our Kubernetes cluster, and we want to show you how it's possible to remote debug a go service in a Kubernetes cluster.
44

55
## Software Prerequisites
66

77
For this scenario we need some software:
88

9-
* Docker Desktop: <https://docs.docker.com/get-docker/> (used version: 19.03.8)
10-
* kind (Kubernetes in Docker): <https://kind.sigs.k8s.io.> (used version: v0.7.0)
11-
* Kubectl: <https://kubernetes.io/de/docs/tasks/tools/install-kubectl/> (used version: 1.17.2)
12-
* Visual Studio Code: <https://code.visualstudio.com/download> (used version: 1.32.3)
9+
* [Docker Desktop](https://docs.docker.com/get-docker) (used version: 19.03.8)
10+
* [kind (Kubernetes in Docker)](https://kind.sigs.k8s.io) (used version: v0.7.0)
11+
* [Kubectl](https://kubernetes.io/de/docs/tasks/tools/install-kubectl) (used version: 1.17.2)
12+
* [Visual Studio Code](https://code.visualstudio.com/download) (used version: 1.32.3)
1313

14-
We decided to use `kind` instead of `minikube`, since it´s a very good tool for testing kubernetes locally and we can use our docker images without a docker registry.
14+
We decided to use `kind` instead of `minikube`, since it's a very good tool for testing Kubernetes locally, and we can use our docker images without a docker registry.
1515

1616
## Big Picture
1717

18-
First we will briefly explain how it works:
18+
First we will briefly explain how it works. We start by creating a new Kubernetes cluster `local-debug-k8s` on our local system.
1919

20-
1. We create a new kubernetes cluster `local-debug-k8s` on our local system
21-
22-
* you need a docker container with delve (<https://github.com/go-delve/delve>) as main process
23-
* delve needs access to the path with the project data. This is done by mounting `$GOPATH/src` on the pod which is running in the kubernetes cluster
24-
* we start the delve container on port 30123 and bind this port to localhost, so that only our local debugger can communicate with delve
25-
* to debug an API with delve, it´s necessary to set up an ingress network. For this we use port 8090.
20+
* You need a docker container with [delve](https://github.com/go-delve/delve) (the go debugger) as the main process.
21+
* The debugger delve needs access to the path with the project data. This is done by mounting `$GOPATH/src` on the pod which is running in the Kubernetes cluster.
22+
* We start the delve container on port 30123 and bind this port to localhost, so that only our local debugger can communicate with delve.
23+
* To debug an API with delve, it's necessary to set up an ingress network. For this we use port 8090.
2624

2725
A picture serves to illustrate the communication:
2826

2927
![Overview](images/big-picture.png "Big Picture")
3028

3129
### Creating a Kubernetes cluster
3230

33-
`kind` unfortunately doesn´t use the environment variable `GOPATH`, so we have to update this in [config.yaml](cluster/config.yaml#L21):
31+
`kind` unfortunately doesn't use the environment variable `GOPATH`, so we have to update this in [config.yaml](cluster/config.yaml#L21):
3432

3533
```sh
36-
`sed -i.bak 's|'{GOPATH}'|'${GOPATH}'|g' cluster/config.yaml`
34+
sed -i.bak 's|'{GOPATH}'|'${GOPATH}'|g' cluster/config.yaml
3735
```
3836

3937
You can also open [config.yaml](cluster/config.yaml#L21) and replace `{GOPATH}` with the absolute path manually. If you already installed kind (Kubernetes in Docker) on your local system, you can create the cluster with this command:
@@ -69,7 +67,7 @@ nodes:
6967
containerPath: /go/src # path to the project folder inside the worker node
7068
```
7169
72-
Desired result:
70+
Expected result:
7371
7472
```sh
7573
Creating cluster "local-debug-k8s" ...
@@ -97,27 +95,29 @@ Activate the kube-context for `kubectl` to communicate with the new cluster:
9795

9896
#### Install nginx-ingress
9997

100-
For both ports (8090 and 30123) to work it´s necessary to deploy a nginx controller:
98+
For both ports (8090 and 30123) to work, it is necessary to deploy an nginx controller:
10199

102100
```sh
103101
kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml
104102
```
105103

106-
Source: <https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx>
104+
Source: [kind documentation](https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx>)
107105

108-
...and wait until nginx-controller runs:
106+
And wait until the nginx-controller runs:
109107

110108
```sh
111109
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s
112110
```
113111

114112
#### Labelling the worker node
115113

116-
We would suggest to label a worker node where the pod is going to be deployed: by default, a pod is deployed on one of several worker nodes you might have in the kind cluster. To make it work the docker image must be populated on all worker nodes in the cluster (it takes time). Otherwise, you can get into a situation, in which the pod is started on a node where the docker image is missing. Let's work with a dedicated node and safe the time.
114+
We suggest labelling a worker node where the pod is going to be deployed. By default, a pod is deployed on one of several worker nodes you might have in the kind cluster. To make it work, the docker image must be populated on all worker nodes in the cluster (which takes time). Otherwise, you can get into a situation, in which the pod has started on a node where the docker image is missing. Let's work with a dedicated node and safe the time.
117115

118116
So, we label a worker node with _debug=true_:
119117

120-
`kubectl label nodes local-debug-k8s-worker debug=true`
118+
```sh
119+
kubectl label nodes local-debug-k8s-worker debug=true
120+
```
121121

122122
### Creating a docker image
123123

@@ -126,15 +126,18 @@ Our service has only one `/hello` endpoint and writes just a few logs. The inter
126126
```Dockerfile
127127
FROM golang:1.13-alpine
128128

129-
ENV CGO_ENABLED=0 # compile gcc statically
129+
# compile gcc statically
130+
ENV CGO_ENABLED=0
130131
ENV GOROOT=/usr/local/go
131-
ENV GOPATH=${HOME}/go # this path will be mounted in deploy-service.yaml later
132+
# this path will be mounted in deploy-service.yaml later
133+
ENV GOPATH=${HOME}/go
132134
ENV PATH=$PATH:${GOROOT}/bin
133135

134136
EXPOSE 30123 # for delve
135137
EXPOSE 8090 # for API calls
136138

137-
WORKDIR /go/src/github.com/setlog/debug-k8s # ATTENTION: you want to check, if the path to the project folder is the right one here
139+
# ATTENTION: you want to check, if the path to the project folder is the right one here
140+
WORKDIR /go/src/github.com/setlog/debug-k8s
138141

139142
# Install delve, our version is 1.4.1
140143
RUN apk update && apk add git && \
@@ -144,23 +147,31 @@ RUN apk update && apk add git && \
144147
ENTRYPOINT ["/go/bin/dlv", "debug", ".", "--listen=:30123", "--accept-multiclient", "--headless=true", "--api-version=2"]
145148
```
146149

147-
Build the docker image locally, first:
150+
First, build the docker image locally:
148151

149-
`docker build -t setlog/debug-k8s .`
152+
```sh
153+
docker build -t setlog/debug-k8s .
154+
```
150155

151-
Load the docker image into the node _local-debug-k8s-worker_
156+
Load the docker image into the node _local-debug-k8s-worker_:
152157

153-
`kind load docker-image setlog/debug-k8s:latest --name=local-debug-k8s --nodes=local-debug-k8s-worker`
158+
```sh
159+
kind load docker-image setlog/debug-k8s:latest --name=local-debug-k8s --nodes=local-debug-k8s-worker
160+
```
154161

155-
This message will be shown and it is just saying that the image was not there:
162+
This message will be shown, and it is just saying that the image was not there:
156163

164+
```
157165
Image: "setlog/debug-k8s:latest" with ID "sha256:944baa03d49698b9ca1f22e1ce87b801a20ce5aa52ccfc648a6c82cf8708a783" not present on node "local-debug-k8s-worker"
166+
```
158167

159168
### Starting the delve server in the cluster
160169

161170
Now we want to create a persistent volume and its claim in order to mount the project path into the worker node:
162171

163-
`kubectl create -f cluster/persistent-volume.yaml`
172+
```sh
173+
kubectl create -f cluster/persistent-volume.yaml
174+
```
164175

165176
The interesting part here is:
166177

@@ -169,30 +180,37 @@ The interesting part here is:
169180
path: /go/src
170181
```
171182

172-
Lets take a look at the full chain of mounting the local project path into the pod, since you want probably to adjust them to your environment:
183+
Let's take a look at the full chain of mounting the local project path into the pod, since you want probably to adjust them to your environment:
173184

174185
![Mounting](images/mounting.png "How to mount the project folder")
175186

176187
Check, if your persistent volume claim has been successfully created (STATUS must be Bound):
177188

178-
`kubectl get pvc`
189+
```sh
190+
kubectl get pvc
179191

180192
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
181193
go-pvc Bound go-pv 256Mi RWO hostpath 51s
194+
```
182195

183-
You are ready to start the service in the debug mode:
196+
You are ready to start the service in debug mode:
184197

185-
`kubectl create -f cluster/deploy-service.yaml`
198+
```sh
199+
kubectl create -f cluster/deploy-service.yaml
200+
```
186201

187202
Let's go through the deployment.
188203

189-
* Image name is what we loaded into the kind cluster with the command `kind load image...`. _imagePullPolicy_ must be set to _IfNotPresent_, because it is already loaded there and we don't want kubernetes to try doing it once more.
204+
* Image name is what we loaded into the kind cluster with the command `kind load image...`. _imagePullPolicy_ must be set to _IfNotPresent_, because it is already loaded, we don't want Kubernetes to try loading it again.
190205

191-
image: setlog/debug-k8s:latest
192-
imagePullPolicy: IfNotPresent
206+
```yaml
207+
image: setlog/debug-k8s:latest
208+
imagePullPolicy: IfNotPresent
209+
```
193210
194211
* We use the persistent volume claim to mount the project path into the pod and make `/go/src` to be linked with `${GOPATH}/src` on your computer:
195212

213+
```yaml
196214
containers:
197215
- name: debug-k8s
198216
...
@@ -203,19 +221,22 @@ Let's go through the deployment.
203221
- name: go-volume
204222
persistentVolumeClaim:
205223
claimName: go-pvc
224+
````
206225
207226
* As there might be several workers in your cluster, we deploy the pod on the one, that is labelled with _debug=true_. The docker image _setlog/debug-k8s_ has been loaded earlier in it already.
208227
228+
```yaml
209229
nodeSelector:
210230
debug: "true"
231+
```
211232
212-
* Service _service-debug_ has the type _NodePort_ and is mounted into the worker node. This port 30123 is equal to the parameter _--listen=:30123_ in the Dockerfile, what makes possible to send debug commands to the delve server.
233+
* Service _service-debug_ has the type _NodePort_ and is mounted into the worker node. This port 30123 is equal to the parameter _--listen=:30123_ in the Dockerfile, which makes it possible to send debug commands to the delve server.
213234
214235
* Service _debug-k8s_ will be connected to the ingress server in the final step. It serves for exposing the API endpoints we are going to debug.
215236
216237
If you did all steps correctly, your pod should be up and running. Check it with `kubectl get pod`. You should see the output with the pod status _Running_ and two additional services _debug-k8s_ and _service-debug_:
217238

218-
```sh
239+
```
219240
NAME READY STATUS RESTARTS AGE
220241
pod/debug-k8s-6d69b65cf-4fl6t 1/1 Running 0 1h
221242

@@ -225,24 +246,24 @@ service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP
225246
service/service-debug NodePort 10.96.219.86 <none> 30123:30123/TCP 1h
226247
```
227248
228-
_Hint: create a new variable to store the pod name. It can be helpful, if you repeatedly debug the pod_
229-
`PODNAME=$(kubectl get pod -o jsonpath='{.items[0].metadata.name}')`
249+
_Hint: create a new variable to store the pod name using `PODNAME=$(kubectl get pod -o jsonpath='{.items[0].metadata.name}')`. It can be helpful, if you repeatedly debug the pod._
230250
231-
Usualy it takes a couple of seconds to start the debugging process with delve. If your paths are mounted in the proper way, you will find the file `__debug_bin` in the project path on your computer. That is an executable which has been created by delve.
251+
Usually it takes a couple of seconds to start the debugging process with delve. If your paths are mounted in the proper way, you will find the file `__debug_bin` in the project path on your computer. That is an executable which has been created by delve.
232252
233-
Also, you can output logs of the pod by performing `kubectl logs $PODNAME` in order to make sure that the delve API server is listening at 30123.
253+
Also, you can output logs of the pod by performing `kubectl logs $PODNAME` in order to make sure the delve API server is listening at 30123.
234254
235255
Output:
236-
256+
```
237257
API server listening at: [::]:30123
258+
```
238259
239-
_Hint: always wait until this log message is shown for this pod before you start the debugging process. Otherwise the delve server is not up yet and cannot answer to the debugger_
260+
_Hint: always wait until this log message is shown for this pod before you start the debugging process. Otherwise, the delve server is not up yet and cannot answer to the debugger._
240261
241262
### Starting the debug process via launch.json
242263
243264
Now we need a debug configuration in Visual Code. This can be done in `.vscode/launch.json`:
244265
245-
```
266+
```json
246267
{
247268
"version": "0.2.0",
248269
"configurations": [
@@ -251,15 +272,17 @@ Now we need a debug configuration in Visual Code. This can be done in `.vscode/l
251272
"type": "go",
252273
"request": "attach",
253274
"mode":"remote",
254-
"remotePath": "/go/src/github.com/setlog/debug-k8s", // path to the project path inside the pod
255-
"port": 30123, // local port to send the debug commands to
256-
"host": "127.0.0.1", // host to send the debug commands to
275+
"remotePath": "/go/src/github.com/setlog/debug-k8s",
276+
"port": 30123,
277+
"host": "127.0.0.1",
257278
"showLog": true
258279
}
259280
]
260281
}
261282
```
262283

284+
Where `remotePath` is the path to the project path inside the pod, `port` the local port to send the debug commands to, and `host` the host to send the debug commands to.
285+
263286
You find the new configuration in Visual Code here:
264287

265288
![Debug Configuration](images/debug-config.png "Where to find the debug config")
@@ -270,13 +293,17 @@ After starting the debug process there is a new log created by the go service:
270293

271294
We are ready to debug, but we have to trigger the API functions through the ingress service. Deploy it with kubectl:
272295

273-
`kubectl create -f cluster/ingress.yaml`
296+
```sh
297+
kubectl create -f cluster/ingress.yaml
298+
```
274299

275-
...and try it now:
300+
And try accessing it now:
276301

277-
`curl http://localhost:8090/hello`
302+
```sh
303+
curl http://localhost:8090/hello
304+
```
278305

279-
Here you go:
306+
Which should trigger the debugger:
280307

281308
![Breakpoint](images/debug-screen.png "Breakpoint in Visual Code")
282309

@@ -286,4 +313,6 @@ Happy debugging!
286313

287314
If you don't need your kind cluster anymore, it can be removed with following command:
288315

289-
`kind delete cluster --name=local-debug-k8s`
316+
```sh
317+
kind delete cluster --name=local-debug-k8s
318+
```

0 commit comments

Comments
 (0)