You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+83-54Lines changed: 83 additions & 54 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,39 +1,37 @@
1
1
# Preface
2
2
3
-
In a perfect world every written service will work smooth, your test coverage is on top and there are no bugs in your API. But we all know, that we can´t achieve this world, sadly. It´s not unusual that there´s a bug in an API and you have to debug this problem in a production environment. We have faced this problem with our go services in our kubernetes cluster and we want to show you how it´s possible to remote debug a go service in a kubernetes cluster.
3
+
In a perfect world every written service will work smooth, your test coverage is on top and there are no bugs in your API. But we all know, that we can't achieve this world, sadly. It's not unusual that there's a bug in an API and you have to debug this problem in a production environment. We have faced this problem with our go services in our Kubernetes cluster, and we want to show you how it's possible to remote debug a go service in a Kubernetes cluster.
*[Visual Studio Code](https://code.visualstudio.com/download) (used version: 1.32.3)
13
13
14
-
We decided to use `kind` instead of `minikube`, since it´s a very good tool for testing kubernetes locally and we can use our docker images without a docker registry.
14
+
We decided to use `kind` instead of `minikube`, since it's a very good tool for testing Kubernetes locally, and we can use our docker images without a docker registry.
15
15
16
16
## Big Picture
17
17
18
-
First we will briefly explain how it works:
18
+
First we will briefly explain how it works. We start by creating a new Kubernetes cluster `local-debug-k8s` on our local system.
19
19
20
-
1. We create a new kubernetes cluster `local-debug-k8s` on our local system
21
-
22
-
* you need a docker container with delve (<https://github.com/go-delve/delve>) as main process
23
-
* delve needs access to the path with the project data. This is done by mounting `$GOPATH/src` on the pod which is running in the kubernetes cluster
24
-
* we start the delve container on port 30123 and bind this port to localhost, so that only our local debugger can communicate with delve
25
-
* to debug an API with delve, it´s necessary to set up an ingress network. For this we use port 8090.
20
+
* You need a docker container with [delve](https://github.com/go-delve/delve) (the go debugger) as the main process.
21
+
* The debugger delve needs access to the path with the project data. This is done by mounting `$GOPATH/src` on the pod which is running in the Kubernetes cluster.
22
+
* We start the delve container on port 30123 and bind this port to localhost, so that only our local debugger can communicate with delve.
23
+
* To debug an API with delve, it's necessary to set up an ingress network. For this we use port 8090.
26
24
27
25
A picture serves to illustrate the communication:
28
26
29
27

30
28
31
29
### Creating a Kubernetes cluster
32
30
33
-
`kind` unfortunately doesn´t use the environment variable `GOPATH`, so we have to update this in [config.yaml](cluster/config.yaml#L21):
31
+
`kind` unfortunately doesn't use the environment variable `GOPATH`, so we have to update this in [config.yaml](cluster/config.yaml#L21):
sed -i.bak 's|'{GOPATH}'|'${GOPATH}'|g' cluster/config.yaml
37
35
```
38
36
39
37
You can also open [config.yaml](cluster/config.yaml#L21) and replace `{GOPATH}` with the absolute path manually. If you already installed kind (Kubernetes in Docker) on your local system, you can create the cluster with this command:
@@ -69,7 +67,7 @@ nodes:
69
67
containerPath: /go/src # path to the project folder inside the worker node
70
68
```
71
69
72
-
Desired result:
70
+
Expected result:
73
71
74
72
```sh
75
73
Creating cluster "local-debug-k8s" ...
@@ -97,27 +95,29 @@ Activate the kube-context for `kubectl` to communicate with the new cluster:
97
95
98
96
#### Install nginx-ingress
99
97
100
-
For both ports (8090 and 30123) to work it´s necessary to deploy a nginx controller:
98
+
For both ports (8090 and 30123) to work, it is necessary to deploy an nginx controller:
kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s
112
110
```
113
111
114
112
#### Labelling the worker node
115
113
116
-
We would suggest to label a worker node where the pod is going to be deployed: by default, a pod is deployed on one of several worker nodes you might have in the kind cluster. To make it work the docker image must be populated on all worker nodes in the cluster (it takes time). Otherwise, you can get into a situation, in which the pod is started on a node where the docker image is missing. Let's work with a dedicated node and safe the time.
114
+
We suggest labelling a worker node where the pod is going to be deployed. By default, a pod is deployed on one of several worker nodes you might have in the kind cluster. To make it work, the docker image must be populated on all worker nodes in the cluster (which takes time). Otherwise, you can get into a situation, in which the pod has started on a node where the docker image is missing. Let's work with a dedicated node and safe the time.
This message will be shown and it is just saying that the image was not there:
162
+
This message will be shown, and it is just saying that the image was not there:
156
163
164
+
```
157
165
Image: "setlog/debug-k8s:latest" with ID "sha256:944baa03d49698b9ca1f22e1ce87b801a20ce5aa52ccfc648a6c82cf8708a783" not present on node "local-debug-k8s-worker"
166
+
```
158
167
159
168
### Starting the delve server in the cluster
160
169
161
170
Now we want to create a persistent volume and its claim in order to mount the project path into the worker node:
@@ -169,30 +180,37 @@ The interesting part here is:
169
180
path: /go/src
170
181
```
171
182
172
-
Lets take a look at the full chain of mounting the local project path into the pod, since you want probably to adjust them to your environment:
183
+
Let's take a look at the full chain of mounting the local project path into the pod, since you want probably to adjust them to your environment:
173
184
174
185

175
186
176
187
Check, if your persistent volume claim has been successfully created (STATUS must be Bound):
177
188
178
-
`kubectl get pvc`
189
+
```sh
190
+
kubectl get pvc
179
191
180
192
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
181
193
go-pvc Bound go-pv 256Mi RWO hostpath 51s
194
+
```
182
195
183
-
You are ready to start the service in the debug mode:
196
+
You are ready to start the service in debug mode:
184
197
185
-
`kubectl create -f cluster/deploy-service.yaml`
198
+
```sh
199
+
kubectl create -f cluster/deploy-service.yaml
200
+
```
186
201
187
202
Let's go through the deployment.
188
203
189
-
* Image name is what we loaded into the kind cluster with the command `kind load image...`. _imagePullPolicy_ must be set to _IfNotPresent_, because it is already loaded there and we don't want kubernetes to try doing it once more.
204
+
* Image name is what we loaded into the kind cluster with the command `kind load image...`. _imagePullPolicy_ must be set to _IfNotPresent_, because it is already loaded, we don't want Kubernetes to try loading it again.
190
205
191
-
image: setlog/debug-k8s:latest
192
-
imagePullPolicy: IfNotPresent
206
+
```yaml
207
+
image: setlog/debug-k8s:latest
208
+
imagePullPolicy: IfNotPresent
209
+
```
193
210
194
211
* We use the persistent volume claim to mount the project path into the pod and make `/go/src` to be linked with `${GOPATH}/src` on your computer:
195
212
213
+
```yaml
196
214
containers:
197
215
- name: debug-k8s
198
216
...
@@ -203,19 +221,22 @@ Let's go through the deployment.
203
221
- name: go-volume
204
222
persistentVolumeClaim:
205
223
claimName: go-pvc
224
+
````
206
225
207
226
* As there might be several workers in your cluster, we deploy the pod on the one, that is labelled with _debug=true_. The docker image _setlog/debug-k8s_ has been loaded earlier in it already.
208
227
228
+
```yaml
209
229
nodeSelector:
210
230
debug: "true"
231
+
```
211
232
212
-
* Service _service-debug_ has the type _NodePort_ and is mounted into the worker node. This port 30123 is equal to the parameter _--listen=:30123_ in the Dockerfile, what makes possible to send debug commands to the delve server.
233
+
* Service _service-debug_ has the type _NodePort_ and is mounted into the worker node. This port 30123 is equal to the parameter _--listen=:30123_ in the Dockerfile, which makes it possible to send debug commands to the delve server.
213
234
214
235
* Service _debug-k8s_ will be connected to the ingress server in the final step. It serves for exposing the API endpoints we are going to debug.
215
236
216
237
If you did all steps correctly, your pod should be up and running. Check it with `kubectl get pod`. You should see the output with the pod status _Running_ and two additional services _debug-k8s_ and _service-debug_:
_Hint: create a new variable to store the pod name. It can be helpful, if you repeatedly debug the pod_
229
-
`PODNAME=$(kubectl get pod -o jsonpath='{.items[0].metadata.name}')`
249
+
_Hint: create a new variable to store the pod name using `PODNAME=$(kubectl get pod -o jsonpath='{.items[0].metadata.name}')`. It can be helpful, if you repeatedly debug the pod._
230
250
231
-
Usualy it takes a couple of seconds to start the debugging process with delve. If your paths are mounted in the proper way, you will find the file `__debug_bin` in the project path on your computer. That is an executable which has been created by delve.
251
+
Usually it takes a couple of seconds to start the debugging process with delve. If your paths are mounted in the proper way, you will find the file `__debug_bin` in the project path on your computer. That is an executable which has been created by delve.
232
252
233
-
Also, you can output logs of the pod by performing `kubectl logs $PODNAME` in order to make sure that the delve API server is listening at 30123.
253
+
Also, you can output logs of the pod by performing `kubectl logs $PODNAME` in order to make sure the delve API server is listening at 30123.
234
254
235
255
Output:
236
-
256
+
```
237
257
API server listening at: [::]:30123
258
+
```
238
259
239
-
_Hint: always wait until this log message is shown for this pod before you start the debugging process. Otherwise the delve server is not up yet and cannot answer to the debugger_
260
+
_Hint: always wait until this log message is shown for this pod before you start the debugging process. Otherwise, the delve server is not up yet and cannot answer to the debugger._
240
261
241
262
### Starting the debug process via launch.json
242
263
243
264
Now we need a debug configuration in Visual Code. This can be done in `.vscode/launch.json`:
244
265
245
-
```
266
+
```json
246
267
{
247
268
"version": "0.2.0",
248
269
"configurations": [
@@ -251,15 +272,17 @@ Now we need a debug configuration in Visual Code. This can be done in `.vscode/l
251
272
"type": "go",
252
273
"request": "attach",
253
274
"mode":"remote",
254
-
"remotePath": "/go/src/github.com/setlog/debug-k8s", // path to the project path inside the pod
255
-
"port": 30123, // local port to send the debug commands to
256
-
"host": "127.0.0.1", // host to send the debug commands to
Where `remotePath` is the path to the project path inside the pod, `port` the local port to send the debug commands to, and `host` the host to send the debug commands to.
285
+
263
286
You find the new configuration in Visual Code here:
264
287
265
288

@@ -270,13 +293,17 @@ After starting the debug process there is a new log created by the go service:
270
293
271
294
We are ready to debug, but we have to trigger the API functions through the ingress service. Deploy it with kubectl:
272
295
273
-
`kubectl create -f cluster/ingress.yaml`
296
+
```sh
297
+
kubectl create -f cluster/ingress.yaml
298
+
```
274
299
275
-
...and try it now:
300
+
And try accessing it now:
276
301
277
-
`curl http://localhost:8090/hello`
302
+
```sh
303
+
curl http://localhost:8090/hello
304
+
```
278
305
279
-
Here you go:
306
+
Which should trigger the debugger:
280
307
281
308

282
309
@@ -286,4 +313,6 @@ Happy debugging!
286
313
287
314
If you don't need your kind cluster anymore, it can be removed with following command:
0 commit comments