Skip to content

Commit 36c6198

Browse files
dchaykindchaykin
committed
initial version
1 parent a7ff7d0 commit 36c6198

File tree

12 files changed

+435
-2
lines changed

12 files changed

+435
-2
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
__debug_bin

‎.vscode/launch.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"version": "0.2.0",
3+
"configurations": [
4+
{
5+
"name": "Remote debug in Kubernetes",
6+
"type": "go",
7+
"request": "attach",
8+
"mode":"remote",
9+
"remotePath": "/go/src/github.com/setlog/debug-k8s",
10+
"port": 30123,
11+
"host": "127.0.0.1",
12+
"showLog": true
13+
}
14+
]
15+
}

‎Dockerfile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
FROM golang:1.13-alpine
2+
3+
ENV CGO_ENABLED=0
4+
ENV GOROOT=/usr/local/go
5+
ENV GOPATH=${HOME}/go
6+
ENV PATH=$PATH:${GOROOT}/bin
7+
8+
EXPOSE 30123
9+
EXPOSE 8090
10+
11+
WORKDIR /go/src/github.com/setlog/debug-k8s
12+
13+
RUN apk update && apk add git && \
14+
go get github.com/go-delve/delve/cmd/dlv
15+
16+
ENTRYPOINT ["/go/bin/dlv", "debug", ".", "--listen=:30123", "--accept-multiclient", "--headless=true", "--api-version=2"]

‎README.md‎

Lines changed: 286 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,286 @@
1-
# debug-k8s
2-
How to debug a go-service in kubernetes
1+
### Preface
2+
3+
In this article you are going to learn, how to remotely debug a go service in kubernetes. If you are a developer with an unusual power, your services have no dependencies to other services or all those dependencies are mocked in unit tests, you don't need to debug anything in kubernetes. All other developers need to setup debugging and they want to do it in the kubernetes environment as well. Let us share our exciting and painful experience with you now.
4+
5+
### Prerequisites
6+
7+
1. Docker Desktop: https://docs.docker.com/get-docker/
8+
9+
Our version: 19.03.8
10+
2. Kind (Kubernetes in Docker): https://kind.sigs.k8s.io. We decided to use kind instead of minikube, since it is a very good tool for testing kubernetes locally.
11+
12+
Our version: v0.7.0
13+
3. Kubectl: https://kubernetes.io/de/docs/tasks/tools/install-kubectl/
14+
15+
Our version: 1.17.2
16+
4. Visual Studio Code: https://code.visualstudio.com/download
17+
18+
Our version: 1.32.3
19+
20+
### Big Picture
21+
22+
First, we are going to briefly explain, how it works:
23+
* you need a docker container with delve started as a main process in it
24+
* delve (Go debugger) must have an access to the folder with project files. That is done by mounting $GOPATH/src into the pod running in the kubernetes environment
25+
* we start the delve server on the port 30123 and mount this port to the localhost, so that debugger can communicate with the server through it
26+
* in order to trigger API functions we want to debug it is necessary to establish an ingress network. We use the port 8090 for that
27+
28+
All-in-all it will look like this picture demonstrates:
29+
30+
![Overview](images/big-picture.png "Big Picture")
31+
32+
### Creating the Kubernetes cluster
33+
34+
###### Start the cluster
35+
36+
Before starting we need to adjust the cluster config file to your environment. Unfortunately, `kind` does not use the environment variables and we have to inject them into the config file with `sed`:
37+
38+
`sed -i.bak 's|'{GOPATH}'|'${GOPATH}'|g' cluster/config.yaml`
39+
40+
You can also open `cluster/config.yaml` and replace {GOPATH} with the absolute path manually:
41+
42+
extraMounts:
43+
- hostPath: {GOPATH}/src
44+
45+
Assuming you already have installed kind (Kubernetes in Docker) on your local machine, the cluster is created by the following command:
46+
47+
`kind create cluster --config cluster/config.yaml --name=local-debug-k8s`
48+
49+
The cluster has the name `local-debug-k8s` and is created with the custom configuration (parameter `--config cluster/config.yaml`). Let us take a look at `cluster/config.yaml` and explain it:
50+
51+
```kind: Cluster
52+
apiVersion: kind.x-k8s.io/v1alpha4
53+
nodes:
54+
- role: control-plane
55+
kubeadmConfigPatches:
56+
- |
57+
kind: InitConfiguration # necessary, since we are going to install an ingress network in the cluster
58+
nodeRegistration:
59+
kubeletExtraArgs:
60+
node-labels: "ingress-ready=true"
61+
authorization-mode: "AlwaysAllow"
62+
extraPortMappings:
63+
- containerPort: 80 # http endpoint of ingress runs on the port 80
64+
hostPort: 8090 # port on your host machine to call API's of the service
65+
protocol: TCP
66+
- containerPort: 30123 # node port for the delve server
67+
hostPort: 30123 # port on your host machine to communicate with the delve server
68+
protocol: TCP
69+
- role: worker
70+
extraMounts:
71+
- hostPath: {GOPATH}/src # ATTENTION: you might want to replace this path with your ${GOPATH}/src manually
72+
containerPath: /go/src # path to the project folder inside the worker node
73+
```
74+
75+
_Hint: make sure that ports 8090 and 30123 are free on your computer before you create the cluster_
76+
77+
Output:
78+
79+
Creating cluster "local-debug-k8s" ...
80+
✓ Ensuring node image (kindest/node:v1.17.0) 🖼
81+
✓ Preparing nodes 📦 📦
82+
✓ Writing configuration 📜
83+
✓ Starting control-plane 🕹️
84+
✓ Installing CNI 🔌
85+
✓ Installing StorageClass 💾
86+
✓ Joining worker nodes 🚜
87+
Set kubectl context to "kind-local-debug-k8s"
88+
You can now use your cluster with:
89+
90+
kubectl cluster-info --context kind-local-debug-k8s
91+
92+
Have a nice day! 👋
93+
94+
Activate the kube-context, so that _kubectl_ can communicate with the newly created cluster:
95+
96+
`kubectl cluster-info --context kind-local-debug-k8s`
97+
98+
###### Install nginx-ingress
99+
100+
Source: https://kind.sigs.k8s.io/docs/user/ingress/#ingress-nginx
101+
102+
In order to make both port mounts working (8090 and 30123), it is necessary to deploy the nginx controller as well.
103+
Run the following command for it:
104+
105+
`kubectl create -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/kind/deploy.yaml`
106+
107+
...and wait until nginx-controller runs:
108+
109+
`kubectl wait --namespace ingress-nginx --for=condition=ready pod --selector=app.kubernetes.io/component=controller --timeout=120s`
110+
111+
###### Labelling the worker node
112+
113+
We would suggest to label a worker node where the pod is going to be deployed: by default, a pod is deployed on one of several worker nodes you might have in the kind cluster. To make it work the docker image must be populated on all worker nodes in the cluster (it takes time). Otherwise, you can get into a situation, in which the pod is started on a node where the docker image is missing. Let's work with a dedicated node and safe the time.
114+
115+
So, we label a worker node with _debug=true_:
116+
117+
`kubectl label nodes local-debug-k8s-worker debug=true`
118+
119+
### Creating a docker image
120+
121+
Our service has only one `/hello` endpoint and writes just a few logs. The interesting part is in the Dockerfile:
122+
123+
```
124+
FROM golang:1.13-alpine
125+
126+
ENV CGO_ENABLED=0 # compile gcc statically
127+
ENV GOROOT=/usr/local/go
128+
ENV GOPATH=${HOME}/go # this path will be mounted in deploy-service.yaml later
129+
ENV PATH=$PATH:${GOROOT}/bin
130+
131+
EXPOSE 30123 # for delve
132+
EXPOSE 8090 # for API calls
133+
134+
WORKDIR /go/src/github.com/setlog/debug-k8s # ATTENTION: you want to check, if the path to the project folder is the right one here
135+
136+
# Install delve, our version is 1.4.1
137+
RUN apk update && apk add git && \
138+
go get github.com/go-delve/delve/cmd/dlv
139+
140+
# let start delve at the entrypoint
141+
ENTRYPOINT ["/go/bin/dlv", "debug", ".", "--listen=:30123", "--accept-multiclient", "--headless=true", "--api-version=2"]
142+
```
143+
144+
Build the docker image locally, first:
145+
146+
`docker build -t setlog/debug-k8s .`
147+
148+
Load the docker image into the node _local-debug-k8s-worker_
149+
150+
`kind load docker-image setlog/debug-k8s:latest --name=local-debug-k8s --nodes=local-debug-k8s-worker`
151+
152+
This message will be shown and it is just saying that the image was not there:
153+
154+
Image: "setlog/debug-k8s:latest" with ID "sha256:944baa03d49698b9ca1f22e1ce87b801a20ce5aa52ccfc648a6c82cf8708a783" not present on node "local-debug-k8s-worker"
155+
156+
### Starting the delve server in the cluster
157+
158+
Now we want to create a persistent volume and its claim in order to mount the project path into the worker node:
159+
160+
`kubectl create -f cluster/persistent-volume.yaml`
161+
162+
The interesting part here is:
163+
164+
```
165+
hostPath:
166+
path: /go/src
167+
```
168+
169+
Lets take a look at the full chain of mounting the local project path into the pod, since you want probably to adjust them to your environment:
170+
171+
![Mounting](images/mounting.png "How to mount the project folder")
172+
173+
Check, if your persistent volume claim has been successfully created (STATUS must be Bound):
174+
175+
`kubectl get pvc`
176+
177+
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
178+
go-pvc Bound go-pv 256Mi RWO hostpath 51s
179+
180+
You are ready to start the service in the debug mode:
181+
182+
`kubectl create -f cluster/deploy-service.yaml`
183+
184+
Let's go through the deployment.
185+
186+
* Image name is what we loaded into the kind cluster with the command `kind load image...`. _imagePullPolicy_ must be set to _IfNotPresent_, because it is already loaded there and we don't want kubernetes to try doing it once more.
187+
188+
image: setlog/debug-k8s:latest
189+
imagePullPolicy: IfNotPresent
190+
191+
* We use the persistent volume claim to mount the project path into the pod and make `/go/src` to be linked with `${GOPATH}/src` on your computer:
192+
193+
containers:
194+
- name: debug-k8s
195+
...
196+
volumeMounts:
197+
- mountPath: /go/src
198+
name: go-volume
199+
volumes:
200+
- name: go-volume
201+
persistentVolumeClaim:
202+
claimName: go-pvc
203+
204+
* As there might be several workers in your cluster, we deploy the pod on the one, that is labelled with _debug=true_. The docker image _setlog/debug-k8s_ has been loaded earlier in it already.
205+
206+
nodeSelector:
207+
debug: "true"
208+
209+
* Service _service-debug_ has the type _NodePort_ and is mounted into the worker node. This port 30123 is equal to the parameter _--listen=:30123_ in the Dockerfile, what makes possible to send debug commands to the delve server.
210+
211+
* Service _debug-k8s_ will be connected to the ingress server in the final step. It serves for exposing the API endpoints we are going to debug.
212+
213+
If you did all steps correctly, your pod should be up and running. Check it with `kubectl get pod`. You should see the output with the pod status _Running_ and two additional services _debug-k8s_ and _service-debug_:
214+
215+
```
216+
NAME READY STATUS RESTARTS AGE
217+
pod/debug-k8s-6d69b65cf-4fl6t 1/1 Running 0 1h
218+
219+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
220+
service/debug-k8s ClusterIP 10.96.80.193 <none> 8090/TCP 1h
221+
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
222+
service/service-debug NodePort 10.96.219.86 <none> 30123:30123/TCP 1h
223+
```
224+
225+
_Hint: create a new variable to store the pod name. It can be helpful, if you repeatedly debug the pod_
226+
`PODNAME=$(kubectl get pod -o jsonpath='{.items[0].metadata.name}')`
227+
228+
Usualy it takes a couple of seconds to start the debugging process with delve. If your paths are mounted in the proper way, you will find the file `__debug_bin` in the project path on your computer. That is an executable which has been created by delve.
229+
230+
Also, you can output logs of the pod by performing `kubectl logs $PODNAME` in order to make sure that the delve API server is listening at 30123.
231+
232+
Output:
233+
234+
API server listening at: [::]:30123
235+
236+
_Hint: always wait until this log message is shown for this pod before you start the debugging process. Otherwise the delve server is not up yet and cannot answer to the debugger_
237+
238+
### Starting the debug process via launch.json
239+
240+
Now we need a debug configuration in Visual Code. This can be done in `.vscode/launch.json`:
241+
242+
```
243+
{
244+
"version": "0.2.0",
245+
"configurations": [
246+
{
247+
"name": "Remote debug in Kubernetes",
248+
"type": "go",
249+
"request": "attach",
250+
"mode":"remote",
251+
"remotePath": "/go/src/github.com/setlog/debug-k8s", // path to the project path inside the pod
252+
"port": 30123, // local port to send the debug commands to
253+
"host": "127.0.0.1", // host to send the debug commands to
254+
"showLog": true
255+
}
256+
]
257+
}
258+
```
259+
260+
You find the new configuration in Visual Code here:
261+
262+
![Debug Configuration](images/debug-config.png "Where to find the debug config")
263+
264+
After starting the debug process there is a new log created by the go service:
265+
266+
2020/05/28 15:38:53 I am going to start...
267+
268+
We are ready to debug, but we have to trigger the API functions through the ingress service. Deploy it with kubectl:
269+
270+
`kubectl create -f cluster/ingress.yaml`
271+
272+
...and try it now:
273+
274+
`curl http://localhost:8090/hello`
275+
276+
Here you go:
277+
278+
![Breakpoint](images/debug-screen.png "Breakpoint in Visual Code")
279+
280+
Happy debugging!
281+
282+
### Cleaning up
283+
284+
If you don't need your kind cluster anymore, it can be removed with following command:
285+
286+
`kind delete cluster --name=local-debug-k8s`

‎cluster/config.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
kind: Cluster
2+
apiVersion: kind.x-k8s.io/v1alpha4
3+
nodes:
4+
- role: control-plane
5+
kubeadmConfigPatches:
6+
- |
7+
kind: InitConfiguration # necessary, since we are going to install an ingress network in the cluster
8+
nodeRegistration:
9+
kubeletExtraArgs:
10+
node-labels: "ingress-ready=true"
11+
authorization-mode: "AlwaysAllow"
12+
extraPortMappings:
13+
- containerPort: 80 # http endpoint of ingress runs on the port 80
14+
hostPort: 8090 # port on your host machine to call API's of the service
15+
protocol: TCP
16+
- containerPort: 30123 # node port for the delve server
17+
hostPort: 30123 # port on your host machine to communicate with the delve server
18+
protocol: TCP
19+
- role: worker
20+
extraMounts:
21+
- hostPath: {GOPATH}/src # ATTENTION: you might want to change this path to your ${GOPATH}/src
22+
containerPath: /go/src # path to the project folder inside the worker node

0 commit comments

Comments
 (0)