Skip to content

Commit 4130b05

Browse files
networkandcodenetworkandcode
committed
added kubernetes-build-a-private-docker-registry
1 parent 0943294 commit 4130b05

6 files changed

+621
-0
lines changed

‎_posts/kubernetes/.2021-08-14-kubernetes-build-a-private-docker-registry.md‎

Lines changed: 329 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,329 @@
1+
---
2+
#title: kubernetes > build a private docker registry
3+
#categories: kubernetes
4+
---
5+
6+
Let's see how we can build our own private docker registry as a kubernetes workload. You need
7+
familiarity with docker, openssl, kubernets deployments, services to follow along.
8+
9+
On my system, where I have kubectl installed, I am going to first prepare the cert.
10+
11+
Create a directory to keep the certificates, and switch to that directory.
12+
```
13+
networkandcode@ubuntu20:~$ mkdir docker-registry && cd $_
14+
networkandcode@ubuntu20:~/docker-registry$
15+
```
16+
17+
Using openssl, let's generate a private key and public certificate.
18+
```
19+
networkandcode@ubuntu20:~/docker-registry$ openssl req -x509 -newkey rsa:4096 -days 365 \
20+
-nodes -sha256 -keyout tls.key -out tls.crt -subj "/CN=docker-registry"
21+
Generating a RSA private key
22+
.....................++++
23+
........................................................................................................................++++
24+
writing new private key to 'tls.key'
25+
-----
26+
27+
$ ls -l
28+
total 8
29+
-rw-rw-r-- 1 networkandcode networkandcode 1822 Aug 14 12:00 tls.crt
30+
-rw------- 1 networkandcode networkandcode 3276 Aug 14 12:00 tls.key
31+
32+
```
33+
34+
For understanding, let's see why we used the options above.
35+
```
36+
$ openssl req -help
37+
```
38+
39+
The above command should provide the summary, using which we can find the following.
40+
```
41+
-x509 Output a x509 structure instead of a cert request
42+
(Required by some CA's)
43+
-newkey val Specify as type:bits
44+
-days +int Number of days cert is valid for
45+
-nodes Don't encrypt the output key
46+
-* Any supported digest
47+
-keyout outfile File to send the key to
48+
-out outfile Output file
49+
-subject Output the request's subject
50+
```
51+
52+
So, we have used openssl to generate a x509 cerificate, for a new key of type RSA and modulus 4096 bits.
53+
The validiy of the certificate we generate will be valid for 365 days. We are using SHA256 as the hash
54+
function or message digest algorithm. The key will be saved in the file tls.key. The certificate will
55+
be saved in the file tls.crt. The certificate request will have the common name docker-registry in the
56+
subject.
57+
58+
Let's use the generated key and certificate in a kubernetes secret object specially meant for tls.
59+
```
60+
$ kubectl create secret tls docker-registry-tls --key=tls.key --cert=tls.crt
61+
secret/docker-registry created
62+
```
63+
64+
The name of the secret we created is docker-registry, and the secret's type is TLS.
65+
```
66+
$ kubectl get secret docker-registry-tls
67+
NAME TYPE DATA AGE
68+
docker-registry-tls kubernetes.io/tls 2 67s
69+
```
70+
71+
We can save this in a file, which should help reusing it when required.
72+
```
73+
$ kubectl get secret docker-registry-tls -o yaml > docker-registry-tls-secret.yaml
74+
```
75+
76+
Let's say we want to access the docker registry using username 'docker' and password 'p@ssword', we
77+
can generate a bcrypt encrypted credential with htpasswd as follows.
78+
```
79+
$ htpasswd -Bbn docker p@ssword
80+
docker:$2y$05$enG4ZbF26pDVZQ391loSju4qOXc5bPZT81kYqdAGDbdi4o1A4EU5q
81+
```
82+
83+
The options we used are:
84+
```
85+
-B Force bcrypt encryption of the password (very secure).
86+
-b Use the password from the command line rather than prompting for it.
87+
-n Don't update file; display results on stdout.
88+
```
89+
90+
Let's define our secret manifest with this data.
91+
```
92+
$ cat secret-docker-registry-htpasswd.yaml
93+
---
94+
apiVersion: v1
95+
kind: Secret
96+
metadata:
97+
name: docker-registry-htpasswd
98+
stringData:
99+
htpasswd: docker:$2y$05$enG4ZbF26pDVZQ391loSju4qOXc5bPZT81kYqdAGDbdi4o1A4EU5q
100+
...
101+
```
102+
103+
And then create it. Note that this is a generic(Opaque) secret, where as the first one we created was
104+
of type TLS.
105+
```
106+
$ kubectl create -f secret-docker-registry-htpasswd.yaml
107+
secret/docker-registry-htpasswd created
108+
```
109+
110+
So, we now have two secrets, one for TLS key and certificate, and the other for the credential.
111+
```
112+
$ kubectl get secret | grep docker
113+
docker-registry-htpasswd Opaque 1 107s
114+
docker-registry-tls kubernetes.io/tls 2 19m
115+
```
116+
117+
Let's define the config map which contains the overall configuration for our registry.
118+
```
119+
$ cat docker-registry-cm.yaml
120+
---
121+
apiVersion: v1
122+
kind: ConfigMap
123+
metadata:
124+
name: docker-registry
125+
data:
126+
config.yml: |
127+
version: 0.1
128+
log:
129+
fields:
130+
service: registry
131+
storage:
132+
cache:
133+
blobdescriptor: inmemory
134+
filesystem:
135+
rootdirectory: /var/lib/registry
136+
http:
137+
addr: 0.0.0.0:443
138+
headers:
139+
X-Content-Type-Options: [nosniff]
140+
tls:
141+
cert: /etc/ssl/registry/tls.crt
142+
key: /etc/ssl/registry/tls.key
143+
auth:
144+
htpasswd:
145+
realm: "Registry Realm"
146+
path: /etc/docker/authentication
147+
health:
148+
storagedriver:
149+
enabled: true
150+
interval: 10s
151+
threshold: 3
152+
...
153+
```
154+
155+
Create the config map.
156+
```
157+
$ kubectl create -f docker-registry-cm.yaml
158+
configmap/docker-registry created
159+
```
160+
161+
Ok so we have the created the data objects, 2 secrets and one config map. We would also need to setup
162+
a volume so that the registry is persistent and we do not loose data on pod restarts. For this
163+
exercise we would leverage the hostPath volume, so that the data is saved on the kubernetes node itself.
164+
165+
Let's define the deployment.
166+
```
167+
$ cat docker-registry-deploy.yaml
168+
---
169+
apiVersion: apps/v1
170+
kind: Deployment
171+
metadata:
172+
name: docker-registry
173+
spec:
174+
selector:
175+
matchLabels:
176+
function: registry
177+
template:
178+
metadata:
179+
labels:
180+
function: registry
181+
spec:
182+
containers:
183+
- name: docker-registry
184+
# image will be picked from docker hub
185+
image: registry:2.7.1
186+
volumeMounts:
187+
- name: config
188+
mountPath: /etc/docker/registry
189+
readOnly: true
190+
- name: credential
191+
mountPath: /etc/docker/authentication
192+
readOnly: true
193+
- name: registry
194+
mountPath: /var/lib/registry
195+
- name: tls
196+
mountPath: /etc/ssl/registry
197+
readOnly: true
198+
volumes:
199+
- name: config
200+
configMap:
201+
name: docker-registry
202+
- name: credential
203+
secret:
204+
secretName: docker-registry-htpasswd
205+
- name: registry
206+
hostPath:
207+
path: /var/lib/docker-registry
208+
- name: tls
209+
secret:
210+
secretName: docker-registry-tls
211+
```
212+
213+
Create the deployment.
214+
```
215+
$ kubectl create -f docker-registry-deploy.yaml
216+
deployment.apps/docker-registry created
217+
```
218+
219+
Let's check the status of the deployment, replica set, and the pod
220+
```
221+
$ kubectl get all | grep docker-registry
222+
pod/docker-registry-bf9b44585-8897c 1/1 Running 0 3m21s
223+
deployment.apps/docker-registry 1/1 1 1 3m21s
224+
replicaset.apps/docker-registry-bf9b44585 1 1 1 3m21s
225+
```
226+
227+
We can check if the config and secrets are injected to the container.
228+
```
229+
docker-registry-7c8cd4c47-lpqrt
230+
$ kubectl exec -it docker-registry-86896c68c5-vg95k -- /bin/sh
231+
232+
/ # cat /etc/docker/authentication/htpasswd
233+
docker:$2y$05$enG4ZbF26pDVZQ391loSju4qOXc5bPZT81kYqdAGDbdi4o1A4EU5q
234+
235+
/ # ls /etc/docker/registry
236+
config.yml
237+
238+
/ # ls /etc/ssl/registry
239+
tls.crt tls.key
240+
241+
```
242+
243+
Let's expose the docker registry as a service. Note that the container port is 5000.
244+
```
245+
$ cat docker-registry-svc.yaml
246+
---
247+
apiVersion: v1
248+
kind: Service
249+
metadata:
250+
name: docker-registry
251+
spec:
252+
selector:
253+
function: registry
254+
ports:
255+
- name: registry
256+
port: 443
257+
targetPort: 443
258+
...
259+
```
260+
261+
Let's create the service, and validate the endpoints.
262+
```
263+
$ kubectl create -f docker-registry-svc.yaml
264+
service/docker-registry created
265+
266+
$ kubectl get svc docker-registry
267+
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
268+
docker-registry ClusterIP 10.43.135.120 <none> 8000/TCP 64s
269+
270+
$ kubectl get ep docker-registry
271+
NAME ENDPOINTS AGE
272+
docker-registry 10.42.2.3:5000 78s
273+
274+
$ kubectl get po -o wide | grep docker-registry
275+
docker-registry-7c8cd4c47-lpqrt 1/1 Running 0 5m40s 10.42.2.3 node1 <none> <none>
276+
```
277+
278+
So the IP 10.42.2.3 is matching. Please make a note of the service ip and port which is
279+
10.43.135.120:8000, this is where the docker registry is available in the cluster.
280+
281+
Since the nodes need to communicate with the docker registry, we can add a dns entry in the hosts file
282+
on all nodes.
283+
```
284+
$ ssh root@<public ip of master> "echo '10.43.135.120 docker-registry' >> /etc/hosts"
285+
$ ssh root@<public ip of node1> "echo '10.43.135.120 docker-registry' >> /etc/hosts"
286+
$ ssh root@<public ip of node2> "echo '10.43.135.120 docker-registry' >> /etc/hosts"
287+
```
288+
289+
This means the registry can be access with a name from the nodes, i.e. docker-registry:5000.
290+
291+
Next step is to copy the public certificate we generated earlier for the docker-registry as ca.crt in
292+
the nodes, in a specific path /etc/docker/certs.d/registry-name:port, if we replace this with our
293+
registry name and port, it would become /etc/docker/certs.d/docker-registry:8000.
294+
295+
In our nodes, we first need to create the directorirs certs.d and docker-registry:5000. We can do this
296+
with a single command.
297+
```
298+
$ ssh root@<public ip of master> "mkdir -p /etc/docker/certs.d/docker-registry:8000"
299+
$ ssh root@<public ip of node1> "mkdir -p /etc/docker/certs.d/docker-registry:8000"
300+
$ ssh root@<public ip of node2> "mkdir -p /etc/docker/certs.d/docker-registry:8000"
301+
```
302+
303+
We can now copy the certificate.
304+
```
305+
$ scp tls.crt root@<public ip of master>:/etc/docker/certs.d/docker-registry:8000/ca.crt
306+
tls.crt 100% 1822 4.3KB/s 00:00
307+
308+
$ scp tls.crt root@<public ip of node1>:/etc/docker/certs.d/docker-registry:8000/ca.crt
309+
tls.crt 100% 1822 17.3KB/s 00:00
310+
311+
$ scp tls.crt root@<public ip of node2>:/etc/docker/certs.d/docker-registry:8000/ca.crt
312+
tls.crt
313+
```
314+
315+
We have setup the nodes, to interact with our private docker registry. Let's login to one of the nodes
316+
and see if we can authenticate to the docker registry.
317+
```
318+
$ ssh root@<public ip of master>
319+
320+
root@localhost:~# docker login
321+
322+
```
323+
324+
325+
Reference:
326+
- [Docker Docs](https://docs.docker.com/registry/configuration/)
327+
- [Docker Hub](https://hub.docker.com/_/registry)
328+
- [Deploy Your Private Docker Registry as a Pod in Kubernetes](https://medium.com/swlh/deploy-your-private-docker-registry-as-a-pod-in-kubernetes-f6a489bf0180)
329+
- [kubernetes > add a user with tls](https://networkandcode.github.io/kubernetes/2020/02/27/add-a-kubectl-user-with-tls.html)

0 commit comments

Comments
 (0)