completely reworked into an HTTPS_PROXY-based solution

ricardop · ricardop · commit 0abd4ca51ada · 2018-06-29T01:39:02.000+02:00
- emit our own certificates
- configurable via ENVs
- generates config dinamically
diff --git a/.dockerignore b/.dockerignore
@@ -3,3 +3,5 @@
 .gitignore
 LICENSE
 README.md
+docker_mirror_cache
+docker_mirror_certs
diff --git a/.gitignore b/.gitignore
@@ -1 +1,3 @@
 .idea
+docker_mirror_cache
+docker_mirror_certs
diff --git a/Dockerfile b/Dockerfile
@@ -1,26 +1,39 @@
-# Use stable nginx on alpine for a light container
-FROM nginx:stable-alpine
+# We start from my nginx fork which includes the proxy-connect module from tEngine
+# Source is available at https://github.com/rpardini/nginx-proxy-connect-stable-alpine
+# Its equivalent to nginx:stable-alpine 1.14.0, with alpine 3.7
+FROM rpardini/nginx-proxy-connect-stable-alpine:latest
 
-# Add openssl and clean apk cache
-RUN apk add --update openssl && rm -rf /var/cache/apk/*
+# Add openssl, bash and ca-certificates, then clean apk cache -- yeah complain all you want.
+RUN apk add --update openssl bash ca-certificates && rm -rf /var/cache/apk/*
 
-# Generate a self-signed SSL certificate. It will be ignored by Docker clients due to insecure-registries.
-RUN mkdir -p /etc/ssl && \
-    cd /etc/ssl && \
-    openssl genrsa -des3 -passout pass:x -out key.pem 2048 && \
-    cp key.pem key.pem.orig && \
-    openssl rsa -passin pass:x -in key.pem.orig -out key.pem && \
-    openssl req -new -key key.pem -out cert.csr -subj "/C=BR/ST=BR/L=Nowhere/O=Fake Docker Mirror/OU=Docker/CN=docker.proxy" && \
-    openssl x509 -req -days 3650 -in cert.csr -signkey key.pem -out cert.pem
-
-# Create the cache directory
-RUN mkdir -p /docker_mirror_cache
+# Create the cache directory and CA directory
+RUN mkdir -p /docker_mirror_cache /ca
 
 # Expose it as a volume, so cache can be kept external to the Docker image
 VOLUME /docker_mirror_cache
 
+# Expose /ca as a volume. Users are supposed to volume mount this, as to preserve it across restarts.
+# Actually, its required; if not, then docker clients will reject the CA certificate when the proxy is run the second time
+VOLUME /ca
+
 # Add our configuration
 ADD nginx.conf /etc/nginx/nginx.conf
 
-# Test that the configuration is OK
-RUN nginx -t
+# Add our very hackish entrypoint and ca-building scripts, make them executable
+ADD entrypoint.sh /entrypoint.sh
+ADD create_ca_cert.sh /create_ca_cert.sh
+RUN chmod +x /create_ca_cert.sh /entrypoint.sh
+
+# Clients should only use 3128, not anything else.
+EXPOSE 3128
+
+## Default envs.
+# A space delimited list of registries we should proxy and cache; this is in addition to the central DockerHub.
+ENV REGISTRIES="k8s.gcr.io gcr.io quay.io"
+# A space delimited list of registry:user:password to inject authentication for
+ENV AUTH_REGISTRIES="some.authenticated.registry:oneuser:onepassword another.registry:user:password"
+# Should we verify upstream's certificates? Default to true.
+ENV VERIFY_SSL="true"
+
+# Did you want a shell? Sorry. This only does one job; use exec /bin/bash if you wanna inspect stuff
+ENTRYPOINT ["/entrypoint.sh"]
diff --git a/README.md b/README.md
@@ -1,120 +1,100 @@
-### What?
-
-An intricate, insecure, and hackish way of caching Docker images from private registries (eg, not from DockerHub).
-Caches via HTTP man-in-the-middle.
-It is highly dependent on Docker-client behavior, and was only tested against Docker 17.03 on Linux (that's the version recommended by Kubernetes 1.10).
-
-#### Why not use Docker's own registry, which has a mirror feature?
-
-Yes, Docker offers [Registry as a pull through cache](https://docs.docker.com/registry/recipes/mirror/), 
-and, in fact, for a caching solution to be complete, you'll want to run one of those. 
-
-**Unfortunately** this only covers the DockerHub case. It won't cache images from `quay.io`, `k8s.gcr.io`, `gcr.io`, or any such, including any private registries.
-
-That means that your shiny new Kubernetes cluster is now a bandwidth hog, since every image will be pulled from the Internet on every Node it runs on, with no reuse.
+## docker-registry-proxy
 
-This is due to the way the Docker "client" implements `--registry-mirror`, it only ever contacts mirrors for images with no repository reference (eg, from DockerHub).
-When a repository is specified `dockerd` goes directly there, via HTTPS (and also via HTTP if included in a `--insecure-registry` list), thus completely ignoring the configured mirror.
+### TL,DR
 
-_Even worse,_ to complement that client-Docker problem, there is also a one-URL limitation on the registry/mirror side of things, so even if it worked we would need to run multiple mirror-registries, one for each mirrored repo.
-
-
-#### Hey but that sounds like an important limitation on Docker's side. Shouldn't they fix it?
-
-**Hell, yes**. Actually if you search on Github you'll find a lot of people with the same issues.
-* This seems to be the  [main issue on the Registry side of things](https://github.com/docker/distribution/issues/1431) and shows a lot of the use cases.
-* [Valentin Rothberg](https://github.com/vrothberg) from SUSE has implemented the support 
-  the client needs [in PR #34319](https://github.com/moby/moby/pull/34319) but after a lot of discussions and 
-  [much frustration](https://github.com/moby/moby/pull/34319#issuecomment-389783454) it is still unmerged. Sigh.
+A caching proxy for Docker; allows centralized management of registries and their authentication; caches images from *any* registry.
 
+### What?
 
-**So why not?** I have no idea; it's easy to especulate that "Docker Inc" has no interest in something that makes their main product less attractive. No matter, we'll just _hack_ our way.   
+Created as an evolution and simplification of [docker-caching-proxy-multiple-private](https://github.com/rpardini/docker-caching-proxy-multiple-private) 
+using the `HTTPS_PROXY` mechanism and injected CA root certificates instead of `/etc/hosts` hacks and _`--insecure-registry` 
 
-### How?
+As a bonus it allows for centralized management of Docker registry credentials. 
+ 
+You configure the Docker clients (_err... Kubernetes Nodes?_) once, and then all configuration is done on the proxy -- 
+for this to work it requires inserting a root CA certificate into system trusted root certs.
 
-This solution involves setting up quite a lot of stuff, including DNS hacks.
+#### Why not use Docker's own registry, which has a mirror feature?
 
-You'll need a dedicated host for running two caches, both in containers, but you'll need ports 80, 443, and 5000 available.
+Yes, Docker offers [Registry as a pull through cache](https://docs.docker.com/registry/recipes/mirror/), *unfortunately* 
+it only covers the DockerHub case. It won't cache images from `quay.io`, `k8s.gcr.io`, `gcr.io`, or any such, including any private registries.
 
-I'll refer to the caching proxy host's IP address as 192.168.66.62 in the next sections, substitute for your own. 
+That means that your shiny new Kubernetes cluster is now a bandwidth hog, since every image will be pulled from the 
+Internet on every Node it runs on, with no reuse.
 
-#### 0) A regular DockerHub registry mirror
+This is due to the way the Docker "client" implements `--registry-mirror`, it only ever contacts mirrors for images 
+with no repository reference (eg, from DockerHub).
+When a repository is specified `dockerd` goes directly there, via HTTPS (and also via HTTP if included in a 
+`--insecure-registry` list), thus completely ignoring the configured mirror.
 
-Just follow instructions on [Registry as a pull through cache](https://docs.docker.com/registry/recipes/mirror/) - expose it on 0.0.0.0:5000.
-This will only be used for DockerHub caching, and works well enough.
+#### Docker itself should provide this.
 
-#### 1) This caching proxy
+Yeah. Docker Inc should do it. So should NPM, Inc. Wonder why they don't. 😼
 
-This is an `nginx` configured extensively for reverse-proxying HTTP/HTTPS to the registries, and apply caching to it.
+### Usage
 
-It should be run in a Docker container, and **needs** be mapped to ports 80 and 443. Theres a Docker volume you can mount for storing the cached layers.
+- Run the proxy on a dedicated machine.
+- Expose port 3128
+- Map volume `/docker_mirror_cache` for up to 32gb of cached images from all registries
+- Map volume `/ca`, the proxy will store the CA certificate here across restarts
+- Env `REGISTRIES`: space separated list of registries to cache; no need to include Docker Hub, its already there
+- Env `AUTH_REGISTRIES`: space separated list of `registry:username:password` authentication info. Registry hosts here should be listed in the above ENV as well.
 
 ```bash
-docker run --rm --name docker_caching_proxy -it \ 
-           -p 0.0.0.0:80:80 -p 0.0.0.0:443:443 \
-           -v /docker_mirror_cache:/docker_mirror_cache \
-           rpardini/docker-caching-proxy-multiple-private:latest
+docker run --rm --name docker_caching_proxy -it \
+       -p 0.0.0.0:3128:3128 \  
+       -v $(pwd)/docker_mirror_cache:/docker_mirror_cache  \
+       -v $(pwd)/docker_mirror_certs:/ca  \
+       -e REGISTRIES="k8s.gcr.io gcr.io quay.io your.own.registry another.private.registry" \ 
+       -e AUTH_REGISTRIES="your.own.registry:username:password another.private.registry:user:pass"  \ 
+       rpardini/docker-caching-proxy:latest
 ```
 
-**Important**: the host running the caching proxy container should not have any extra configuration or DNS hacks shown below. 
-
-The logging is done to stdout, but the format has been tweaked to show cache MISS/HIT(s) and other useful information for this use case.
+Let's say you did this on host `192.168.66.72`, you can then `curl http://192.168.66.72:3128/ca.crt` and get the proxy CA certificate.
 
-It goes to great lengths to try and get the highest hitratio possible, to the point of rewriting headers from registries when they try to redirect to a storage service like Amazon S3 or Google Storage.
+#### Configuring the Docker clients / Kubernetes nodes
 
-It is very insecure, anyone with access to the proxy will have access to its cached images regardless of authentication, for example.
+On each Docker host that is to use the cache:
 
+- [Configure Docker proxy](https://docs.docker.com/network/proxy/) pointing to the caching server
+- Add the caching server CA certificate to the list of system trusted roots.
+- Restart `dockerd`
 
-#### 2) dockerd DNS hacks
-
-We'll need to convince Docker (actually, `dockerd` on very host) to talk to our caching proxy via some sort of DNS hack.
-The simplest for sure is to just include entries in `/etc/hosts` for each registry you want to mirror, plus a fixed address used for redirects:
+Do it all at once, tested on Ubuntu Xenial:
 
 ```bash
-# /etc/hosts entries for docker caching proxy
-192.168.66.72 docker.proxy
-192.168.66.72 k8s.gcr.io
-192.168.66.72 quay.io
-192.168.66.72 gcr.io
-``` 
-
-Only `docker.proxy` is always required, and each registry you want to mirror also needs an entry.
-
-I'm sure you can do stuff to the same effect with your DNS server but I won't go into that.
- 
-#### 3) dockerd configuration for mirrors and insecure registries
-
-Of course, we don't have a TLS certificate for `quay.io` et al, so we'll need to tell Docker to treat all proxied registries as _insecure_.
-
-We'll also point Docker to the "regular" registry mirror in item 0. 
-
-To do so in one step, edit `/etc/docker/daemon.json` (tested on Docker 17.03 on Ubuntu Xenial only):
-
-```json
-{
-  "insecure-registries": [
-    "k8s.gcr.io",
-    "quay.io",
-    "gcr.io"
-  ],
-  "registry-mirrors": [
-    "http://192.168.66.72:5000"
-  ]
-}
+# Add environment vars pointing Docker to use the proxy
+cat << EOD > /etc/systemd/system/docker.service.d/http-proxy.conf
+[Service]
+Environment="HTTP_PROXY=http://192.168.66.72:3128/"
+Environment="HTTPS_PROXY=http://192.168.66.72:3128/"
+EOD
+
+# Get the CA certificate from the proxy and make it a trusted root.
+curl http://192.168.66.123:3128/ca.crt > /usr/share/ca-certificates/docker_caching_proxy.crt
+echo docker_caching_proxy.crt >> /etc/ca-certificates.conf
+update-ca-certificates --fresh
+
+# Reload systemd
+systemctl daemon-reload
+
+# Restart dockerd
+systemctl restart docker.service
 ```
 
-After that, restart the Docker daemon: `systemctl restart docker.service`
-
 ### Testing
 
-Clear the local `dockerd` of everything not currently running: `docker system prune -a -f` (this prunes everything not currently running, beware).
+Clear `dockerd` of everything not currently running: `docker system prune -a -f` *beware*
+
 Then do, for example, `docker pull k8s.gcr.io/kube-proxy-amd64:v1.10.4` and watch the logs on the caching proxy, it should list a lot of MISSes.
+
 Then, clean again, and pull again. You should see HITs! Success.
 
-### Gotchas
+Do the same for `docker pull ubuntu` and rejoice.
+
+Test your own registry caching and authentication the same way; you don't need `docker login`, or `.docker/config.json` anymore.
 
-Of course, this has a lot of limitations
+### Gotchas
 
-- Any HTTP/HTTPS request to the domains of the registries will be proxied, not only Docker calls. *beware*
-- If you want to proxy an extra registry you'll have multiple places to edit (`/etc/hosts` and `/etc/docker/daemon.json`) and restart `dockerd` - very brave thing to do in a k8s cluster, so set it up beforehand
-- If you authenticate to a private registry and pull through the proxy, those images will be served to any client that can reach the proxy, even without authentication. *beware*
+- If you authenticate to a private registry and pull through the proxy, those images will be served to any client that can reach the proxy, even without authentication. *beware*
+- Repeat, this will make your private images very public if you're not careful.
diff --git a/create_ca_cert.sh b/create_ca_cert.sh
@@ -0,0 +1,118 @@
+#! /bin/bash
+
+set -Eeuo pipefail
+
+declare -i DEBUG=0
+
+logInfo() {
+    echo "INFO: $@"
+}
+
+PROJ_NAME=DockerMirrorBox
+logInfo "Will create certificate with names $ALLDOMAINS"
+
+CADATE=$(date "+%Y.%m.%d %H:%M")
+CAID="$(hostname -f) ${CADATE}"
+
+CN_CA="${PROJ_NAME} CA Root ${CAID}"
+CN_IA="${PROJ_NAME} Intermediate IA ${CAID}"
+CN_WEB="${PROJ_NAME} Web Cert ${CAID}"
+
+CN_CA=${CN_CA:0:64}
+CN_IA=${CN_IA:0:64}
+CN_WEB=${CN_WEB:0:64}
+
+mkdir -p /certs /ca
+cd /ca
+
+CA_KEY_FILE=/ca/ca.key
+CA_CRT_FILE=/ca/ca.crt
+CA_SRL_FILE=/ca/ca.srl
+
+if [ -f "$CA_CRT_FILE" ] ; then
+    logInfo "CA already exists. Good. We'll reuse it."
+else
+    logInfo "No CA was found. Generating one."
+    logInfo "*** Please *** make sure to mount /ca as a volume -- if not, everytime this container starts, it will regenerate the CA and nothing will work."
+    
+    openssl genrsa -des3 -passout pass:foobar -out ${CA_KEY_FILE} 4096
+
+    logInfo "generate CA cert with key and self sign it: ${CAID}"
+    openssl req -new -x509 -days 1300 -sha256 -key ${CA_KEY_FILE} -out ${CA_CRT_FILE} -passin pass:foobar -subj "/C=NL/ST=Noord Holland/L=Amsterdam/O=ME/OU=IT/CN=${CN_CA}" -extensions IA -config <(
+cat <<-EOF
+[req]
+distinguished_name = dn
+[dn]
+[IA]
+basicConstraints = critical,CA:TRUE
+keyUsage = critical, digitalSignature, cRLSign, keyCertSign
+subjectKeyIdentifier = hash
+EOF
+)
+
+    [[ ${DEBUG} -gt 0 ]] && logInfo "show the CA cert details"
+    [[ ${DEBUG} -gt 0 ]] && openssl x509 -noout -text -in ${CA_CRT_FILE}
+    
+    echo 01 > ${CA_SRL_FILE}
+
+fi
+
+cd /certs
+
+logInfo "Generate IA key"
+openssl genrsa -des3 -passout pass:foobar -out ia.key 4096 &> /dev/null
+
+logInfo "Create a signing request for the IA: ${CAID}"
+openssl req -new -key ia.key -out ia.csr -passin pass:foobar -subj "/C=NL/ST=Noord Holland/L=Amsterdam/O=ME/OU=IT/CN=${CN_IA}" -reqexts IA -config <(
+cat <<-EOF
+[req]
+distinguished_name = dn
+[dn]
+[IA]
+basicConstraints = critical,CA:TRUE,pathlen:0
+keyUsage = critical, digitalSignature, cRLSign, keyCertSign
+subjectKeyIdentifier = hash
+EOF
+)
+
+[[ ${DEBUG} -gt 0 ]] && logInfo "Show the singing request, to make sure extensions are there"
+[[ ${DEBUG} -gt 0 ]] && openssl req -in ia.csr -noout -text
+
+logInfo "Sign the IA request with the CA cert and key, producing the IA cert"
+openssl x509 -req -days 730 -in ia.csr -CA ${CA_CRT_FILE} -CAkey ${CA_KEY_FILE} -out ia.crt -passin pass:foobar -extensions IA -extfile <(
+cat <<-EOF
+[req]
+distinguished_name = dn
+[dn]
+[IA]
+basicConstraints = critical,CA:TRUE,pathlen:0
+keyUsage = critical, digitalSignature, cRLSign, keyCertSign
+subjectKeyIdentifier = hash
+EOF
+) &> /dev/null
+
+
+[[ ${DEBUG} -gt 0 ]] && logInfo "show the IA cert details"
+[[ ${DEBUG} -gt 0 ]] && openssl x509 -noout -text -in ia.crt
+
+logInfo "Initialize the serial number for signed certificates"
+echo 01 > ia.srl
+
+logInfo "Create the key (w/o passphrase..)"
+openssl genrsa -des3 -passout pass:foobar -out web.orig.key 2048 &> /dev/null
+openssl rsa -passin pass:foobar -in web.orig.key -out web.key  &> /dev/null
+
+logInfo "Create the signing request, using extensions"
+openssl req -new -key web.key -sha256 -out web.csr -passin pass:foobar -subj "/C=NL/ST=Noord Holland/L=Amsterdam/O=ME/OU=IT/CN=${CN_WEB}" -reqexts SAN -config <(cat <(printf "[req]\ndistinguished_name = dn\n[dn]\n[SAN]\nsubjectAltName=${ALLDOMAINS}"))
+
+[[ ${DEBUG} -gt 0 ]] && logInfo "Show the singing request, to make sure extensions are there"
+[[ ${DEBUG} -gt 0 ]] && openssl req -in web.csr -noout -text
+
+logInfo "Sign the request, using the intermediate cert and key"
+openssl x509 -req -days 365 -in web.csr -CA ia.crt -CAkey ia.key -out web.crt -passin pass:foobar -extensions SAN -extfile <(cat <(printf "[req]\ndistinguished_name = dn\n[dn]\n[SAN]\nsubjectAltName=${ALLDOMAINS}"))  &> /dev/null
+
+[[ ${DEBUG} -gt 0 ]] && logInfo "Show the final cert details"
+[[ ${DEBUG} -gt 0 ]] && openssl x509 -noout -text -in web.crt
+
+logInfo "Concatenating fullchain.pem..."
+cat web.crt ia.crt ${CA_CRT_FILE}  > fullchain.pem
diff --git a/entrypoint.sh b/entrypoint.sh
diff --git a/nginx.conf b/nginx.conf

Original file line number	Diff line number	Diff line change
`@@ -1 +1,3 @@`
`1`	`1`	`.idea`
	`2`	`+docker_mirror_cache`
	`3`	`+docker_mirror_certs`