Merge pull request #3 from obeone/feature/private-registry

Private registry

Author: obeone

.github/workflows/build-and-publish.yaml

@@ -70,7 +70,7 @@ jobs:
           COSIGN_EXPERIMENTAL: true
           DIGEST: ${{ steps.docker_build.outputs.digest }}
 
-      - name: Updage Docker Hub description
+      - name: Update Docker Hub description
         uses: peter-evans/dockerhub-description@v4
         with:
           username: ${{ secrets.DOCKERHUB_USERNAME }}

README.md

@@ -1,12 +1,26 @@
 # Multi-Registry Pull Through Cache Setup Guide 🚀
 
-Welcome to the Multi-Registry Pull Through Cache Setup Guide! This project is designed to help you create an efficient, bandwidth-saving local mirror for Docker Hub images and other container registries. By setting up a pull-through cache, you can significantly reduce internet traffic and improve the speed of image pulls for your containerized environments. 🌐💨
+Welcome to the Multi-Registry Pull Through Cache Setup Guide! This project is designed to help you create an efficient, bandwidth-saving local mirror for Docker Hub images and other container registries, including private registries. By setting up a pull-through cache, you can significantly reduce internet traffic and improve the speed of image pulls for your containerized environments. 🌐💨
 
 This script will generate a `compose.yaml` file (same as `docker-compose.yml`). It will include one registry service for each registry mirror you wish to set up. A Traefik Proxy will be placed as a frontend for routing and providing a TLS endpoint. Additionally, a Redis service will be provided to enhance performance.
 
+## Features ✨
+
+- **Multi-Registry Support**: Set up mirrors for Docker Hub, GitHub Container Registry (GHCR), or any other container registries.
+- **Private Registry Integration**: Easily add a private registry to push and pull images from your private repositories.
+- **Bandwidth Optimization**: Save bandwidth by caching container images locally, reducing repeated downloads.
+- **Speed Improvements**: Expedite image pulls within your network, leading to faster deployments and scaling.
+- **Docker and Kubernetes Compatibility**: Seamlessly integrates with Docker and Kubernetes environments.
+- **Advanced Configuration**: Customize registry settings, including TTL and credentials, with a user-friendly setup.
+- **Simple Deployment**: Leverage Docker Compose for a straightforward setup and management of services.
+- **Traefik Proxy Integration**: Use Traefik for intelligent routing and secure TLS endpoints out of the box.
+- **Performance Enhancements**: Utilize Redis to improve caching operations and reduce latency.
+- **Environment-Friendly**: Lower external data transfer, contributing to a reduced carbon footprint.
+
 ## Table of Contents 📚
 
 - [Multi-Registry Pull Through Cache Setup Guide 🚀](#multi-registry-pull-through-cache-setup-guide-)
+  - [Features ✨](#features-)
   - [Table of Contents 📚](#table-of-contents-)
   - [Purpose of the Project 🎯](#purpose-of-the-project-)
   - [How to Set Up the Project 🛠️](#how-to-set-up-the-project-️)
@@ -46,7 +60,6 @@ By using a pull-through cache, you can:
 - Access to the internet to pull initial images
 - Basic knowledge of Docker, Kubernetes, and container registries
 
-
 ### Running with Docker
 
 #### Run the Setup Script
@@ -100,7 +113,7 @@ docker run --rm -ti -v "./config.yaml:/app/config.yaml" -v "./compose:/app/compo
    ```
 
 5. **Review Advanced Configuration**
-   After running the setup script, you should manually review the `config.yaml` file for advanced configurations such as TLS settings, Let's Encrypt, and more. More details about this file in [CONFIG.md](CONFIG.md)
+   After running the setup script, you should manually review the `config.yaml` file for advanced configurations such as TLS settings, Let's Encrypt, addition of private registries, and more. More details about this file in [CONFIG.md](CONFIG.md)
 
    ```bash
    nano config.yaml # or use your preferred text editor
@@ -199,12 +212,12 @@ For Kubernetes clusters, you'll probably need to configure each node's container
 
 ## Conclusion 🎉
 
-Congratulations! You've now set up a multi-registry pull-through cache that will serve as a local mirror for your container images. Enjoy faster image pulls, reduced external bandwidth, and a more resilient container environment!
+Congratulations! You've now set up a multi-registry pull-through cache that will serve as a local mirror for your container images, including private registry configurations. Enjoy faster image pulls, reduced external bandwidth, and a more resilient container environment!
 
 ---
 
 Feel free to contribute to this project by submitting issues or pull requests. For questions or support, open an issue on the project's GitHub page. Happy caching! 🐋💾
 
 ---
 
-**Keywords**: Docker, container registry, pull-through cache, Docker Hub mirror, containerd, dockerd, Kubernetes, k3s, RKE, RKE2, image caching, setup guide, local mirror, container image optimization.
+**Keywords**: Docker, container registry, pull-through cache, Docker Hub mirror, private registry, containerd, dockerd, Kubernetes, k3s, RKE, RKE2, image caching, setup guide, local mirror, container image optimization.

config.sample.yaml

@@ -5,24 +5,31 @@
 # in the compose file. The url is the url of the registry. The username and password are the credentials for the registry.
 # You can add elements to the list. The setup will generate the compose file for every registry, and these elements will
 # be available for generation of parts configs (like traefik, see below)
+# Type is `cache` for caching registries, `registry` for normal registries.
 registries:
-- name: dockerhub
-  url: https://registry-1.docker.io
-  username: user
-  password: pass
-- name: ghcr
-  url: https://ghcr.io
-  username: user
-  password: pass
-- name: nvcr
-  url: https://nvcr.io
-  username: user
-  password: pass
-- name: quay
-  url: https://quay.io
-  username: user
-  password: pass
-  ttl: 720h
+  - name: dockerhub
+    type: cache
+    url: https://registry-1.docker.io
+    username: user
+    password: pass
+  - name: ghcr
+    type: cache
+    url: https://ghcr.io
+    username: user
+    password: pass
+  - name: nvcr
+    type: cache
+    url: https://nvcr.io
+    username: user
+    password: pass
+  - name: quay
+    type: cache
+    url: https://quay.io
+    username: user
+    password: pass
+    ttl: 720h
+  - name: private
+    type: registry
 
 docker:
   # The docker compose config. This config is used to generate the docker-compose.yaml file. You can use the baseConfig
@@ -36,15 +43,15 @@ docker:
         image: traefik:v2.10
         restart: always
         command:
-        - '--providers.file.filename=/etc/traefik/traefik.yaml'
+          - "--providers.file.filename=/etc/traefik/traefik.yaml"
         ports:
-        - '80:80'
-        - '443:443'
+          - "80:80"
+          - "443:443"
         volumes:
-        - './traefik.yaml:/etc/traefik/traefik.yaml:ro'
-        - './acme:/etc/traefik/acme:rw'
+          - "./traefik.yaml:/etc/traefik/traefik.yaml:ro"
+          - "./acme:/etc/traefik/acme:rw"
         networks:
-        - 'registries'
+          - "registries"
         # Here is a sample of how to use environment variables in the docker compose config
         # used to configure (one of) ACME DNS challenge
         # environment:
@@ -57,27 +64,26 @@ docker:
         image: redis:7.2
         restart: always
         networks:
-        - 'registries'
+          - "registries"
         volumes:
-        - './redis.conf:/usr/local/etc/redis/redis.conf:ro'
+          - "./redis.conf:/usr/local/etc/redis/redis.conf:ro"
         command:
-        - redis-server
-        - /usr/local/etc/redis/redis.conf
+          - redis-server
+          - /usr/local/etc/redis/redis.conf
 
     networks:
       registries: {}
 
-
   perRegistry:
     compose:
       image: registry:2
       restart: always
       volumes:
-      - './{name}.yaml:/etc/docker/registry/config.yml:ro'
+        - "./{name}.yaml:/etc/docker/registry/config.yml:ro"
       networks:
-      - registries
+        - registries
       environment:
-      - 'REGISTRY_HTTP_SECRET=$REGISTRY_HTTP_SECRET'
+        - "REGISTRY_HTTP_SECRET=$REGISTRY_HTTP_SECRET"
 
 traefik:
   # The traefik config. This config is used to generate the traefik.yaml file. As with the docker compose config, you can
@@ -132,26 +138,26 @@ traefik:
     router:
       rule: "Host(`{name}.registry-cache.example.net`)"
       entryPoints:
-      - web
-      - websecure
+        - web
+        - websecure
 
       # You must use the placeholder {name} in the service. The setup will replace the placeholder with the name of the registry.
-      service: '{name}'
+      service: "{name}"
       tls: {}
 
     service:
       loadBalancer:
         # You must the declared service name in the compose config.
         servers:
-        - url: "http://{name}:5000"
+          - url: "http://{name}:5000"
 
 registry:
   # The registry config is the same as the official registry config (https://docs.docker.com/registry/configuration/)
   # in one file. As it a different file for every registry, you can use placeholders from the registry config in the
   # whole config. The setup will replace the placeholder with the value from the registry config. For example, the
   # placeholder {name} will be replaced with the name of the registry.
   baseConfig:
-    version: '0.1'
+    version: "0.1"
     health:
       storagedriver:
         enabled: true
@@ -161,7 +167,7 @@ registry:
       addr: :5000
       headers:
         X-Content-Type-Options:
-        - nosniff
+          - nosniff
     log:
       fields:
         service: registry
@@ -186,4 +192,5 @@ registry:
       # Don’t set a db number. If redis key exists, db will be set to the right number.
 
     # Proxy will be configured by the setup. You basically don't need to touch this.
+    # Proxy will be removed in case of normal registry type. It is only used for cache registries.
     proxy: {}

functions.py

@@ -104,12 +104,17 @@ def create_registry_config(config, registry, db):
     Returns:
         dict: The updated configuration.
     """
-    config['proxy']['remoteurl'] = registry['url']
-    if 'username' in registry and 'password' in registry:
-        config['proxy']['username'] = registry['username']
-        config['proxy']['password'] = registry['password']
-    if 'ttl' in registry:
-        config['proxy']['ttl'] = registry['ttl']
+    
+    if registry['type'] == 'cache':
+        config['proxy']['remoteurl'] = registry['url']
+        if 'username' in registry and 'password' in registry:
+            config['proxy']['username'] = registry['username']
+            config['proxy']['password'] = registry['password']
+        if 'ttl' in registry:
+            config['proxy']['ttl'] = registry['ttl']
+    elif 'proxy' in config:
+        del config['proxy']
+        
 
     interpolated = interpolate_strings(config, registry)
 

generate.py

@@ -8,24 +8,25 @@
 import functions
 from functions import console
 from rich.text import Text
+import copy
 
 
 # Load config
 try:
     with open('config.yaml', 'r', encoding='UTF-8') as file:
-        config = yaml.safe_load(file)
+        base_config = yaml.safe_load(file)
         console.print(Text("Config loaded successfully", style="bold green"))
 except FileNotFoundError:
     console.print(Text("Error: config.yaml not found", style="bold red"))
     raise
 
 try:
-    registries = config['registries']
-    docker_config = config['docker']['baseConfig']
-    docker_perregistry = config['docker']['perRegistry']
-    traefik_config = config['traefik']['baseConfig']
-    traefik_perregistry = config['traefik']['perRegistry']
-    registry_config = config['registry']['baseConfig']
+    registries = base_config['registries']
+    docker_config = base_config['docker']['baseConfig']
+    docker_perregistry = base_config['docker']['perRegistry']
+    traefik_config = base_config['traefik']['baseConfig']
+    traefik_perregistry = base_config['traefik']['perRegistry']
+    registry_config = base_config['registry']['baseConfig']
 except KeyError as e:
     console.print(Text(f"Error: Missing key in config file - {e}", style="bold red"))
     raise
@@ -45,7 +46,13 @@
 
     # Creating registry configuration file
     try:
-        registry_config_file = functions.create_registry_config(registry_config, registry, count_redis_db)
+        # Retrocompatibility with old config files without type field
+        if 'type' not in registry:
+            registry['type'] = 'cache'
+            console.print(Text(f"No type specified for registry {name}, defaulting to cache", style="bold yellow"))
+
+        registry_config_copy = copy.deepcopy(registry_config)
+        registry_config_file = functions.create_registry_config(registry_config_copy, registry, count_redis_db)
         functions.write_yaml_file(f'compose/{name}.yaml', registry_config_file)
         console.print(Text(f"Registry configuration file created for {name}", style="bold green"))
     except Exception as e:
@@ -54,7 +61,8 @@
 
     # Unsset password before interpolate variables
     try:
-        registry.pop('password')
+        if 'password' in registry:
+            registry.pop('password')
     except KeyError:
         console.print(Text(f"Error: Missing 'password' key in registry configuration for {name}", style="bold red"))
         raise