[Feature request] Support self-hosted registry out of the box.

[m3thom]

To utilize all kamal features (e.g., zero downtime, rollback, etc.), I need a registry.

In my case, I have only one server right now and I don't want to use any external registry provider.
So, I want to host both my application and my registry on the same server.

From my understanding, it's fairly easy to host a private registry Docker Distribution. As of now, I believe the Kamal ecosystem could support this.

We could categorize this under the kamal accessories section since we don't need to build the registry.
We could use kamal-proxy to manage the network and TLS.

For the authentication setup, there might be some manual steps involved, but it's quite straightforward.

$ mkdir auth
$ docker run \
  --entrypoint htpasswd \
  httpd:2 -Bbn testuser testpassword > auth/htpasswd

While these steps need to be done manually, is it possible to support this self-hosted registry feature out of the box?

Thank you for this amazing library! I really want to use it for my application!

[m3thom]

Or we could have something like this:

registry:
  server: registry.app.io  # kamal-proxy will use this for the registry networking and TLS.
  username: myuser
  password: KAMAL_REGISTRY_PASSWORD
  self_host:
    image: registry:2 # Specific the image to use from the Docker.
    host: my.app.io # The host server to deploy to.
    # registry volume, mount, driver and other settings.

[kristianpaul]

For one single server its okay to change docker configuration, may be even for every various hosts since kamal could accommodate docker config.json to work with local registries?

[kjlape]

I love this idea, and I've given some thought to a few possible solutions. I'm not familiar with kamal's codebase yet, but I'd be happy to help out with the implementation if we strike a concordance here.

Challenge

Balance simple first-time setup with security. We need an easy way to rotate keys for our Docker registry if we're going to maintain one ourselves.

Potential Solutions

1. Kamal Command Extension

• Add a new command to Kamal that wraps the OP's httpd command
• This would handle auth rotation directly through Kamal's CLI interface
• Benefit: Maintains consistency with existing Kamal workflows

2. Separate Configuration File

• Instead of expanding the main config, create config/registry-deploy.yml
• Treats registry as an independent Docker service
• Benefit: Cleaner separation of concerns, more maintainable long-term

Registry Bootstrap Process

It's a chicken or the egg problem! How do we deploy a registry as a container without leaning on cloud solutions?

Initial Setup Options

1. Direct Image Transfer (Recommended)

   • One-time transfer of raw image file to server
   • Simple, straightforward approach
   • Minimal complexity in implementation

2. Tunneling Approaches (Considered but not recommended)

   • As a prototype, I've used ngrok for tunneling
   • Alternative: SSH tunnel back to local machine for self-hosted registry access
   • Drawback: More complex than direct image transfer without clear advantages

Recommendation

Use a separate configuration file approach (config/registry-deploy.yml or something) combined with direct image transfer for bootstrap. This provides a simple architecture. We all understand copying files to a server.

See also #69. I forgot I commented there too! 😅 The comment you're looking at is a more complete rundown of my thoughts.

Bonus: Workaround for Today

I've been experimenting with using ngrok to bootstrap my docker registry. It's a bit of a hack, but it works with kamal today. I have a kamal deploy config that deploys a Dockerfile based on the registry:2 image (with htpasswd auth baked in). I run that image locally, use ngrok to proxy it to the web, then deploy using the ngrok url as the registry server. After that, I can change the docker registry to the remote hosted one and use that for future deploys.

[m3thom]

@pingortle Thank you for your comment. I only have basic knowledge about Docker-related things, so I may be wrong, but here are my thoughts:

Balance simple first-time setup with security. We need an easy way to rotate keys for our Docker registry if we're going to maintain one ourselves.

For this point, the approach I have in mind should be able to handle the situation because it behaves somewhat like a registry provider. All you need to do is change the KAMAL_REGISTRY_PASSWORD environment variable for secret rotation.

1. Self-hosted registry: The registry will be hosted on the same server as my app. This should eliminate the need for tunneling.
2. Deployment process for a new server:
   1. Even though I have only one app for now, there may be more in the future, so it's a good idea to separate the self-hosted registry setup into a new repository. So we can still use the config/deploy.yml file, but using config/registry-deploy.yml for clarity is also an option.
   2. The self-hosted registry repo should be the first app to deploy to the server.
   3. To avoid the chicken-or-egg problem, we need to tweak how kamal bootstraps the server. When we run the kamal setup command, it will usually try to log in to the registry. Instead, it should detect the self-hosted configs and use them to set up the registry (image pull, basic auth setup, networking, etc.).

# my-registroy/config/deploy.yml
registry:
  server: registry.app.io  # kamal-proxy will use this for the registry networking and TLS.
  username: myuser
  password: KAMAL_REGISTRY_PASSWORD
  self_host:
    image: registry:2 # Specific the image to use from the Docker.
    host: my.app.io # The host server to deploy to.
    # registry volume, mount, driver and other settings.

4. Once the registry is set up, I should be able to deploy my app to the same server using that registry.

# my-app/config/deploy.yml
registry:
  server: registry.app.io
  username: myuser
  password: KAMAL_REGISTRY_PASSWORD

5. In the future, if we have more servers for other apps, they should be able to use the same registry configurations without any issues.

Since we are self-hosting the registry, we will also need to handle the storage system ourselves.

• Registry data could be lost: For me, if the data is lost, I can always rebuild my app.
• Garbage collection: We can handle this by setting up garbage collection as a cron job. However, for now, running the garbage collection command manually with kamal alias once a week should be sufficient.

[kjlape]

This is making sense to me. The smallest tweak we could make to kamal would be to allow deploying a public image without a build step like we already support for accessories. We wouldn't need to special-case the docker registry in that case.

Here's my (redacted, but otherwise complete) experimental deploy.yml that I'm running for a couple projects right now.

You may notice the ngrok url as the docker host. That was just to bootstrap the first deploy, which is what I would like to avoid with the feature we are discussing now.

service: docker-registry
image: my-company/docker-registry
servers:
  web:
    - 1.3.3.7

proxy:
  ssl: true
  host: docker.my-company.com
  app_port: 5000
  healthcheck:
    path: /

registry:
  username: username
  server: <some url>.ngrok-free.app
  password:
    - KAMAL_REGISTRY_PASSWORD

builder:
  arch: arm64 # This is the platform I tested on, but it should work on any platform.

volumes:
  - auth:/auth

env:
  clear:
    REGISTRY_HTTP_ADDR: 0.0.0.0:5000
    REGISTRY_AUTH: htpasswd
    REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm
    REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd

aliases:
  shell: app exec --interactive --reuse "sh"
  htpasswd-set: server exec docker run --volume auth:/auth --rm --entrypoint htpasswd httpd:2 -Bb /auth/htpasswd
  htpasswd-delete: server exec docker run --volume auth:/auth --rm --entrypoint htpasswd httpd:2 -D /auth/htpasswd
  htpasswd-print: server exec docker run --volume auth:/auth --rm --entrypoint cat httpd:2 /auth/htpasswd

The Dockerfile is just a single line: FROM registry:2. If we could skip build, push, and login as you suggest, I think this setup would work just fine. Would we want to build automatic docker registries into kamal eventually? Perhaps. But I think enabling this style of pre-built image deploys is a good first step.

Would something like this work for you?

[m3thom]

Currently, I'm still using our good old friend Capistrano, but your code looks good to me.

P.S. From the Kamal 2.0 talk at Rails World 2024 (started from 4:10), it seems the Kamal team has some plans on the registry topic. I wonder if they could share some information with the community.

[djmb]

Going to close this one, as #1355 gives you the ability to deploy with a local registry (though it runs on the deploy host, not the application one)