Skip to content

Conversation

@joaoqalves
Copy link

Context

When users change Docker Hub (or another registry) accounts in deploy.yml after a successful deploy, kamal deploy fails with authorization errors. This occurs because
Docker buildx builders cache registry credentials, and when the builder name remains unchanged (e.g., "kamal-local-docker-container"), Docker reuses the existing builder with credentials from the previous account.

The buildx builder attempts to push using the new account credentials but still references cached layers from the old account, resulting in "insufficient_scope: authorization failed" errors. Users must manually run kamal build remove to clear the old builder before deploying with a new account.

Solution

We introduce a new optional namespace configuration option under the builder section that allows users to prefix builder names with a custom namespace. This enables users to isolate builders when switching Docker Hub accounts or when running multiple services on the same deployment host, preventing credential conflicts.

This approach was chosen over automatically including the service name in builder names because:

  1. Backward compatibility: Existing deployments continue to work without any changes. The namespace option is opt-in, so there are no breaking changes for current users.
  2. Flexibility: Users can choose their own namespace strategy. Some may want to namespace by project, by team, by environment, or by Docker Hub account. Automatically using the service name would be too rigid and might not match user needs.
  3. Explicit control: Making it a configuration option makes the intent clear and gives users control over when and how to namespace their builders. Automatic service name inclusion could cause confusion if users don't understand why their builder names changed.
  4. Opt-in migration: Users experiencing the credential conflict issue can add the namespace option to their deploy.yml and run kamal build remove to migrate, without affecting other users who aren't experiencing the problem.

Usage

builder:
  arch: amd64
  namespace: my-project

Fixes: #1383

…onflicts

Context
=====

When users change Docker Hub (or another registry) accounts
in `deploy.yml` after a successful deploy, kamal deploy fails
with authorization errors. This occurs because
Docker `buildx` builders cache registry credentials, and when the builder
name remains unchanged (e.g., "kamal-local-docker-container"), Docker
reuses the existing builder with credentials from the previous account.

The `buildx` builder attempts to push using the new account credentials
but still references cached layers from the old account, resulting in
`"insufficient_scope: authorization failed"` errors. Users must manually
run `kamal build remove` to clear the old builder before deploying with
a new account.

Solution
========

We introduce a new **optional** `namespace` configuration option under the
builder section that allows users to prefix builder names with a custom
namespace. This enables users to isolate builders when switching Docker
Hub accounts or when running multiple services on the same deployment
host, preventing credential conflicts.

This approach was chosen over automatically including the service name
in builder names because:

1. **Backward compatibility**: Existing deployments continue to work
   without any changes. The namespace option is opt-in, so there are no
   breaking changes for current users.
2. **Flexibility**: Users can choose their own namespace strategy. Some
   may want to namespace by project, by team, by environment, or by
   Docker Hub account. Automatically using the service name would be too
   rigid and might not match user needs.
3. **Explicit control**: Making it a configuration option makes the
   intent clear and gives users control over when and how to namespace
   their builders. Automatic service name inclusion could cause confusion
   if users don't understand why their builder names changed.
4. **Opt-in migration**: Users experiencing the credential conflict
   issue can add the namespace option to their deploy.yml and run
   `kamal build remove` to migrate, without affecting other users who
   aren't experiencing the problem.

Usage
=====

```yaml
builder:
  arch: amd64
  namespace: my-project
```

Fixes: basecamp#1383
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker build fails to push when switching Docker Hub accounts

1 participant