Unable to use K3d local registry in Pod spec

[witchbutter]

Question / Where do you need Help?

K3d local registry, created by K3d

Scope of your Question

k3d version v5.4.3
k3s version v1.23.6-k3s1 (default)

However I ran this with image rancher/k3s:v1.22.10-k3s1. Here is my k3d.yaml I pass at creation time:

apiVersion: k3d.io/v1alpha4
kind: Simple
servers: 1
agents: 2
image: rancher/k3s:v1.22.10-k3s1
registries:
  create:
    name: k3d-registry
    hostPort: "5005"
ports:
  - port: 8080:8080
    nodeFilters:
      - loadbalancer

I am trying to use K3d with skaffold so that I can change worker node configurations and use Traefik. I have this problem even when skaffold is not used and I just use kubectl apply -f. I create a k3d cluster with a registry. I can tag and push to that registry from my local mac, but it is not usable within the cluster.

I see that the K3d local registry is insecure and using HTTP. I hope that's not an issue but this is the error I get:

  Warning  Failed       22s (x3 over 65s)  kubelet            Failed to pull image "k3d-registry.localhost/miniweb:latest": rpc error: code = Unknown desc = failed to pull and unpack image "k3d-registry.localhost/miniweb:latest": failed to resolve reference "k3d-registry.localhost/miniweb:latest": failed to do request: Head "https://k3d-registry.localhost/v2/miniweb/manifests/latest": x509: certificate is valid for 44109fba3105bd338a04cc3f070f8eeb.d64161662d0012df32818ec7be5df296.traefik.default, not k3d-registry.localhost

It fails if I use k3d-registry.localhost/containername or if I use k3d-registry/containername in the pod spec. I have checked CoreDNS and its ConfigMap shows the registry:

NodeHosts:
----
192.168.65.2 host.k3d.internal
172.21.0.5 k3d-miniweb-agent-1
172.21.0.4 k3d-miniweb-agent-0
172.21.0.3 k3d-miniweb-server-0
172.21.0.6 k3d-registry
172.21.0.2 k3d-miniweb-tools
172.21.0.7 k3d-miniweb-serverlb

[iwilltry42]

Hi @witchbutter , thanks for opening this issue!

You're actually missing out the port there in the pod spec: k3d-registry:5000 (default registry internal port) or k3d-registry:5005 (port you set).

[witchbutter]

I did some more experimentation. As it turns out no matter what local port I set, in this case 5005, the port used inside the kubernetes cluster is 5000. So even thought my skaffold.yaml says:

registries:
  create:
    name: k3d-registry
    hostPort: "5005"

The port exposed to docker on the laptop is 5005, but the kubernetes internal port is 5000.

Also it seems that using the 'latest' tag in the deployment yaml doesn't work because I had to run skaffold run --tail --tag latest in order to get the deployment to start.