Gitlab private registry only 1 node of 3 get ErrImagePull

[Sampozzo]

Description

I'm trying to deploy 3 replicas of an image from a private gitlab registry. I setup a secret like this:

---
apiVersion: v1
data:
  .dockerconfigjson: <mybase64secret>
kind: Secret
metadata:
  creationTimestamp: "2023-05-23T12:34:58Z"
  name: gitlab-pull-secret
  namespace: test-namespace
  resourceVersion: "64982"
  uid: 095f3454-5ede-4f10-9e72-5bdb92a72e8b
type: kubernetes.io/dockerconfigjson

it works on 2 of 3 nodes, and i have one that get ErrImagePull. How is it possible?

This is a clean cluster created now for testing.

Kube.tf file

my kube.tf is really simple like the example, 3 masters and 3 workers.

Screenshots

Describe the pod in error:

Platform

Linux

[Sampozzo]

update: i've tryed deleting and recreating the node, i've tryed recreating 4 times the whole cluster, with ARM and AMD... always the same problem, 2 pods working, 1 pod in error.

The problem is always on one node, if i try to create a 9 replica, all the pods on one worker doesn't pull the image.

[mysticaltech]

@Sampozzo Even if you recreate many times Hetzner still gives you the same IP for one of the nodes. You need to contact Hetzner support and let them know that the IP in question is blacklisted by the Google Container Registry (probably what gitlab users under the hood). So that they change that IP for you.

[Sampozzo]

Hi @mysticaltech thank for the reply. Just to be sure, i made a new cluster 1.26 (i was using 1.25) and now all is working. Don't know if it is a coincidence or not.

The problem now is that rancher doesn't like 1.26 to install cluster tools :)