Image pull backoff until Spegel restart

[Starttoaster]

Spegel version

v0.6.0

Kubernetes distribution

EKS

Kubernetes version

v1.34.2

CNI

AWS VPC CNI

Describe the bug

Was debugging an imagePullBackoff issue in one of my clusters today. I had just upgraded the nodes yesterday from EKS v1.33 to v1.34, which I'm guessing was related. A Pod was stuck in imagePullBackoff and I noticed the event said context canceled:

Failed to pull image "ghcr.io/$IMAGE_NAME:sha-b90d62f67b7d3c7cf27fb91cc59ed546c4795630": rpc error: code = Canceled desc = failed to pull and unpack image "ghcr.io/$IMAGE_NAME:sha-b90d62f67b7d3c7cf27fb91cc59ed546c4795630": context canceled

I decided it wasn't an auth issue. I half thought it might just be the image registry having issues, but I then inspected Spegel logs, and noticed several logs like the following:

2026-01-07 10:12:10.831 ERROR{
  "time": "2026-01-07T18:12:10.831144603Z",
  "level": "ERROR",
  "source": {
    "function": "github.com/spegel-org/spegel/pkg/registry.(*Registry).mirrorHandler.func2",
    "file": "github.com/spegel-org/spegel/pkg/registry/registry.go",
    "line": 316
  },
  "msg": "request to mirror failed, retrying with next",
  "ref": "sha256:663b447ab1f2fdb605fc608e89eb65159a16a646b120c6fa76aa19c8e1df1bb6",
  "path": "/v2/$IMAGE_NAME/blobs/sha256:663b447ab1f2fdb605fc608e89eb65159a16a646b120c6fa76aa19c8e1df1bb6",
  "attempt": 1,
  "mirror": "10.101.77.184:5000",
  "err": "Get \"http://10.101.77.184:5000/v2/$IMAGE_NAME/blobs/sha256:663b447ab1f2fdb605fc608e89eb65159a16a646b120c6fa76aa19c8e1df1bb6?ns=ghcr.io\": dial tcp 10.101.77.184:5000: i/o timeout"
}

Seemed like Spegel was having some issues reaching its peers, so my obvious first step was to restart the Spegel Daemonset. That actually fixed it right away. The Pod started up nearly immediately. The incident left me with some questions/concerns however, namely:

• I would have expected a Spegel mirror issue to result in containerd bypassing the Spegel mirror and pulling the image from the registry. Should this not be the case?
• I'm not sure why Spegel was failing to reach its mirrors, when Spegel containers themselves were reporting healthy. I'd have hoped for Spegel to enter an unhealthy state and restart via a liveness probe, but apparently the readiness probes never reached an unhealthy state, and there are no liveness probes to restart Spegel in the case that it is unrecoverable. I suppose that means the only way to recover here was for me to manually restart the daemonset.
• Unfortunately I wasn't scraping Spegel's metrics endpoint during this. Were there any metrics I could have been using to watch out for issues like this? My first guess is the libp2p_swarm_dial_errors_total timeseries.

[dovi-klausner]

@Starttoaster Try WA: for now, downgrade Spegel to 0.5.1 - that's what worked for us.

We have similar deployment:

• EKS v1.34
• Nodes are AL 2023
• Containerd v2
• And yes: we've configured the NodeConfig to have the exact same values as mentioned here:

...
--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: application/node.eks.aws
Mime-Version: 1.0

---
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
  containerd:
    config: |
      [plugins.'io.containerd.cri.v1.images']
      discard_unpacked_layers = false
      [plugins."io.containerd.cri.v1.images".registry]
      config_path = "/etc/containerd/certs.d"

--MIMEBOUNDARY
...

We had Spegel 0.6.0, and we had problems similar to yours:

{
	"time":"2026-01-08T08:10:20.91717115Z",
	"level":"ERROR",
	"source": {
		"function":"github.com/spegel-org/spegel/pkg/registry.(*Registry).mirrorHandler.func2",
		"file":"github.com/spegel-org/spegel/pkg/registry/registry.go",
		"line":316
	},
	"msg":"request to mirror failed, retrying with next",
	"ref":"sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
	"path":"/v2/path/to/image/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
	"attempt":1,
	"mirror":"10.220.114.18:5000",
	"err":"Get \"http://10.220.114.18:5000/v2/path/to/image/blobs/sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1?ns=quay.io\": dial tcp 10.220.114.18:5000: i/o timeout"
}

As mentioned above: problem gone once we downgraded to v0.5.1

[Starttoaster]

@dovi-klausner unfortunately my hands are a bit tied as v0.6.0 came with some pretty important advancements to dual-stack IPv6 cluster support.

[phillebaba]

@Starttoaster this looks related to the fact that I have not set a dial timeout and there is no timeout for blobs. It makes sense that you are seeing this during an upgrade and you would end up with IPs that no longer exist. I will see if I can come up with a good fix, but next time you see this could you check if a Pod with the IP exists?

[cotzo]

had the same issue now with a node removed from cluster, pods after drain moved to new nodes and Spegel was timing out trying to get the cached image from the old node that didn't exist anymore

[Starttoaster]

@phillebaba just to be clear, in my case I'm looking for a Pod with the IP of 10.101.77.184, correct? I'll keep an eye out if I encounter this again, I update my nodes on a monthly cadence so it might happen in a few weeks. Though I also use cluster-autoscaler so it may be sooner too.

[phillebaba]

The IP that you are looking for is dynamic. Basically a dial timeout to a Pod IP that no longer exists. Its a bit hard to reproduce and test as it is so network dependent. It turns out that Kubernetes will not timeout failed dials immediately. So we either need a happy eyeballs implementation or some other solution that races to find an IP that responds. Setting a short dial timeout is short term an easy solution but there is probably a better one long term.

[Starttoaster]

Sorry, I think that's exactly what I meant. Just 10.101.77.184 in the context of the logs I'd shared above, would be what I was looking for, in any of the existing Pods at the time this issue is presenting. I'll keep a look-out for a reproduction of this. I'm running Spegel in 13 clusters so hopefully I can manage to reproduce it anyway.

[phillebaba]

Sorry sometimes I miss things and talk past people, I wish I could blame it on ESL...

I realize why I haven't thought too much about this problem. It is because I ended up re-implementing the transfer logic in the enterprise version of Spegel with a bitswap inspired solution over a libp2p stream. The benefit of this is that libp2p does dial races to find the best IP with some weights towards specific IP versions and protocols.

Once I spent some time thinking about this issue it pretty much makes sense. It is pretty logical. I could probably reproduce it locally. While dial timeouts would solve the locking waiting for the kernel to timeout the connection, it would skip a possibly good peer accessible on another IP.

[tripflex]

I can confirm these issues as well, just found this after opening an issue myself #1193 ... basically for us we have nodes scale up and down pretty frequently based on demand for GPUs. This causes a lot of issues with stale DHT entries and when a new node scales up it just times out trying to connect to stale DHT entries, ultimately defeating the purpose of caching in general for us (when it takes similar amount of time to pull the image as it does to wait for the timeout)

Ideally there should be some way to lower the TTL for entries, or have some easy way to manage or handle removing stale entries for nodes that are no longer available due to scaling up and down