Registry behind Cloudflare Tunnel/Proxy - fail to push image (client disconnected during blob PATCH)

[nktnet1]

There's a limit of 100MB on Cloudflare Free/Pro plan, which means the image pushing process to the configured rollback registry may fail unexpectedly. On the registry logs, we see:

fail to push image (client disconnected during blob PATCH)

Does anyone have a workaround for this apart from disabling the proxy? (which isn't viable with Cloudflare Tunnel)

Using a VPN/proxy like wireguard could be worth looking into.

References:

• Image registry error when layer is larger than 1GB sameersbn/docker-gitlab#2819
• Issue during push "client disconnected during blob PATCH" goharbor/harbor#20472
• https://www.reddit.com/r/selfhosted/comments/17tr4l9/private_docker_registry_behind_cloudflare_tunnel/
• https://gitlab.com/gitlab-org/charts/gitlab/-/work_items/1347
• https://forum.gitlab.com/t/runner-cannot-push-all-layers-to-registry-504-gateway-timeout-self-hosted-kubernetes-cloud-native-helm-chart/26720/2
• https://news.ycombinator.com/item?id=41459288

[djsisson]

if you are pushing internally, then setup an sni with an internal ip address, e.g registry.example.com 10.0.0.2
where this is the lan ip of the node running traefik, or any node if traefik is ran as a service

make sure you use dns challenge for ca certs, assuming you have blocked all ports, meaning internal requests will see a valid ca

if you want to push via a runner, you can run cloudflare one as part of the pipeline to connect to your internal network with split tunnels, the above name will still resolve bypassing any limits

[nktnet1]

Thank you for replying!

Hmm the pushing is done from a VPS to a home lab (different physical location and network). The home lab has no public IP addresses or open ports (won't be possibele to open), and strictly uses Cloudflare Tunnel to expose registry.domain.com as HTTPS.

It's also triggered by Dokploy itself (the registry is created in the "Registry" page from the sidebar, then enabled and chosen for my application under the "Deployment" tab), so not sure if I have much control over it :/.

Documentation:

• https://docs.dokploy.com/docs/core/applications/rollbacks#registry-based-rollback-to-specific-versions

[djsisson]

this works the same way as pushing from an action runner, you need your vps to be using cloudflare one, such that its also on on the virtual cf network with your home lab

then point your registry dns name to the private vip of the cf network of your homelab, this keeps the traffic inside the cf network, so it doesn't cross the external boundary which has the limits

[nktnet1]

Thanks @djsisson.

I ended up going with Tailscale serve (since I already have Tailscale set up) and that resolved the issue.

E.g. assuming you have the port mapping 5000:5000 from the container to host:

sudo tailscale serve -bg 5000

With Cloudflare, I believe there's Cloudflare Mesh which could've also achieved the same result.