How to fix Traefik v2 / v3 (v2.11.x / v3.x.x) stopping Immich uploads that go longer than 60 seconds (502 Bad Gateway)

[othyn]

TLDR, here's the fix:

Add the following to your Traefik config and restart the Traefik instance. For more examples, see the docs: https://doc.traefik.io/traefik/routing/entrypoints/#respondingtimeouts

entryPoints:
  ...
  name:
    ...
    # This is the fix:
    transport:
      respondingTimeouts:
        readTimeout: "0s"

---

What's going on?

I was having issues with Immich not being able to upload large files. The app would stop uploading, whilst the app remained responsive, during large file uploads. This is new behaviour, as I've been able to upload to my hearts content for months. So what's changed? My hunch was it was network related, so I transferred the same file onto my desktop to see if the upload still failed via a browser, it did.

Digging deeper into the issue, the first thought was that it was buffer related, as these things usually are. But as I'm using Traefik, which by default doesn't impose/define any buffer or upload file size limitations, as it tries to behave as much like a transparent reverse proxy as possible.

Doing the file transfer on a desktop was also handy for one main reason, the network inspector. Popping that open revealed that requests were being terminated prematurely with a 502, but didn't reveal anything else of value, even the request timings (which will be important later), as they weren't that accurate and included the transport and execution times from the browser as well. So nothing stood out as being the problem.

Next port of call was enabling DEBUG level logging and access logging on the Traefik instance to try and debug what on earth was going on. After doing that, all became clear:

# Only access logging enabled at first...
10.10.0.10 - - [17/Apr/2024:10:34:32 +0000] "POST /api/asset/upload HTTP/2.0" 502 11 "-" "-" 638 "photos@docker" "http://172.18.111.1:3001/" 60000ms
10.10.0.10 - - [17/Apr/2024:10:36:28 +0000] "POST /api/asset/upload HTTP/2.0" 502 11 "-" "-" 855 "photos@docker" "http://172.18.111.1:3001/" 60000ms

# Hello! That 60000ms is an immediate red flag and smells like Traefik doing something to the connection as its bang on 60000ms each time. If it was anything else in the network chain that would be +/- a few ms as overhead is added in.
# Let's enable the DEBUG log level to see why it is that Traefik may be terminating the connection...

# Now enabling both a DEBUG log level and access logging, it revealed the important bits...
time="2024-04-17T11:47:54+01:00" level=debug msg="'502 Bad Gateway' caused by: readfrom tcp 172.18.1.100:47366->172.18.111.1:3001: i/o timeout"
10.10.0.10 - - [17/Apr/2024:10:46:54 +0000] "POST /api/asset/upload HTTP/2.0" 502 11 "-" "-" 201 "photos@docker" "http://172.18.111.1:3001/" 60000ms

The important thing here is it revealed that it was indeed Traefik killing the connection, and crucially why Traefik killed the connection, the readfrom config value was set to 60s and therefor triggering the i/o timeout. But this is new, no? It's never done this before. Why is it doing it now?

Burned by Watchtower. It automatically upgraded to Traefik v2.11.2 that has a breaking change in a minor semver patch version, a big no-no in the software game. There is an ongoing discussion as to why exactly this change came in, and its lead to a lot of heated discussion on their repo. It also revealed why the issue was so hard to track down information for online, as the issue was only opened less than a week ago.

Finding the fix!

Initial bug thread: traefik/traefik#10592
The culprit: traefik/traefik#10594 (comment)
The wider discussion thread: traefik/traefik#10598

Tucked away right at the bottom of the changelog, https://doc.traefik.io/traefik/migration/v2/#v2112, is this little doozy:

EntryPoint.Transport.RespondingTimeouts.ReadTimeout

Starting with v2.11.2 the entryPoints readTimeout option default value changed to 60 seconds.

For HTTP, this option defines the maximum duration for reading the entire request, including the body. For TCP, this option defines the maximum duration for the first bytes to be read on the connection.

The default value was previously set to zero, which means no timeout.

This change has been done to avoid Traefik instances with the default configuration to be hanging forever while waiting for bytes to be read on the connection.

Increasing the readTimeout value could be the solution notably if you are dealing with the following errors:

• TCP: Error while handling TCP connection: readfrom tcp X.X.X.X:X->X.X.X.X:X: read tcp X.X.X.X:X->X.X.X.X:X: i/o timeout
• HTTP: '499 Client Closed Request' caused by: context canceled
• HTTP: ReverseProxy read error during body copy: read tcp X.X.X.X:X->X.X.X.X:X: use of closed network connection

And there we go, the i/o timeout error being linked back to it being a change to the default EntryPoint.Transport.RespondingTimeouts.ReadTimeout value.

Checking the docs for this section, we can now apply a fix (notice that the docs now say it defaults to 60 seconds instead of being disabled, further confirming this as the culprit): https://doc.traefik.io/traefik/routing/entrypoints/#respondingtimeouts

entryPoints:
  ...
  name:
    ...
    # This is the fix:
    transport:
      respondingTimeouts:
        readTimeout: "0s"

This will revert readTimeout to its previous default value of being disabled, instead of 60 seconds, and solve the issue.

Happy uploading!

[gartsolutions]

Great job! We had the same issue! Rollback traefik version to 2.11.1 fixed issue.

[othyn]

Glad it helped! Yep, also a valid fix too

[truncsphere]

@othyn
Thanks for this, I'm just starting with Immich and have configured with Traefik already.
Did you add a port to your config for the timeout per the docs? I have it setup as such currently:

 [entyPoints.immich]
   [entryPoints.immich.transport]
     [entryPoints.immich.transport.respondingTimeouts]
       readTimeout = 0```

[lakemike]

Thanks! I'm using traefik 3.0.0 and I also had to change the timeout. This is how it looks like in docker-compose.yml if your endpoint is called websecure (command section):
- --entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s

[jbcodemonkey]

I am using traefik on synology in front of immich. Videos larger than 700MB were all failing at ~670MB with container log message:
"WARN [Api:FileUploadInterceptor] Request error while uploading file, cleaning up"

This was the fix! All large files are now uploading successfully.

Thank you!

[AlexeyMatskevich]

If you use truenas scale and install immich from truecharts (with their traefik) - then you can add "Extra Args" in gui for traefik chart with value

--entryPoints.websecure.transport.respondingTimeouts.readTimeout=0

[Alfonce76]

do you need to add under http and https entry points if you have them? or is http enough?

[lakemike]

Not sure I fully understand your question. http would be immich itself and, hence, not required.
My endpoint is called websecure, other setups may call it https. This is where you need it.

[danni-storm]

I am on traefik 3.1 and it was driving me crazy that large uploads kept receiving gateway errors. Just pasted this into my traefik.yml file for websecure and web, restarted the traefik container, and it uploaded right away. Perfect! Thank you so much for figuring this out!

    transport:
      respondingTimeouts:
        readTimeout: 0s

[voltage6082]

Thank you all, I was desperately trying to solve this. I did not notice it with immich but with any other service (Synology DSM, Nextcloud, etc.). When uploading large files >2GB, the upload timed out and I got a standard error inside the application that the file could not be uplaoded. I rolled back to version v2.10.6 and it started working again.
I made the following change to my static configuration (traefik.yaml):

entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
    transport:
      respondingTimeouts:
        readTimeout: 0s
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        readTimeout: 0s

and now it works with the :latest release of traefik, currently 3.1.1
Thanks for your help!

[martins250]

Thanks for the above. I also came across the above problem. It needed both web and websecure timeout lines to fix (works with Traefik 3.1), as follows:

- --entrypoints.web.transport.respondingTimeouts.readTimeout=0s
- --entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s

[alexrec-alt]

Traefik docs says default value for 'readTimeout' is '0s'. Anyways setting this value fixes upload 502 Bad Gateway issue. I decided to set this in config instead of labels, as there is number of proxied endpoints with download/upload options.

[joselsegura]

Hi!

Thank you for your great research about the timeout problem.

I followed your suggestion and set the read timeout to 0s using the already discussed --entrypoints.websecure.transport.respondingtimeouts.readtimeout=0s

In my case, the Immich Android client upload is not reset, but I noticed an important drop on the speed of the connection. Is that normal or expected? I don't think so, but just double checking with you.

[othyn]

It's a known issue with the upstream package they use for http transfers and https overheads. More powerful devices can just brute force it, but phones tend to struggle a bit more.

#8560 (reply in thread)

I leave the port open on my host and just use the direct IP:port of the Immich server for mobile devices, it's far faster.

[joselsegura]

The weird thing is that, before using the readtimeout option, it worked quite faster than now... but it cannot complete in 1 minute. Now, it takes ages, but the connection is not abruptly cut...

[othyn]

Okay weird, never heard of or had that problem so no idea on that one I'm afraid!

[joselsegura]

Just for the record: I tried to modify the configuration from 0s to 300s (an appropriate limit, if it works at full speed), but it seems to be as slow as with 0s.

Just a question for people having this problem syncing from mobile: if I upload the same file from a browser, is your Android app aware that the file is the same and it will stop trying to sync it???

In the past I had a problem with a large file that was not uploaded from Android app because the issue with traeffik readTimeout, but when I uploaded it using immich-go, the Android app was trying to upload it over and over again until I removed it from my Android phone storage

[joselsegura]

Now I'm trying the direct upload to the local IP. I exposed the Immich HTTP port to the host machine and connecting using http://MY_LOCAL_IP:2283. It is painfully slow, so it should be a problem in the Android client, even when you are connecting directly to the server

[kazoob]

For reference, the Immich documentation has a configuration example for Traefik v3 with the following timeouts:

[...]
entryPoints:
  websecure:
    address: :443
    # this section needs to be added
    transport:
      respondingTimeouts:
        readTimeout: 600s
        idleTimeout: 600s
        writeTimeout: 600s

https://immich.app/docs/administration/reverse-proxy#traefik-proxy-example-config