Unable to update firmware using port 443 #316
-
Describe the BugI'm not totally sure if this is a bug with the container or controller software itself. I've been running the docker container and accessing it via Traefik using the default management port (8043) for a while. Using some redirect/header modification rules in Traefik, I don't have any issues updating device firmware. However, I recently switched to using port 443 as the management port to simplify my Traefik config. After the switch, device firmware updates stopped working. The file starts uploading but never seems to complete and eventually times out after 10 minutes. I wound up reverting my Traefik/Omada updates, so it was using port 8043 again, and once I did that, the firmware update completed successfully. At this point, I feel like the issue is likely related to the Omada software, but when I brought it up on their forum, I was advised to come here instead. Expected BehaviorDevice firmware updates should complete successfully. Steps to Reproduce
How You're Launching the ContainerContainer LogsAdditional ContextI'd like to try running the controller software natively, but on port 443 to try and rule out Docker or Traefik as the culprit, but am not sure how easy it would be to convert the docker install to bare-metal. Would it be just installing the native client (probably on a spare machine) and copying the three bind mount folders to the same locations after the install is complete? |
Beta Was this translation helpful? Give feedback.
Replies: 20 comments 2 replies
-
|
Well, when looking at this FAQ document, I am not sure what it means when it comes to port
The usage for number two there is what I am not exactly sure of. When I uploaded a firmware to perform a downgrade on one of my WAPs, it didn't start listening on any other ports so I don't think it needs to do that specifically during an upgrade or anything. I use port 443 using macvlan on my docker container but do so directly without Traefik in front. What part of the upgrade does it fail on? I was able to upload a firmware fine and it applied successfully. When you say that you're having problems uploading the firmware, do you get to the point where you can upload the firmware, it shows a uploaded and then you get a button to click to perform the upgrade or does it fail just uploading the file? If the problem is when uploading the file, could you open your browser's developer tools to see if there are any errors reported on the network tab while uploading? You could try to run the controller natively - you would either need to try to do a backup/restore via the software or you could just try to copy over the persistent data directories to where the controller expects them and that ought to work, as long as you have matching OS versions of packages, especially MongoDB; not sure about how picky it might be with Java. |
Beta Was this translation helpful? Give feedback.
-
|
Any update on where exactly it was failing? |
Beta Was this translation helpful? Give feedback.
-
|
I have the same issue. It likely has to do with the way the device tries to connect to the controller. I'm using traefik reverse proxy with a certificate from LE. The management hostname is set to a public dns record that resolves to the IP of traefik. |
Beta Was this translation helpful? Give feedback.
-
|
I haven't had a chance to play with the network really the past few weeks. Glad (is that the right word???) that someone else has experienced it as well though. I might be able to find some time between work & family being home this week to bring the wifi down to test this. |
Beta Was this translation helpful? Give feedback.
-
|
Also, do you have a link to the forum thread where you asked about this? I am curious if they can have someone take a look at the behavior of the update code to see the controller is advertising a non-routable address or a port that isn't exposed through the reverse proxy. The port documentation makes it sound like it should be using the https port for upgrades but it wouldn't surprise me if there is something untested with a reverse proxy in place. |
Beta Was this translation helpful? Give feedback.
-
|
Here's a link to the thread - https://community.tp-link.com/en/business/forum/topic/587358. Just to clarify though, I'm not too sure that the reverse proxy itself is causing the issue, since it works fine behind traefik using the "standard" management port (8043). The proxy config for using port 8043 is a bit convoluted (has to adjust some headers & redirects for the auth flow), but in the end, firmware updates still work. It was when I changed (i.e. simplified) the config to remove the extra header/redirect stuff for the login and used port 443 directly by changing the management port in the Omada UI. When I did that, I removed the original port 8043 from being exposed to the host network. I wonder, even though the port was changed in the Omada software (and was indeed accessed via 443), if some things related to the firmware update flow were still looking to communicate on port 8043 (which now had access cut off). One of my planned experiments was to keep the new config (using port 443) but keeping 8043 still exposed. As soon as I get a change, I'll let you know how it goes. |
Beta Was this translation helpful? Give feedback.
-
|
Over the holiday break I finally had some time to think about this issue further and realize the likely problem. Traefik is currently exposed on port 443 and I'm mostly using http host rules to route to the correct container. For the Omada controller, that's something like I think this can likely be closed, although if you do have any information on the actual firmware update logic (protocol, common headers, etc.) it could prove useful in trying to set up the proxy routing. |
Beta Was this translation helpful? Give feedback.
-
|
This is really confusing as to where any why it would be failing. If I am understanding correctly, you're only exposing Traefik on 443 in both scenarios, right? I wasn't sure if your mention of it working when the controller is using 8043 was there you had 8043 also exposed on Traefik as well or if it was still only using 443 but with more complex rewrite rules. I wish they would provide some more detailed information about how the upgrade process worked as it is hard to understand why it doesn't work. |
Beta Was this translation helpful? Give feedback.
-
|
I think the problem is primarily that the device itself needs to communicate back to the controller. When the controller is running on 8043 it can do so, since nothing else is using that port. However, in the scenario where the controller is running on port 443, traefik is already using that port. I imagine the packets go into traefik and die because traefik doesn't know where to send them; the only omada rules I have in place are based on the http request's hostname (e.g. send packets on 443 for host If the device uses http to communicate its update status, perhaps additional rules could be created to also route those particular packets to the omada container. The available rules for http are listed here https://doc.traefik.io/traefik/routing/routers/#rule (can use host, headers, path, etc.) If the communication is straight TCP (or UDP) the matching is more limited and might not even be possible. |
Beta Was this translation helpful? Give feedback.
-
|
So if the configuration is Traefik is listening on 443 and 8043 with only the Omada Controller on 8043, that would mean that the Omada controller should be the default server that traffic is sent to if no host match is found. Makes me curious if the device are just not passing a hostname or what exactly the devices are sending for a host header or SNI in the case of https. I suppose the quick test would be to set the Omada Controller as the default server backend to use in case of no matches and see if it succeeds then. That's the only scenario where I can think of where it would succeed with Traefik as the reverse proxy to 8043 but it would fail using 443, common with other services. |
Beta Was this translation helpful? Give feedback.
-
|
I also have hit this in my k3s cluster using Traefik ingress. I definitely remember doing upgrades to my devices sometime last year without issue, but it doesn't work any longer. I know that 5.7 added the "Device Management Hostname/IP" option, so I wonder if something changed with 5.7. |
Beta Was this translation helpful? Give feedback.
-
|
Nvm what I have written before. No idea if it worked somehow or I was just thinking it worked, I am now back to where everyone else is. tp-link must have done something really strange with one of their updates. One of my next steps will probably be to move the container to a separate host to test if it works without traefik. |
Beta Was this translation helpful? Give feedback.
-
|
So while I know this isn't going to solve the issue for everyone, there is a help article related to failures of firmware upgrades that may explain some of the behaviors that people are seeing: https://community.tp-link.com/en/business/forum/topic/559150 |
Beta Was this translation helpful? Give feedback.
-
|
Its very interesting, that only switches and routers are not capable of updating firmware. Accses Points are still getting updates from my controller behind nginx revers proxy (manager).. Very strange behaviour. |
Beta Was this translation helpful? Give feedback.
-
|
I would assume that would be because certain (newer) firmwares will use the controller's HTTPS port for updates, other older firmwares will use a specific update port as documented in the linked forum post above. |
Beta Was this translation helpful? Give feedback.
-
Yes, but every port is open.. I set mgm port in controller to 443 and on reverse proxy on 443 and I am forwarding every port from 29810-14 to container instance itself. So very annoying thats its not working anymore. Hope there will be an update in future for that. |
Beta Was this translation helpful? Give feedback.
-
|
Another data point on this problem: I have the Omada container behind Traefik and encountered the same issue with not being able to update the firmware on a switch. The following confniguration allowed me to update the firmware:
That was it. I didn't even configure Omada with a valid certificate. I simply connected to the managment interface through Traefik and the firmware upgrade now worked. |
Beta Was this translation helpful? Give feedback.
-
|
Just to add to this, if your Omada controller is sitting on two VLANs, i.e. one default and one for managing switches (and where they have their own addresses assigned from), then you need to expose 8043 on both VLANs. |
Beta Was this translation helpful? Give feedback.
-
|
I just ran into this, but didn't want to needlessly expose port 8043 on my host. I did a little bit of digging to understand what a device does when you try to do a firmware upgrade, and it's really quite simple:
This doesn't work for a number of reasons, depending on your exact setup:
There are a number of ways to address this, including the above which bypasses the reverse proxy by allowing the device to communicate with the container directly on port 8043. As mentioned, I didn't want to do that since it's unnecessary, so I did the following instead:
This works just fine. It's annoying that it's a problem at all when Unifi deals with this seamlessly, so I submitted this feature request to allow the controller to handle reverse proxies: https://community.tp-link.com/en/business/forum/topic/706282?page=1 |
Beta Was this translation helpful? Give feedback.
-
|
Had this issue as well, in my opinion it's a major flaw to handle updates like this. There's absolute no reason for the updates to go through the controller management port, and it gets very messy if you are working with gateways/switches in remote locations. Simplest solution as pointed above would be to bypass the reverse proxy but this is very much against best practices and very insecure for dealing with remote device updating as it means we need to expose the controller management port (18043) to the internet. |
Beta Was this translation helpful? Give feedback.
-
I posted on their forum about this. The client devices omit SNI. As pointed out earlier in this issue, they don't verify SSL either. You can expose the HTTPS port with a self-signed certificate and updates still work. Here's a truncated part of logs from my Caddy proxy that show the lack of SNI: caddy | {
"level": "debug",
"ts": 1750626015.6632626,
"logger": "events",
"msg": "event",
"name": "tls_get_certificate",
"id": "e38c95b7-baf3-4631-975f-55347424e935",
"origin": "tls",
"data": {
"client_hello": {
...
"ServerName": "",
...
}
}
}
caddy | {
"level": "debug",
"ts": 1750626015.6633017,
"logger": "tls.handshake",
"msg": "no certificate matching TLS ClientHello",
"remote_ip": "x.x.x.x",
"remote_port": "42224",
"server_name": "",
"remote": "x.x.x.x:42224",
"identifier": "y.y.y.y",
...
"cert_cache_fill": 0.0002,
"load_or_obtain_if_necessary": true,
"on_demand": false
}
caddy | {
"level": "debug",
"ts": 1750626015.6633554,
"logger": "http.stdlib",
"msg": "http: TLS handshake error from x.x.x.x:42224: no certificate available for 'y.y.y.y'"
}The You can also see the TLS handshake failure if you grab a I was able to make a little progress by using Caddy's default_sni global option, but updates would still fail saying the firmware downloaded from the controller was invalid. There must be some headers or something that don't get forwarded properly. I have to make it work for now, so I ended up doing what @dougmeredith suggested. I'm not thrilled with it, but it seems like the only viable option for now. |
Beta Was this translation helpful? Give feedback.
-
|
Same problem here - but I am using Cloudflare as Porxy and have no possbibility to configure a bypass for this issue. |
Beta Was this translation helpful? Give feedback.
Another data point on this problem:
I have the Omada container behind Traefik and encountered the same issue with not being able to update the firmware on a switch. The following confniguration allowed me to update the firmware:
That was it. I didn't even configure Omada with a valid certificate. I simply connected to the managment interface through Traefik and the firmware upgrade now worked.