Split DNS issue? Can't connect to local reverse proxy at Site2 using same domain name as Site1.

[partytimeexcellent]

My main Pangolin instance is on a VPS with public DNS records pointing to it, accessed via pangolin.uniquedomain.com

There are two Sites at two different physical locations, and both use a local-only Pangolin instance as a reverse proxy.

Location one Pangolin instance is accessed at https://proxy.uniquedomain.com and all traffic from the main VPS instance is sent to this URL. This works perfectly, and there are many resources all with names like service1.uniquedomain.com that resolve perfectly fine with this setup.

At location two, I have the exact same setup, except the URLs all have an additional subdomain (.site2), https://proxy.site2.uniquedomain.com, service1.site2.uniquedomain.com, etc, (I have also tried proxy-site2.uniquedomain.com). Unfortunately the main Pangolin VPS instance is not properly connecting to this local Site2 Pangolin reverse proxy. All requests just time out. (and also cause RAM usage to spike). For resources that are protected by SSO Auth on the VPS instance, redirecting to the sign-in page works, but then after successful login, it goes back to timing out.

The weird thing is, there doesn't seem to be a problem using uniquedomain.com at both sites because I can create resources that skip the local Pangolin reverse proxy and point directly to the service, such as service1-site2.uniquedomain.com while concurrently sending traffic to Site1 just fine. So I'm confused about what possible DNS issue might be happening

I have tried numerous combinations of entries into the "TLS Server Name" and "Custom Host Header" fields without luck. (Site1 works fine with no entries in these fields.). I tried this because I have a third site with a different domain name, and I needed to use TLS Server Name: uniquedomain2.com, and Custom Host Header: proxy.uniquedomain2.com to get it working, although that was presenting a 502 error without them, not timing out.

There is no issue with the Site2 Pangolin reverse proxy if I open local ports and change the public DNS to point directly to it instead of the VPS, everything behaves normally. Private resources also work perfectly fine at both Sites.

Trying to make sense of my limited knowledge in the Traefik logs:
302 error on "/" is often the first thing to come up when I try to access the URL. Followed by 504 or 499.
Sometimes I see 302 on /?resource_session_request_param=pumreoqwbugyh7d..... after an auth login.

So, I've exhausted my troubleshooting on this setup. Anyone have ideas why I can't connect to this Site2 Pangolin reverse proxy instance? Is there any type of logging that could show why It's timing out?

Thanks for any insights!

[partytimeexcellent]

Ok here's an update that should help track down the issue:

In the main VPS Pangolin instance, if I change the resource targets at Site2 from FQDN to IP (https://service1.site2.uniquedomain.com:443 to https://10.0.0.100:443), I can access the resources just fine.

So the question is, why can VPS Pangolin find Site1 Pangolin at uniquedomain.com, but can't find Site2 at site2.uniquedomain.com

This must be a DNS issue (it's always DNS, lol), but I am lost on how to find or change anything related to this. Thanks for any pointers

[kapuett]

Could you check in network tab of your browser if behaviour is the same as in https://github.com/orgs/fosrl/discussions/2255? I am not so sure it really is a DNS issue. If your behaviour is the same as in https://github.com/orgs/fosrl/discussions/2255 I rather think it's a wrong redirect somewhere. Of course that's not happening if you route IP based.
One remark of your setup in general: if I got it right, you use three pangolin instances? I get that it's more convenient that way but for your sites that's a lot of overhead. You could just use one traefik instance each instaed.