Application Firewall can block DoH without User-Agent header

[Shamar]

Some hosting providers employ application firewalls to protect their customers websites.
When the provider is particularly paranoiac, it might block or redirect requests with an empty or missing User-Agent header.

I see it was removed in #14 and for sure it looks like a reasonable default, but I think that there should be an option to enable a predefined UA or even set a UA string, possibly through a dedicated input in the RethinkDNS app.

As an alternative, I suggest to revert #14 because the privacy gain is minimal (your IP is already available to the DoH server and the fact that you are using RethinkDNS instead of, say, Firefox, isn't a valuable data leak to the DoH service provider, and the connection to the DoH is encrypted anyway so no one else can learn that info).

[Shamar]

Let's mention @PeterDaveHello who might have a different perspective on the topic.

[PeterDaveHello]

While I understand that removing the User-Agent (UA) string offers minimal privacy gains, it is still part of user behavior profiling and fingerprinting. While this may not be a complete solution, it does reduce one identifiable data point used in tracking.

Furthermore, iOS and Firefox both use an empty UA string. If firewalls are having trouble with this, it could affect a broader range of users than expected. I’m curious about the specifics of the problem. Are there particular websites or firewalls that are causing issues with an empty UA string?

[Shamar]

I faced this issue with OVH's application firewall: a nginx that was set between the browser and the apache webserver running my doh.cgi.

When it was enabled, I started receiving 302 responses that broke all the DoH clients. Apparently, they fixed the issue after a long chat with the tech support, but I can't say if they fixed it for everybody or just for my own services.

iOS and Firefox both use an empty UA string.

Yes, and this is why there is a minimal privacy gain, but such header is only visible to the DoH server that you are trusting anyway with your IP and navigation that are much more valuable than the DoH client you are using.

Also Firefox let you set network.trr.send_user-agent_headers in about:config to fix similar issues, and I think a good compromise would be to provide a similar option in the RethinkDNS UI.

[ignoramous]

I think a good compromise would be to provide a similar option in the RethinkDNS UI

It is a terrible compromise, as it adds another knob in an already complicated UI. It is what it is...

Instead of letting the user configure UA, what about setting it to "intra" (if enabled via UI). Thoughts?

[Shamar]

To be honest, @ignoramous, I like RethinkDNS interface a lot: given the empowerment it grants to users, I would not define its UI "complicated" at all.

I think that a input in the Advanced section of DNS configuration would not add any real cognitive overhead to the users, but it will enable people who face this issue to fix it without reducing in any way the default privacy provided to all other users.

As I said, OVH solved the issue (at least for me) so I have no pressure with this.
I reported the issue just because it took several days to identify the underlying issue and a couple of hours to understand enough of RethinkDNS codebase to locate the "offending" line.

So, to me, setting it to "intra" is a good solution for people who face this specific issue, without causing much harms to people who don't. Yet I'd make the header inclusion opt-in, with a new toggle in the previously mentioned UI.

[Lanius-collaris]

@Shamar
Did you use OVH's Web Cloud? Why not deploy a DoH server on normal VPS?

[Shamar]

Yes, I used a shared hosting for dogfooding my doh.cgi.

The idea behind that little software is that a shared hosting providing CGI support is way cheaper than a VPS and it can leverage the security of a professional hosting provider.

It's all explained in the POLITICS.txt file.

Simply put, the easier and cheaper it is to have your own privacy preserving DoH resolver, the more people will be able to set it up, distributing the DNS resolution and reducing overall surveillance.

Using a VPS would not allow me to identify systematic problems with this solution (which indeed occurred with the 302s on the User-Agent).

[ignoramous]

I think that a input in the Advanced section of DNS configuration would not add any real cognitive overhead to the users,

Well, the thing is Rethink supports DNSCrypt, DoT, DNS53 and not just DoH. This user-agent stuff is valid for DoH only and yet needs its own global setting.

What if, instead of setting empty str, no UA header is set at all? Will that work for these WAFs (which otherwise issue 302s)?

[Shamar]

Well, the thing is Rethink supports DNSCrypt, DoT, DNS53 and not just DoH. This user-agent stuff is valid for DoH only and yet needs its own global setting.

Well, you could just add the toggle/input into the configuration UI of the Resolver URI in DoH (and maybe ODoH).

I mean, it's not necessarily a global setting: it would be much more correct to configure it on a per-resolver basis as different DoH might behave differently on this topic.

What if, instead of setting empty str, no UA header is set at all? Will that work for these WAFs (which otherwise issue 302s)?

When I tried, OVH's waf returned a 302 on both missing UA and empty UA.

[ignoramous]

When I tried, OVH's waf returned a 302 on both missing UA and empty UA.

Thanks for spending time to test this.

mean, it's not necessarily a global setting: it would be much more correct to configure it on a per-resolver basis

Per resolver means it'd have to go in to the db against each DoH entry... I'll think about it. I was thinking global because it is simpler. Though, per DoH entry also makes ample sense.

[ignoramous]

In v055o+, turning ON Configure -> Network -> Endpoint Independent Mapping will also set user agent on DoH and ODoH requests:

firestack/intra/doh/doh.go

Lines 749 to 752 in b68ccab

	if settings.SetUserAgent.Load() {
	req.Header.Set("user-agent", settings.IntraUa)
	}
	return

Custom DoH headers + user agents per DoH transport is non trivial amount of work, and so, we'll work on it when other pressing bugs and feature requests have been addressed.