Feature Request: identify traffic from duckdb-wasm via request headers

[carlopi]

Discussed in #2109

^{Originally posted by jdangerx September 26, 2025}
We host some Parquet files in S3 that people access both through a browser-based tool built on duckdb-wasm as well as through direct link downloads and their own scripts.

We can get S3 access logs which show request headers like User-Agent, but all the browser-based requests are mixed together under similar User-Agents. It would be nice to identify how much traffic is coming from the tool vs. direct downloads!

I saw #1967, which is kind of related in that the user is trying to add custom headers (for authentication, not analytics purposes). If we could add extra headers like that, it would solve my problem... however it seems like that extra-header capability is part of the HTTPFS extension which is not supported in duckdb-wasm since duckdb-wasm is trapped inside the browser's HTTP stack.

We can certainly work around this issue - it's not critical - but it would be nice to be able to identify duckdb-wasm based traffic from S3 access logs, whether it's through setting custom headers, sending a new DuckDB-User-Agent header (as suggested in Discord), or some other creative solution!

[carlopi]

There are a few different issues:

1. by default duckdb-wasm has a custom (non httpfs-based) HTTP-stack
   This is the simplest, since the default can be changed simply via : SET builtin_httpfs = false; LOAD httpfs;
   We will also change the default in a future duckdb-wasm release

2. there is currently an error where extra_headers are not properly implemented at HTTPWasmClient -level, that I would handle in the next days
   Here there are no workaround user side, I will update in this issue when fixed

3. DuckDB-Wasm in a browser environment is acting behind the browser that acts as men-in-the middle, validating and possibly removing some headers, so that even if you can provide via SQL the EXTRA_HTTP_HEADERS some of those will NOT be visible from the outside
   Here probably some validation rules should be added so creating a secret with invalid header is notified back to user. In particular I am certain that User-Agent is overridden by the browser, so an alternative header should be identified for this purpose. I plan on implementing 2 and then providing instructions on how to test this.

[zaneselvans]

Thanks for looking into this!

[carlopi]

Update: I got this working, like:

SET builtin_httpfs = false; LOAD httpfs;
CREATE SECRET http (
       TYPE HTTP,
       EXTRA_HTTP_HEADERS MAP {
         'DuckDB-Custom-User-Agent' : 'quack'
        }
);

and then https traffic gets the extra header. Obv map is arbitrary.

NOTE: adding extra non-default headers might upgrade from Simple HTTP request to requests gate-keeped via CORS (might upgrade = if it was not CORS already)

I will update when PR is merged.

[jdangerx]

That's great news, @carlopi ! Thanks for getting a fix working so quickly, do you have any updates on the PR / is there anything we can do to help test things out?