appears to mishandle failing HTTP2 upgrade / doesn't send Authorization header #1821
Description
Confirm by changing [ ] to [x] below to ensure that it's a bug:
- I've gone though Developer Guide and API reference
- I've searched for previous similar issues and didn't find any solution
Describe the bug
To be honest, I'm not completely sure where the bug is/who is at fault (original issue: NixOS/nix#5707).
I try to use Minio as an S3 backend for a nix cache, nix uses aws-sdk-cpp for uploading to S3 stores, minio currently doesn't fully support HTTP/2 and some reverse-proxies like lighttpd, which I also use, don't handle the upgrade implicitly. thus, the upgrade request to h2 is ignored, which then results in an authentication error from the backend, because (and that's what probably changed) aws-sdk-cpp stopped supplying the Authorization: header in the initial requests which wants to upgrade to h2. It then doesn't seem to detect that the upgrade failed and thus doesn't retry via plain HTTP/1.1 + Authorization.
SDK version number
1.8.121
Platform/OS/Hardware/Device
NixOS (Linux)
To Reproduce (observed behavior)
I don't have any reduced snippet yet, but the related code is here: https://github.com/NixOS/nix/blob/master/src/libstore/s3-binary-cache-store.cc, https://github.com/NixOS/nix/blob/master/src/libstore/s3.hh
Expected behavior
When a connection/request can't be upgraded to h2 (or alternatively h2c), then IF the request failed (because of missing authentication, which doesn't always happen, e.g. if the IAM policy is "wide" enough to allow completely unauthenticated requests to the endpoint+bucket+method combination) the request should be re-sent via HTTP/1.1 (without Upgrade: h2(c) and with(!) Authorization: ... instead).
Logs/output
See issue above. I haven't yet looked deep enough into the aws-sdk-cpp API to reproduce it in a standalone example, and nix doesn't offer enough interesting output per default. In the original issue, I posted some server-side HTTP traces, which hopefully illustrate some part of the problem.
To enable logging, set the following system properties:
REMEMBER TO SANITIZE YOUR PERSONAL INFO
options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
Aws::InitAPI(options)
(maybe I do that later, but it would take some time to recompile nix itself)
Additional context
see above and linked issue.