Add support for authenticated forward HTTP proxies

[ewels]

Closes #7305
Depends on seqeralabs/libseqera#82 (published as lib-httpx 2.4.0)

Problem

Behind an authenticating corporate forward proxy, Nextflow fails immediately with 407 Proxy Authentication Required on HTTPS, while git and Python tooling on the same host work fine. Two independent JVM behaviours cause this, and either one alone is enough to break proxy authentication, so both must be fixed together:

1. java.net.http.HttpClient ignores Authenticator.setDefault(...). Proxy credentials only take effect when the authenticator is set on the client builder itself. The launcher previously only set the default authenticator, so every HxClient/HttpClient-based code path (SCM providers, plugin registry, Wave, Seqera Platform, SRA) never authenticated to the proxy. Fixing this alone is still not enough for HTTPS, because of (2).

2. The JDK strips Basic credentials from the HTTPS CONNECT tunnel. jdk.http.auth.tunneling.disabledSchemes=Basic is the JDK default (see $JAVA_HOME/conf/net.properties), so even a correctly-wired authenticator is silently dropped when tunnelling HTTPS through the proxy. Fixing this alone is also not enough, because without (1) the authenticator is never consulted in the first place.

Routing was already fine: a client with no explicit proxy uses ProxySelector.getDefault(), which honours the -Dhttps.proxyHost properties set by the launcher.

Design

Per the final design of seqeralabs/libseqera#82, lib-httpx 2.4.0 exposes an explicit proxy API — HxClient.Builder.proxy(ProxySelector) / .authenticator(Authenticator) / .withProxyConfig(HxProxyConfig) — and deliberately does not read the environment itself: resolving HTTP_PROXY/HTTPS_PROXY is the consumer's job. Nextflow therefore resolves the environment in one place (HttpClientHelper in nf-commons) and wires it into every HTTP client explicitly.

Changes

• Bump io.seqera:lib-httpx 2.3.0 → 2.4.0 in nf-commons, nf-seqera, nf-wave (compileOnly + testImplementation) and nf-google, keeping the existing scopes and version-override pins in lockstep.

• Launcher: when the parsed proxy configuration carries credentials, default jdk.http.auth.tunneling.disabledSchemes and jdk.http.auth.proxying.disabledSchemes to empty — only if the user has not set them (System.getProperty(name) == null guard, never overridden). This happens in setupEnvironment(), before any HTTP client is created. The existing Authenticator.setDefault(...) is kept for legacy HttpURLConnection paths (e.g. nf-httpfs).

• New HttpClientHelper in nf-commons, resolving HTTPS_PROXY/HTTP_PROXY/ALL_PROXY/NO_PROXY (both cases, URL-encoded user:pass@host:port credentials) from SysEnv and building an explicit HxProxyConfig. ProxyConfig moved from the nextflow module to nf-commons (same package) so its URL parsing can be reused, and extended to URL-decode credentials. Two entry points:

  • applyProxy(HttpClient.Builder) — for raw JDK clients: sets toProxySelector() and, when credentials are present, toAuthenticator() on the builder. Applied in scm/RepositoryProvider.newHttpClient(), datasource/SraExplorer.getHttpClient(), nf-wave WaveClient.newHttpClient0() (used by both the authenticated and plain Wave clients), nf-seqera TowerXAuth and DataLinksResourceHandler.
  • applyProxy(HxClient.Builder) — for lib-httpx clients: applies withProxyConfig(...). Applied in plugin/HttpPluginRepository, module/RegistryClientFactory, nf-seqera TowerClient and AuthCommandImpl.

  Each site keeps its existing version/redirect/timeout/executor settings, including nf-wave's conditional virtual-threads executor. The lib-httpx proxy selector also bypasses loopback addresses (localhost, 127.*, ::1) by design.

• Docs: HTTP_PROXY/HTTPS_PROXY entries now document embedded credentials (as FTP_PROXY already did) and the disabledSchemes behaviour, with the NXF_OPTS='-Djdk.http.auth.tunneling.disabledSchemes=' fallback for users who prefer to set it themselves.

Tests

• MockAuthProxyServer test fixture (raw-socket forward proxy in the style of the one in Add authenticated forward-proxy support to HxClient seqeralabs/libseqera#82): returns 407 without Proxy-Authorization, answers plain HTTP requests with the correct Basic credentials, and tunnels CONNECT requests to the requested target port on loopback.
• HttpClientHelperTest: env resolution (both cases, ALL_PROXY fallback, URL-decoded credentials); plain HTTP 407→retry-with-credentials→200; wrong credentials fail after the 407 challenge; HTTPS CONNECT tunnelling against a local HTTPS origin with a self-signed certificate, requested via a non-loopback host name so it is only reachable through the proxy (the module test JVM runs with -Djdk.http.auth.tunneling.disabledSchemes=, mirroring what the launcher sets at runtime); NO_PROXY and loopback bypass at the selector level; no-proxy-configured leaves the client untouched. Tests drive the environment via SysEnv — no process-environment reflection.
• HttpPluginRepositoryProxyTest: a plugin registry metadata fetch against a registry host only reachable through the authenticating proxy.
• WaveClientProxyTest: the Wave HTTP transport authenticates through the proxy, resolved from the environment.
• ProxyConfigTest: URL-decoding of credentials.
• LauncherTest: the disabledSchemes properties default to empty only when the proxy has credentials and the user has not set the property; user-set values are never overridden; untouched when the proxy has no credentials.

All of the above verified locally against the published lib-httpx 2.4.0 artifact.

Compatibility

• No proxy configured → behaviour unchanged (no selector/authenticator is applied, and the disabledSchemes properties are only touched when proxy credentials are present).
• User-set jdk.http.auth.*.disabledSchemes values always win.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VV3ykikhtoSqjpG5gH4J6K

[netlify]

✅ Deploy Preview for nextflow-docs canceled.

Name	Link
🔨 Latest commit	152bbd0
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs/deploys/6a55d51ea72dcf0008ad3b95

[ewels]

Note on the CI failure: the build fails at dependency resolution because io.seqera:lib-httpx:2.4.0 is not yet published to the Seqera Maven repository — this PR depends on seqeralabs/libseqera#82. CI will stay red until that release is out; the branch will be rebased and CI re-run once it's available. No code change is needed here.

---

Generated by Claude Code

[christopher-hakkaart]

Style suggestions.

[ewels]

lib-httpx 2.4.0 is now published, so the dependency-resolution failure blocking CI is gone.

The published artifact reflects the final design of seqeralabs/libseqera#82: the environment auto-resolution was dropped during review in favour of an explicit API (HxClient.Builder.withProxyConfig(...)), with HTTP_PROXY/HTTPS_PROXY resolution left to the consumer. I've adapted the PR accordingly (c3500da):

• HttpClientHelper (nf-commons) now resolves HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY itself — reusing ProxyConfig, moved to nf-commons and extended to URL-decode credentials — and builds an explicit HxProxyConfig.
• The four sites that previously relied on the library's auto-resolution (HttpPluginRepository, RegistryClientFactory, TowerClient, AuthCommandImpl) are now wired explicitly via a new applyProxy(HxClient.Builder) overload.
• Tests drive the environment through SysEnv instead of process-environment reflection, and account for the selector's built-in loopback bypass.

Branch is merged up to current master, and all touched-module test suites pass locally against the real 2.4.0 artifact. PR description updated to match. Watching CI.

---

Generated by Claude Code

[ewels]

CI is green ✅ — all build and test matrix jobs (JDK 17 and 25) pass on head 152bbd0, now building against the published lib-httpx 2.4.0. The PR is ready for review/merge.

---

Generated by Claude Code

[pditommaso]

Hi Phil, my inclination is to close this in favour of #7331 why rely on existing helpers for proxy handling. Let me know if something is missing

[ewels]

No objection to consolidating on #7331 — but a few things it currently misses that shouldn't get lost, whichever PR moves forward:

1. jdk.http.auth.tunneling.disabledSchemes — the JDK default (Basic, see $JAVA_HOME/conf/net.properties) strips the proxy credentials from the HTTPS CONNECT request even when the authenticator is correctly wired, so HTTPS targets behind an authenticating proxy still fail with 407. That is the primary scenario in Support authenticated forward HTTP proxies across Nextflow's HTTP stack #7305 (corporate proxy, HTTPS everywhere). This PR clears the property in the launcher when the proxy configuration carries credentials (never overriding a user-set value) and documents the NXF_OPTS='-Djdk.http.auth.tunneling.disabledSchemes=' fallback. The WireMock test in Support authenticated forward HTTP/HTTPS proxies across the client stack #7331 deliberately avoids CONNECT tunnelling ("no jdk.http.auth.tunneling.disabledSchemes caveat"), so the gap isn't visible there — the end-to-end CONNECT test here (MockAuthProxyServer + local TLS origin) could be lifted over to cover it.
2. ALL_PROXY fallback and URL-decoding of user:pass in the proxy URL (credentials with special characters) — both handled here, with the env-vars.mdx docs already reviewed by @christopher-hakkaart.
3. Two more raw-HttpClient sites in nf-seqera — TowerXAuth and fs/handler/DataLinksResourceHandler build plain JDK clients and need the proxy selector + authenticator as well. Support authenticated forward HTTP/HTTPS proxies across the client stack #7331 currently patches the pre-refactor nf-tower module and shows merge conflicts against current master, so these sites would be missed after rebasing.

Conversely, one thing #7331 does better which this PR should adopt if kept: configuring HxClient.Builder directly instead of passing a pre-built client via .httpClient(...), so the internal token-refresh clients inherit the proxy too (here WaveClient's refresh client would currently miss it).

Happy either way — just flagging the pieces above so they land somewhere.

---

Generated by Claude Code

[pditommaso]

Thanks @ewels — all three have now landed on #7331 (branch merged up to current master, one trivial build.gradle conflict). Status per your list:

1. jdk.http.auth.tunneling.disabledSchemes — done. Launcher.setProxy now clears the property when the proxy carries credentials, and never overrides a value the user set (e.g. via NXF_OPTS). This is the actual Support authenticated forward HTTP proxies across Nextflow's HTTP stack #7305 path (HTTPS + authenticating proxy → 407 on CONNECT).

2. ALL_PROXY fallback + URL-decoding of user:pass — done. ALL_PROXY/all_proxy is used as a fallback when no scheme-specific variable is set (scheme-specific still wins); ProxyConfig.parse now percent-decodes the userinfo (literal + preserved).

3. TowerXAuth and DataLinksResourceHandler — done. DataLinksResourceHandler was converted from a plain JDK HttpClient to HxClient (no bearer token, so the signed-URL fetch still doesn't carry the Seqera Authorization header) with withProxyConfig; TowerXAuth keeps its plain JDK HttpClient (cookie-based refresh flow) and gets the proxy selector + proxy authenticator wired explicitly.

   One correction on the framing: the nf-tower→nf-seqera refactor you mentioned isn't in master — those files still live in plugins/nf-tower, exactly where Support authenticated forward HTTP/HTTPS proxies across the client stack #7331 patches them, so the merge was clean (no rebase hazard). TowerClient/AuthCommandImpl were already wired; these two were simply the remaining raw-client sites.

Also confirmed the "conversely" point is already the case on #7331 — clients are configured on HxClient.Builder directly (not via .httpClient(...)), so the token-refresh clients inherit the proxy.

Each item is a separate commit with unit tests (RED→GREEN). Still open, if worth adding: (a) the doc note for ALL_PROXY / the NXF_OPTS disabledSchemes fallback in env-vars.mdx, and (b) lifting your end-to-end CONNECT-tunnelling test over — the current WireMock test deliberately covers only plain-HTTP, so #1 isn't exercised end-to-end yet. Happy to port both if you'd like.