Intermittent 400 Bad request with ssl-passthrough and mTLS (default certificate used) #13930
Description
When using ssl-passthrough with nginx.ingress.kubernetes.io/ssl-passthrough: "true" and a backend that requires mTLS, requests sometimes succeed but sometimes fail with:
<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.25.5</center>
</body>
</html>
We are using the version v1.12.1 of nginx controller
With curl -vk, when it fails, we show that the fake certificate (or default cert) is used during the TLS handshake.
allocate connect buffer!
Establish HTTP proxy tunnel to nginx-test.dev10.toto:443
> CONNECT nginx-test.dev10.toto:443 HTTP/1.1
> Host: nginx-test.dev10.toto:443
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
< Connection: close
<
Proxy replied 200 to CONNECT request
CONNECT phase completed!
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
CONNECT phase completed!
CONNECT phase completed!
{ [5 bytes data]
TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
TLSv1.3 (IN), TLS handshake, Unknown (8):
{ [19 bytes data]
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [897 bytes data]
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
TLSv1.3 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
} [1 bytes data]
TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
ALPN, server accepted to use h2
Server certificate:
subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
start date: Sep 11 14:04:10 2025 GMT
expire date: Sep 11 14:04:10 2026 GMT
issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
Using Stream ID: 1 (easy handle 0x56121ba56620)
} [5 bytes data]
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
> GET / HTTP/2
> Host: nginx-test.dev10.toto
> User-Agent: curl/7.58.0
> Accept: */*
>
{ [5 bytes data]
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
{ [1 bytes data]
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
TLSv1.3 (OUT), TLS Unknown, Unknown (23):
} [1 bytes data]
TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
< HTTP/2 400
< date: Thu, 11 Sep 2025 14:49:53 GMT
< content-type: text/html
< content-length: 237
< strict-transport-security: max-age=31536000; includeSubDomains
<
{ [237 bytes data]
TLSv1.3 (IN), TLS Unknown, Unknown (23):
{ [1 bytes data]
100 237 100 237 0 0 2393 0 --:--:-- --:--:-- --:--:-- 2393
Connection #0 to host 127.0.0.1 left intact
<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx/1.25.5</center>
</body>
</html>
backend logs:
│ 2025/09/11 14:32:47 [info] 35#35: *2056 client sent no required SSL certificate while reading client request headers, client: 172.20.76.224, server: nginx-test.dev10.toto, request: "GET / HT │
│ TP/1.1", host: "nginx-test.dev10.toto"
when it works, the correct backend certificate is presented and the mTLS handshake succeeds.
controller args:
containers:
- args:
- /nginx-ingress-controller
- --enable-annotation-validation=true
- --default-backend-service=$(POD_NAMESPACE)/ingress-nginx-defaultbackend
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --election-id=ingress-nginx-leader
- --controller-class=k8s.io/ingress-nginx
- --ingress-class=nginx
- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
- --validating-webhook=:8443
- --validating-webhook-certificate=/usr/local/certificates/cert
- --validating-webhook-key=/usr/local/certificates/key
- --watch-ingress-without-class=true
- --enable-metrics=true
- --enable-topology-aware-routing=true
- --default-ssl-certificate=$(POD_NAMESPACE)/tls-secret
- --disable-full-test=true
- --enable-ssl-passthrough=true
- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --v=5
controller configmaps:
data:
allow-cross-namespace-resources: "true"
client-header-buffer-size: 16k
compute-full-forwarded-for: "true"
disable-ipv6: "true"
disable-ipv6-dns: "true"
enable-underscores-in-headers: "true"
forwarded-for-header: proxy_protocol
large-client-header-buffers: 8 16k
proxy-add-original-uri-header: "true"
use-forwarded-headers: "true"
use-proxy-protocol: "true"
ingress annotations:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: HTTPS
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-passthrough: "true"