Active health checks fail when using upstream TLS with client certificate

[tsaarni]

What steps did you take and what happened:

Prerequisites:

• Contour uses a client certificate for Envoy upstream TLS.
• HTTPProxy uses upstream TLS
• HTTPProxy uses Envoy's active health checks.

All of these must be true for the bug to happen. If removing any of them, the problem goes away.

The Problem:

The upstream service is unavailable for certain period of time:

• Scenario 1: The client gets a 503 Service Unavailable "no healthy upstream" error for 4 minutes after Envoy restarts
  • This is counted from the moment Envoy is back up and ready with all configuration from Contour. It doesn't include the time Envoy takes to be ready serve requests.
  • The downtime lasts for healthyThresholdCount * no_traffic_interval (for example: 4 * 60s = 4min).
  • The no_traffic_interval defaults to 60 seconds in Envoy and is not configurable in Contour.
• Scenario 2: The client gets a 503 Service Unavailable "no healthy upstream" error for several seconds after rotating the Envoy client certificate.
  • The downtime lasts for healthyThresholdCount * intervalSeconds seconds (for example: 4 * 5s = 20s).

What did you expect to happen:

There should be no service interruption.

Anything else you would like to add:

I've added the steps to reproduce this in the comments below.

Environment:

• Contour version:
• Kubernetes version: (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

[tsaarni]

How to reproduce

Set up the test environment

Following steps deploy and configure Contour and the backend service for being able to reproduce the problem.

# 1. Install Contour
#  
# The versions are currently following, but the problem can be reproduced with older versions as well
#   - Contour v1.33.1
#   - Envoy to v1.35.8
kubectl apply -f https://projectcontour.io/quickstart/contour.yaml


# 2. Create configuration file for Contour to enable upstream TLS with client certificate.
#   - Client certificate is stored in Secret "envoy-client-cert" in namespace "projectcontour".
kubectl -n projectcontour apply -f - << 'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: contour
  namespace: projectcontour
data:
  contour.yaml: |
    tls:
      envoy-client-certificate:
        name: envoy-client-cert
        namespace: projectcontour
EOF

# 3. Restart Contour to pick up new configuration.
kubectl -n projectcontour delete pod -lapp=contour --force


# 4. Create certificates for upstream TLS (https://github.com/tsaarni/certyaml/)
#
# With following PKI structure:
#
#   internal-root-ca
#     |
#     +-- envoy (client certificate for Envoy)
#     +-- echoserver (server certificate for upstream service "echoserver")
#

cat > certs.yaml << 'EOF'
subject: cn=internal-root-ca
---
subject: cn=envoy
issuer: cn=internal-root-ca
sans:
- DNS:envoy
---
subject: cn=echoserver
issuer: cn=internal-root-ca
sans:
- DNS:echoserver
EOF

mkdir -p certs
certyaml --destination certs certs.yaml


# 5. Upload certificates and keys to Kubernetes.
kubectl create secret generic internal-root-ca --from-file=ca.crt=certs/internal-root-ca.pem --dry-run=client -o yaml | kubectl apply -f -
kubectl create secret tls echoserver-cert --cert=certs/echoserver.pem --key=certs/echoserver-key.pem --dry-run=client -o yaml | kubectl apply -f -
kubectl -n projectcontour create secret tls envoy-client-cert --cert=certs/envoy.pem --key=certs/envoy-key.pem --dry-run=client -o yaml | kubectl apply -f -


# 6. Create upstream service "echoserver" (https://github.com/tsaarni/echoserver/)
#   - Deployment and Service
#   - HTTPProxy with
#       - TLS enabled towards upstream service, using client certificate authentication.
#       - Active health checks enabled.
kubectl apply -f - << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  selector:
    matchLabels:
      app: echoserver
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: ghcr.io/tsaarni/echoserver:latest
        env:
        - name: ENV_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: ENV_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: ENV_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: ENV_NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: ENV_POD_UID
          valueFrom:
            fieldRef:
              fieldPath: metadata.uid
        - name: TLS_CERT_FILE
          value: /certs/tls.crt
        - name: TLS_KEY_FILE
          value: /certs/tls.key
        - name: SSLKEYLOGFILE
          value: /tmp/wireshark-keys.log
        ports:
        - name: http-api
          containerPort: 8080
        - name: https-api
          containerPort: 8443
        volumeMounts:
        - name: echoserver-cert
          mountPath: /certs
          readOnly: true
        - name: wireshark-keys
          mountPath: /tmp/
      volumes:
      - name: echoserver-cert
        secret:
          secretName: echoserver-cert
          optional: true
      - name: wireshark-keys
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
spec:
  ports:
  - name: http
    port: 80
    targetPort: http-api
  - name: https
    port: 443
    targetPort: https-api
  selector:
    app: echoserver
---
apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: echoserver
spec:
  virtualhost:
    fqdn: echoserver.127-0-0-101.nip.io
  routes:
    - services:
        - name: echoserver
          port: 443
          protocol: tls
          validation:
            subjectName: echoserver
            caSecret: internal-root-ca
      healthCheckPolicy:
        path: /status
        healthyThresholdCount: 5
        intervalSeconds: 5
        timeoutSeconds: 2
        unhealthyThresholdCount: 3
EOF


# 7. Wait until echoserver starts responding successfully.
curl http://echoserver.127-0-0-101.nip.io

Scenario 1: 4 minutes of unavailability after Envoy restart

Follow these steps and the client receives a 503 "no healthy upstream" error for 4 minutes after Envoy is READY. The service only starts working again after those 4 minutes.

# 8. Restart Envoy.
kubectl -n projectcontour delete pod -lapp=envoy --force


# 9. Wait for Envoy to come up again
#   - pod status becomes READY 2/2
#   - client can connect again without "Connection refused" 
kubectl -n projectcontour get pods -lapp=envoy --watch


# 10. Send HTTP request to echoserver to observe that echoserver becomes unavailable.
#
# It will respond with:
#
#  HTTP/1.1 503 Service Unavailable
#  content-length: 19
#  content-type: text/plain
#  date: Thu, 22 Jan 2026 11:52:09 GMT
#  server: envoy
#
#  no healthy upstream
#

curl -i http://echoserver.127-0-0-101.nip.io


# 11. Poll until HTTP success response is received from Envoy.
#
#   - After Envoy becames ready, it will continue to return 503 for ~4 minutes.
#
until curl -f http://echoserver.127-0-0-101.nip.io; do sleep 1; done; date

Restarting Contour doesn't help. It still takes about 4 minutes to work.

# 12. Restart Envoy again to start 4 minutes unavailability from the beginning.
kubectl -n projectcontour delete pod -lapp=envoy --force
kubectl -n projectcontour get pods -lapp=envoy --watch       # Wait until READY becomes 2/2.

curl -i http://echoserver.127-0-0-101.nip.io                 # Now returns 503.

kubectl -n projectcontour delete pod -lapp=contour --force   # Restart Contour.

curl -i http://echoserver.127-0-0-101.nip.io                 # Still returns 503 for ~4 minutes.

Scenario 2: 20 seconds unavailability after Envoy client certificate rotation

Follow these steps to update the Kubernetes Secret with new Envoy client certificate+key which causes downtime:

• The downtime starts ~6 seconds after Secret is updated.
• Downtime lasts for 20 seconds (healthyThresholdCount * intervalSeconds).

# 13. Re-crete Envoy client certificate to simulate certificate rotation.
rm certs/envoy.pem certs/envoy-key.pem
certyaml --destination certs certs.yaml


# 14. Upload new Envoy client certificate to Kubernetes.
kubectl -n projectcontour create secret tls envoy-client-cert --cert=certs/envoy.pem --key=certs/envoy-key.pem --dry-run=client -o yaml | kubectl -n projectcontour apply -f -


# 15. Poll Envoy to see how the responses change during certificate rotation.
#   - It will first respond with 200, then 503 and then 200 again

while true; do curl -I http://echoserver.127-0-0-101.nip.io; sleep 1; done

[tsaarni]

xref envoyproxy/envoy#43116