Traffic from nginx ingress controller is not mTLS'd

[akorp]

We have an issue when traffic from the nginx ingress controller to our applications does not have mTLS in our AKS cluster.
Our linkerd setup is with annotations: nginx.ingress.kubernetes.io/service-upstream: "true" with controller: podAnnotations: linkerd.io/inject: enabled, helm stable-2.12.0.
Any help or suggestions on how to debug this issue is much appreciated.

❯ k describe deploy ingress-nginx-controller
Name:                   ingress-nginx-controller
Namespace:              system-ingress
CreationTimestamp:      Fri, 02 Sep 2022 10:57:49 +0200
Labels:                 app.kubernetes.io/component=controller
                             app.kubernetes.io/instance=ingress-nginx
                             app.kubernetes.io/managed-by=Helm
                             app.kubernetes.io/name=ingress-nginx
                             app.kubernetes.io/part-of=ingress-nginx
                             app.kubernetes.io/version=1.3.1
                             helm.sh/chart=ingress-nginx-4.2.5
Annotations:            deployment.kubernetes.io/revision: 14
                                  meta.helm.sh/release-name: ingress-nginx
                                  meta.helm.sh/release-namespace: system-ingress
Selector:               app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Replicas:               2 desired | 2 updated | 2 total | 2 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 1 max surge
Pod Template:
  Labels:           app=ingress-nginx
                    app.kubernetes.io/component=controller
                    app.kubernetes.io/instance=ingress-nginx
                    app.kubernetes.io/name=ingress-nginx
  Annotations:      config.linkerd.io/proxy-log-level: debug
                    kubectl.kubernetes.io/restartedAt: 2022-09-23T12:57:12+02:00
                    linkerd.io/inject: enabled
  Service Account:  ingress-nginx
  Containers:
   controller:
    Image:       registry.k8s.io/ingress-nginx/controller:v1.3.1@sha256:54f7fe2c6c5a9db9a0ebf1131797109bb7a4d91f56b9b362bde2abd237dd1974
    Ports:       80/TCP, 443/TCP, 10254/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Args:
      /nginx-ingress-controller
      --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      --election-id=ingress-controller-leader
      --controller-class=k8s.io/ingress-nginx
      --ingress-class=nginx
      --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      --validating-webhook=:8443
      --validating-webhook-certificate=/usr/local/certificates/cert
      --validating-webhook-key=/usr/local/certificates/key
      --default-ssl-certificate=system-ingress/wildcard-lab-int-<removed>-io-tls
    Limits:
      cpu:     500m
      memory:  1Gi
    Requests:
      cpu:      200m
      memory:   512Mi
    Liveness:   http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
    Readiness:  http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:        (v1:metadata.name)
      POD_NAMESPACE:   (v1:metadata.namespace)
      LD_PRELOAD:     /usr/local/lib/libmimalloc.so
    Mounts:
      /usr/local/certificates/ from webhook-cert (ro)
  Volumes:
   webhook-cert:
    Type:               Secret (a volume populated by a Secret)
    SecretName:         ingress-nginx-admission
    Optional:           false
  Priority Class Name:  ingress-controller
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   ingress-nginx-controller-6f7cc584dd (2/2 replicas created)
Events:          <none>

❯ k describe ing platform-test-api
Name:             platform-test-api
Labels:           app=platform-test-api
Namespace:        team-platform
Address:          10.130.117.250
Ingress Class:    nginx
Default backend:  <default>
TLS:
  SNI routes platform-test-api.lab-int.<removed>.io
Rules:
  Host                                         Path  Backends
  ----                                         ----  --------
  platform-test-api.lab-int.<removed>.io
                                               /   platform-test-api:app (10.130.117.38:8080,10.130.117.64:8080)
Annotations:                                   kubernetes.io/ingress.allow-http: false
                                               kubernetes.io/ingress.class: nginx
                                               nginx.ingress.kubernetes.io/force-ssl-redirect: true
                                               nginx.ingress.kubernetes.io/service-upstream: true
Events:                                        <none>

Traffix is secured between prometheus and application, but not between ingress-controller and application.

❯ linkerd viz edges po -n team-platform
SRC                                  DST                                  SRC_NS          DST_NS              SECURED
prometheus-58f576946-5szhg           platform-test-api-68fbcdfdc9-84cp9   linkerd-viz     team-platform       √
prometheus-58f576946-5szhg           platform-test-api-68fbcdfdc9-ltzkh   linkerd-viz     team-platform       √
platform-test-api-68fbcdfdc9-84cp9   splunk-otel-collector-agent-bnfcq    team-platform   system-monitoring   Not Provided By Service Discovery
platform-test-api-68fbcdfdc9-ltzkh   splunk-otel-collector-agent-ptbkr    team-platform   system-monitoring   Not Provided By Service Discovery

When calling endpoint getting such response from linkerd tap

❯ linkerd viz tap deploy platform-test-api -n team-platform -o json

{
  "source": {
    "ip": "10.130.117.99",
    "port": 41522,
    "metadata": {
      "control_plane_ns": "linkerd",
      "deployment": "ingress-nginx-controller",
      "namespace": "system-ingress",
      "pod": "ingress-nginx-controller-6f7cc584dd-wcn5j",
      "pod_template_hash": "6f7cc584dd",
      "serviceaccount": "ingress-nginx",
      "tls": "no_tls_from_remote"
    }
  },
  "destination": {
    "ip": "10.130.117.17",
    "port": 8080,
    "metadata": {
      "authz_group": "",
      "authz_kind": "default",
      "authz_name": "all-unauthenticated",
      "control_plane_ns": "linkerd",
      "deployment": "platform-test-api",
      "namespace": "team-platform",
      "pod": "platform-test-api-68fbcdfdc9-84cp9",
      "pod_template_hash": "68fbcdfdc9",
      "route_group": "",
      "route_kind": "default",
      "route_name": "default",
      "serviceaccount": "platform-test-api",
      "srv_group": "",
      "srv_kind": "default",
      "srv_name": "all-unauthenticated",
      "tls": "loopback"
    }
  },
  "routeMeta": null,
  "proxyDirection": "INBOUND",
  "requestInitEvent": {
    "id": {
      "base": 5,
      "stream": 7
    },
    "method": "GET",
    "scheme": "HTTP",
    "authority": "platform-test-api.lab-int.<removed>.io",
    "path": "/open/count",
    "headers": [
      {
        "name": "host",
        "valueStr": "platform-test-api.lab-int.<removed>.io"
      },
      {
        "name": "x-request-id",
        "valueStr": "fc6d45333fda2d4958b0e17a65eb85ce"
      },
      {
        "name": "x-real-ip",
        "valueStr": "10.130.117.99"
      },
      {
        "name": "x-forwarded-for",
        "valueStr": "10.130.117.99"
      },
      {
        "name": "x-forwarded-host",
        "valueStr": "platform-test-api.lab-int.<removed>.io"
      },
      {
        "name": "x-forwarded-port",
        "valueStr": "443"
      },
      {
        "name": "x-forwarded-proto",
        "valueStr": "https"
      },
      {
        "name": "x-forwarded-scheme",
        "valueStr": "https"
      },
      {
        "name": "x-scheme",
        "valueStr": "https"
      },
      {
        "name": "cache-control",
        "valueStr": "max-age=0"
      },
      {
        "name": "sec-ch-ua",
        "valueStr": "\"Google Chrome\";v=\"105\", \"Not)A;Brand\";v=\"8\", \"Chromium\";v=\"105\""
      },
      {
        "name": "sec-ch-ua-mobile",
        "valueStr": "?0"
      },
      {
        "name": "sec-ch-ua-platform",
        "valueStr": "\"macOS\""
      },
      {
        "name": "upgrade-insecure-requests",
        "valueStr": "1"
      },
      {
        "name": "user-agent",
        "valueStr": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
      },
      {
        "name": "accept",
        "valueStr": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
      },
      {
        "name": "sec-fetch-site",
        "valueStr": "none"
      },
      {
        "name": "sec-fetch-mode",
        "valueStr": "navigate"
      },
      {
        "name": "sec-fetch-user",
        "valueStr": "?1"
      },
      {
        "name": "sec-fetch-dest",
        "valueStr": "document"
      },
      {
        "name": "accept-encoding",
        "valueStr": "gzip, deflate, br"
      },
      {
        "name": "accept-language",
        "valueStr": "en-US,en;q=0.9,nb;q=0.8,es;q=0.7,ru;q=0.6,no;q=0.5,uk;q=0.4"
      },
      {
        "name": "cookie",
        "valueStr": "_ga=GA1.2.1522281047.1660040613; utag_main=v_id:018282218ab9000d7842039e659f05075003406d00c98$_sn:1$_se:3$_ss:0$_st:1660042425892$ses_id:1660040612538%3Bexp-session$_pn:1%3Bexp-session; JSESSIONID=E3C3DFBFF164FEA873A4ED34F28C9BED"
      }
    ]
  }
}
{
  "source": {
    "ip": "10.130.117.99",
    "port": 41522,
    "metadata": {
      "control_plane_ns": "linkerd",
      "deployment": "ingress-nginx-controller",
      "namespace": "system-ingress",
      "pod": "ingress-nginx-controller-6f7cc584dd-wcn5j",
      "pod_template_hash": "6f7cc584dd",
      "serviceaccount": "ingress-nginx",
      "tls": "no_tls_from_remote"
    }
  },
  "destination": {
    "ip": "10.130.117.17",
    "port": 8080,
    "metadata": {
      "authz_group": "",
      "authz_kind": "default",
      "authz_name": "all-unauthenticated",
      "control_plane_ns": "linkerd",
      "deployment": "platform-test-api",
      "namespace": "team-platform",
      "pod": "platform-test-api-68fbcdfdc9-84cp9",
      "pod_template_hash": "68fbcdfdc9",
      "route_group": "",
      "route_kind": "default",
      "route_name": "default",
      "serviceaccount": "platform-test-api",
      "srv_group": "",
      "srv_kind": "default",
      "srv_name": "all-unauthenticated",
      "tls": "loopback"
    }
  },
  "routeMeta": null,
  "proxyDirection": "INBOUND",
  "responseInitEvent": {
    "id": {
      "base": 5,
      "stream": 7
    },
    "sinceRequestInit": {
      "nanos": 3668310
    },
    "httpStatus": 200,
    "headers": [
      {
        "name": "server-timing",
        "valueStr": "traceparent;desc=\"00-07bd67d3b7aedf4741ec634141c27a14-5018b2ce039b0271-01\""
      },
      {
        "name": "access-control-expose-headers",
        "valueStr": "Server-Timing"
      },
      {
        "name": "vary",
        "valueStr": "Origin"
      },
      {
        "name": "vary",
        "valueStr": "Access-Control-Request-Method"
      },
      {
        "name": "vary",
        "valueStr": "Access-Control-Request-Headers"
      },
      {
        "name": "x-content-type-options",
        "valueStr": "nosniff"
      },
      {
        "name": "x-xss-protection",
        "valueStr": "1; mode=block"
      },
      {
        "name": "cache-control",
        "valueStr": "no-cache, no-store, max-age=0, must-revalidate"
      },
      {
        "name": "pragma",
        "valueStr": "no-cache"
      },
      {
        "name": "expires",
        "valueStr": "0"
      },
      {
        "name": "strict-transport-security",
        "valueStr": "max-age=31536000 ; includeSubDomains"
      },
      {
        "name": "x-frame-options",
        "valueStr": "DENY"
      },
      {
        "name": "content-type",
        "valueStr": "application/json"
      },
      {
        "name": "transfer-encoding",
        "valueStr": "chunked"
      },
      {
        "name": "date",
        "valueStr": "Tue, 27 Sep 2022 08:15:53 GMT"
      }
    ]
  }
}

and also logs from linkerd-proxy on ingress-conroller during a call to the app

{"timestamp":"[ 33531.996359s]","level":"DEBUG","fields":{"method":"GET","uri":"http://platform-test-api.lab-int.testgjensidige.io/open/count","version":"HTTP/1.1"},"target":"linkerd_proxy_http::client","spans":[{"name":"outbound"},{"client.addr":"10.130.117.130:48728","name":"accept"},{"addr":"10.2.0.234:8080","name":"proxy"},{"v":"1.x","name":"http"},{"name":"http1"}],"threadId":"ThreadId(1)"}
{"timestamp":"[ 33531.996387s]","level":"DEBUG","fields":{"headers":"{\"host\": \"platform-test-api.lab-int.testgjensidige.io\", \"x-request-id\": \"8bff50cb2107db7afc78c8cf7a89b45b\", \"x-real-ip\": \"10.130.117.130\", \"x-forwarded-for\": \"10.130.117.130\", \"x-forwarded-host\": \"platform-test-api.lab-int.testgjensidige.io\", \"x-forwarded-port\": \"443\", \"x-forwarded-proto\": \"https\", \"x-forwarded-scheme\": \"https\", \"x-scheme\": \"https\", \"cache-control\": \"max-age=0\", \"sec-ch-ua\": \"\\\"Google Chrome\\\";v=\\\"105\\\", \\\"Not)A;Brand\\\";v=\\\"8\\\", \\\"Chromium\\\";v=\\\"105\\\"\", \"sec-ch-ua-mobile\": \"?0\", \"sec-ch-ua-platform\": \"\\\"macOS\\\"\", \"upgrade-insecure-requests\": \"1\", \"user-agent\": \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36\", \"accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9\", \"sec-fetch-site\": \"none\", \"sec-fetch-mode\": \"navigate\", \"sec-fetch-user\": \"?1\", \"sec-fetch-dest\": \"document\", \"accept-encoding\": \"gzip, deflate, br\", \"accept-language\": \"en-US,en;q=0.9,nb;q=0.8,es;q=0.7,ru;q=0.6,no;q=0.5,uk;q=0.4\", \"cookie\": \"_ga=GA1.2.1522281047.1660040613; utag_main=v_id:018282218ab9000d7842039e659f05075003406d00c98$_sn:1$_se:3$_ss:0$_st:1660042425892$ses_id:1660040612538%3Bexp-session$_pn:1%3Bexp-session; JSESSIONID=E3C3DFBFF164FEA873A4ED34F28C9BED\"}"},"target":"linkerd_proxy_http::client","spans":[{"name":"outbound"},{"client.addr":"10.130.117.130:48728","name":"accept"},{"addr":"10.2.0.234:8080","name":"proxy"},{"v":"1.x","name":"http"},{"name":"http1"}],"threadId":"ThreadId(1)"}
{"timestamp":"[ 33531.996487s]","level":"DEBUG","fields":{"message":"reuse idle connection for (\"http\", platform-test-api.lab-int.testgjensidige.io)"},"target":"hyper::client::pool","spans":[{"name":"outbound"},{"client.addr":"10.130.117.130:48728","name":"accept"},{"addr":"10.2.0.234:8080","name":"proxy"},{"v":"1.x","name":"http"},{"name":"http1"}],"threadId":"ThreadId(1)"}
{"timestamp":"[ 33531.996652s]","level":"DEBUG","fields":{"message":"flushed 1250 bytes"},"target":"hyper::proto::h1::io","threadId":"ThreadId(1)"}
{"timestamp":"[ 33532.002462s]","level":"DEBUG","fields":{"message":"parsed 15 headers"},"target":"hyper::proto::h1::io","threadId":"ThreadId(1)"}
{"timestamp":"[ 33532.002483s]","level":"DEBUG","fields":{"message":"incoming body is chunked encoding"},"target":"hyper::proto::h1::conn","threadId":"ThreadId(1)"}
{"timestamp":"[ 33532.002494s]","level":"DEBUG","fields":{"message":"incoming chunked header: 0x3 (3 bytes)"},"target":"hyper::proto::h1::decode","threadId":"ThreadId(1)"}
{"timestamp":"[ 33532.002796s]","level":"DEBUG","fields":{"message":"flushed 595 bytes"},"target":"hyper::proto::h1::io","spans":[{"name":"outbound"},{"client.addr":"10.130.117.130:48728","name":"accept"},{"addr":"10.2.0.234:8080","name":"proxy"},{"v":"1.x","name":"http"}],"threadId":"ThreadId(1)"}

❯ linkerd check
Linkerd core checks
===================

kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-cni-plugin
------------------
√ cni plugin ConfigMap exists
√ cni plugin ClusterRole exists
√ cni plugin ClusterRoleBinding exists
√ cni plugin ServiceAccount exists
√ cni plugin DaemonSet exists
√ cni plugin pod is running on all nodes

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2022-09-29T06:57:33Z
    see https://linkerd.io/2.12/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
‼ proxy-injector cert is valid for at least 60 days
    certificate will expire on 2022-09-28T06:57:33Z
    see https://linkerd.io/2.12/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
√ sp-validator webhook has valid cert
‼ sp-validator cert is valid for at least 60 days
    certificate will expire on 2022-09-28T06:57:34Z
    see https://linkerd.io/2.12/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints
√ policy-validator webhook has valid cert
‼ policy-validator cert is valid for at least 60 days
    certificate will expire on 2022-09-28T06:57:34Z
    see https://linkerd.io/2.12/checks/#l5d-policy-validator-webhook-cert-not-expiring-soon for hints

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    is running version 2.12.0 but the latest stable version is 2.12.1
    see https://linkerd.io/2.12/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
‼ control plane is up-to-date
    is running version 2.12.0 but the latest stable version is 2.12.1
    see https://linkerd.io/2.12/checks/#l5d-version-control for hints
√ control plane and cli versions match

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-69f44dd76c-zczxl (stable-2.12.0)
	* linkerd-identity-5c58cd894-mnwv7 (stable-2.12.0)
	* linkerd-proxy-injector-566dbb58b8-zftt2 (stable-2.12.0)
    see https://linkerd.io/2.12/checks/#l5d-cp-proxy-version for hints
√ control plane proxies and cli versions match

Linkerd extensions checks
=========================

linkerd-smi
-----------
‼ Linkerd extension command linkerd-smi exists
    exec: "linkerd-smi": executable file not found in $PATH
    see https://linkerd.io/2.12/checks/#extensions for hints

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
‼ tap API server cert is valid for at least 60 days
    certificate will expire on 2022-09-28T03:04:23Z
    see https://linkerd.io/2.12/checks/#l5d-tap-cert-not-expiring-soon for hints
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-67c8dbb95d-n56fj (stable-2.12.0)
	* prometheus-58f576946-5szhg (stable-2.12.0)
	* tap-9c95cc4dd-8b6nz (stable-2.12.0)
	* tap-injector-54c9cfb879-5bs4v (stable-2.12.0)
	* web-76ffcf8447-6sklt (stable-2.12.0)
    see https://linkerd.io/2.12/checks/#l5d-viz-proxy-cp-version for hints
√ viz extension proxies and cli versions match
√ prometheus is installed and configured correctly
√ can initialize the client
√ viz extension self-check

Status check results are √