Linkerd and TCP connections not working.

[alifiroozi80]

Hello folks
I have multiple microservices.
They all communicate with TCP, and some of them also have HTTP.
I've injected the Linkerd into them, and now nothing is working!
I've searched and realized I should add the config.linkerd.io/opaque-ports annotations to Deployments and Services.
Added and did not work again.
Here is an example (Other Microservices are like this user example):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: users
  name: users
  namespace: staging
spec:
  replicas: 1
  selector:
    matchLabels:
      app: users
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/opaque-ports: "3002"
      labels:
        app: users
      name: users-pod
    spec:
      containers:
      - image: MY-REPO/users:f2e2c
        name: users-container
        ports:
        - containerPort: 3002
          protocol: TCP

apiVersion: v1
kind: Service
metadata:
  annotations:
    config.linkerd.io/opaque-ports: "3002"
    ###
    # I have also tried the svc port: `config.linkerd.io/opaque-ports: "5000"`, didn't work either
   ###
  name: users-service
  namespace: staging
spec:
  ports:
  - name: ws
    port: 5000
    protocol: TCP
    targetPort: 3002
  selector:
    app: users
  type: ClusterIP

Logs seemed ok:

✗ k -n staging logs users-c7d55c786-x6b7l
Defaulted container "linkerd-proxy" out of: linkerd-proxy, users-container, linkerd-init (init)
[     0.009122s]  INFO ThreadId(01) linkerd2_proxy: release 2.210.4 (5a910be) by linkerd on 2023-11-22T17:01:46Z
[     0.012860s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.015042s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.015060s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.015064s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.015067s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.015070s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.staging.serviceaccount.identity.linkerd.cluster.local
[     0.015073s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.015077s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.039768s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=default.staging.serviceaccount.identity.linkerd.cluster.local

Here is API microservice (all the ms talk to this)

Defaulted container "linkerd-proxy" out of: linkerd-proxy, api-container, linkerd-init (init)
[     0.003941s]  INFO ThreadId(01) linkerd2_proxy: release 2.210.4 (5a910be) by linkerd on 2023-11-22T17:01:46Z
[     0.005105s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.006225s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.006323s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.006327s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.006329s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.006331s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.staging.serviceaccount.identity.linkerd.cluster.local
[     0.006334s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.006337s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.021013s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=default.staging.serviceaccount.identity.linkerd.cluster.local
[   154.882829s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.0.156.169:58522 server.addr=10.0.42.81:8080
[   164.033735s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.0.139.67:17888 server.addr=10.0.42.81:8080
[   164.060806s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=connection closed before message completed client.addr=10.0.139.67:14878 server.addr=10.0.42.81:8080

I've also added all the TCP ports to all deployments:

config.linkerd.io/opaque-ports: 8001,3001,3003,3005,3004,3002,3007

didn't help.

For the Connection closed error=connection closed before message completed added the following:

config.linkerd.io/proxy-inbound-connect-timeout: "20"
 config.linkerd.io/proxy-outbound-connect-timeout: 20s

Didnt help neither.

---

Apps, cannot talk to gather (Timeout), but as soon as I disabled Linkerd, everything returned to being OK.

[alifiroozi80]

Ok, I found the trick.
I had to:

1. Add config.linkerd.io/opaque-ports: 5000 to my services.
2. Add config.linkerd.io/opaque-ports: 8001,3001,3003,3005,3004,3002,3007 to all of my deployments (even those that are HTTP)
3. No need to change the timeout settings.