Postgresql-ha with linkerd

[pepsi1k]

Hello!

I want to deploy postgresql with 3 replicas and configure mtls between them.

1. At the beginning I deployed postgres without additional linkerd's annotations, only injector: enabled. In this case postgres replication stuck in ready status 1/2.

kubectl get pods:

postgres-postgresql-ha-pgpool-6c76c6f754-br8cg   2/2     Running     0          34m
postgres-postgresql-ha-postgresql-0              2/2     Running     0          34m
postgres-postgresql-ha-postgresql-1              1/2     Running     10         34m
postgres-postgresql-ha-postgresql-2              1/2     Running     10         34m

logs of postgresql-0:

INFO ThreadId(01) outbound:accept{peer.addr=10.12.50.218:39686 target.addr=10.12.70.171:5432}: linkerd2_app_core::serve: Connection closed error=Broken pipe (os error 32)

Readiness probe filed in postgresql-{1..2}:

kubelet  Liveness probe failed: psql: could not connect to server: Connection refused
           Is the server running on host "127.0.0.1" and accepting
           TCP/IP connections on port 5432?

When I check access to postgres via netcat it throws an error:

wait-for-it.sh: waiting 15 seconds for postgres-postgresql-ha-pgpool.postgres.svc.cluster.local:5432
wait-for-it.sh: timeout occurred after waiting 15 seconds for postgres-postgresql-ha-pgpool.postgres.svc.cluster.local:5432

But if I run my pod with psql-cli, without checking tcp connection, I can connect to postgres.

bash-5.0# psql -U "postgres" -d "postgres" -h postgres-postgresql-ha-pgpool.postgres.svc.cluster.local -p 5432
psql (12.5, server 11.9)
postgres=#

After a random number of restarts, in my case (99/201) the replicas go up 2/2 ready status, but errors in linkerd-proxy remains
At the beginning I thought the problem was with linkerd and tcp, but I deployed redis (which also uses tcp) with 3 replicas and they work fine

2. The second case is when postgres is deployed with the skip-outbound-ports and skip-inbound-port pod annotations. Everything is developing perfectly, but I'm losing mtls!.

I have used this chart:

Name	chart version	app version
bitnami/postgresql-ha	6.3.5	11.10.0
linkerd/linkerd2	2.9.1	stable-2.9.1
linkerd/linkerd2-cni	2.9.1	stable-2.9.1

Any Ideas?

[adleong]

Hey @pepsi1k! The latest edge release edge-21.1.1 introduces a new "opaque transport" feature that allows the proxy to securely transport server-speaks-first and otherwise opaque TCP
traffic. You can use the config.linkerd.io/opaque-ports annotation on pods and namespaces to configure ports that should skip the proxy's protocol detection.

[pepsi1k]

Hi @adleong! Thanks for the answer, but it was not help me.
The steps i took:

1. I installed the edge-21.1.1 and add config.linkerd.io/opaque-ports: '5432' to pods, but after run postgres my replicas still in ready status 1/2
2. I restarted postgresql and all the replicas up with 2/2 ready status. Everything worked fine, I could check the tcp connection, the error in linkerd-proxy was gone. The only thing was the readinessProbe error, but that didn't worry me.
3. After that two steps, I restarted postgresql again to make sure everything was working, but it was stuck in status 1/2 again

logs postgres-1 postgresql:

Warning  Unhealthy 3m3s (x88 over 17m)  kubelet Readiness probe failed: psql: could not connect to server: Connection refused
  Is the server running on host "127.0.0.1" and accepting
  TCP/IP connections on port 5432?

logs postgres-1 linkerd-proxy:

time="2021-01-14T10:06:25Z" level=info msg="running version edge-21.1.1"
[     0.002305s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.002806s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.002820s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.002824s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.002828s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.002831s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.postgres.serviceaccount.identity.linkerd.cluster.local
[     0.002835s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.002838s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003034s]  INFO ThreadId(01) outbound: linkerd2_app: listen.addr=127.0.0.1:4140 ingress_mode=false
[     0.003131s]  INFO ThreadId(01) inbound: linkerd2_app: listen.addr=0.0.0.0:4143
[     0.015832s]  INFO ThreadId(02) daemon:identity: linkerd2_app: Certified identity: default.postgres.serviceaccount.identity.linkerd.cluster.local

[pepsi1k]

Hi, @adleong! Sorry for the target message.

I don't fully understand how annotation config.linkerd.io/opaque-ports: works.
But it should be used with these annotations config.linkerd.io/skip-outbound-ports: some-port?

[zxxc]

@pepsi1k Have you tried opaque-ports with Redis?

I think we have similar issues with redis, postgresql, dynamodb

For now, we decided to remove linkerd from production, as we are experiencing a lot of random connectivity errors with databases and even timeout errors with HTTP services (it can be related to db issues)

[pepsi1k]

@zxxc No, I haven't tried opaque-ports with redis, because redis started without this annotation. The redis cluster deployed with a master and two replicas, аnd mtls worked between them