Replies: 12 comments 8 replies
-
|
@Flou21 You may be able to create a |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the quick reply. but i it does not work, I still get the following error message. |
Beta Was this translation helpful? Give feedback.
-
|
Another message from me. I changed the traffic split configuration again and now it works a bit better than before but not completly. and now i get this output the subscribe seems to work, but I have another problem with security.protocol but I'm working on it. |
Beta Was this translation helpful? Give feedback.
-
|
I have now the same problem as before but a better error message. Now I have the same problem as before, a service wants to access redis so it calls redis-headless-default.redis.svc.cluster.local. It get's a response that there are 3 redis instances (redis-node-0.redis-headless.redis.svc.cluster.local, redis-node-1.redis-headless.redis.svc.cluster.local, redis-node-2.redis-headless.redis.svc.cluster.local) after that successfull response it takes the master redis instance (in this case instance 1) and tries to communicate with it. But that request fails, because redis-node-1.redis-headless.redis.svc.cluster.local is not a valid hostname in the mirrored cluster. I tried to create the TrafficSplits to redirect the traffic but it didn't work. If anyone has another idea please tell me. Here is the redis error log just in case. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @Flou21, let me see if I understand this correctly:
Is this correct? Have you installed the multicluster extension with $ kubectl get service -n redis
# want to see the output for this
$ kubectl get ep -n redis
# want to see endpoints for services
$ kubectl get ep redis-headless -o yaml
# want to see output for headless endpoints objectIt seems to me to be the same problem as with Kafka. Oliver mentioned having a Note: in this case, it might be necessary to also include the actual hosts in the TrafficSplit resource, not just the headless service. It seems the headless service is queried fine by your client in the source cluster, but unless you also add the hosts under the headless service, you will run into the same issue (i.e calling |
Beta Was this translation helpful? Give feedback.
-
|
Hello @mateiidavid You understood that correctly. I think you are referring to this guide: https://linkerd.io/2.11/tasks/multicluster-using-statefulsets/ Here are the commands you wanted to see I tried creating all variants of TrafficSplits. and the same for the other nodes. I tried creating more TrafficSplits but they didn't work and then I deleted them. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @Flou21, Is your environment k3d at the moment? It seems that you have managed to successfully set up a multicluster environment, so you can follow the rest of the steps that pertain to the To be a bit more specific, to enable the feature, you can simply re-link your 2 clusters in your existing environment. You'd link them in the same way you have done before, with an additional install option: Without the feature enabled, you won't actually have any hosts so you can directly communicate with the pods, you will only have the core headless service. I'd suggest re-linking with the feature enabled and then trying again. You'll notice a difference in the set-up if you execute the same commands (e.g |
Beta Was this translation helpful? Give feedback.
-
|
Hello @mateiidavid My "main" cluster in the eu is normal k8s on rented servers. I will try to relink the clusters with this option today, maybe this is the only missing option. |
Beta Was this translation helpful? Give feedback.
-
|
hey @mateiidavid I reinstalled linkerd with multicluster and linked the clusters again. It looks better now, but it still does not work. I now have these services in my "us" cluster. this are the services in my "default" cluster endpoints: yaml for the service I tried again creating for every redis node. And this is the error message I still get. I hope this can help you I appreciate your help so much |
Beta Was this translation helpful? Give feedback.
-
|
Just a quick question to verify whether this would work, can you modify one of the TrafficSplits to be: service: redis-node-1.redis-headless.redis.svc.cluster.local
backends:
- service: redis-node-1.redis-headless-default.redis.svc.cluster.local
weight: 1000minstead of service: redis-node-1.redis-headless.redis.svc.cluster.local
backends:
- service: redis-node-1-default.redis.svc.cluster.local
weight: 1000mI'm curious if setting it directly to the instance will work in this case. |
Beta Was this translation helpful? Give feedback.
-
|
@mateiidavid I still get the same error message. Is there an intended way to configure this type of multi cluster communication. |
Beta Was this translation helpful? Give feedback.
-
|
So, I've looked into this a bit. My guess is that DNS isn't properly configured in your Out of curiosity though, what FQDN is your |
Beta Was this translation helpful? Give feedback.
-
|
I use the bitnami redis helm chart version 6.2.6 (https://github.com/bitnami/charts/tree/master/bitnami/redis) output from the pod in the I get the same error message when I want to connect to Hope this helps |
Beta Was this translation helpful? Give feedback.
-
|
@mateiidavid were you able to reproduce the problem? |
Beta Was this translation helpful? Give feedback.
-
|
@mateiidavid I really don't want to annoy you, but the issue is still relevant and I don't know what i should do |
Beta Was this translation helpful? Give feedback.
-
Hey @Flou21, I'm sorry I kept you waiting. I had a couple of pressing things to do since I last replied and I haven't had any time to look into this. I got around to it today, but unfortunately I wasn't able to reproduce this. I'll put all of my steps down here so we can go through them together.
# in my west cluster
:; k get po
NAME READY STATUS RESTARTS AGE
nginx-set-2 2/2 Running 0 15m
nginx-set-1 2/2 Running 0 14m
nginx-set-0 2/2 Running 0 14m
redis-node-0 3/3 Running 5 8m35s
redis-node-2 3/3 Running 5 8m14s
redis-node-1 3/3 Running 5 8m24s
# all nodes log this
16:20:13.35 WARN ==> redis-headless.default.svc.cluster.local does not contain the IP of this pod: 10.42.0.26
# this is weird, the endpoints object for sure contains the IP address
# not sure why it's complaining about it, but it's also out of scope here
# not likely it was introduced by us.
# In east, list pods
:; k get po
NAME READY STATUS RESTARTS AGE
curl-56dc7d945d-2tmpp 2/2 Running 0 44m
redis-client 2/2 Running 0 25m
# In east, list services
# we expect here 4 services for redis: one for headless
# and 3 for each node we have (3 in total).
:; k get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 90m
nginx-svc-west ClusterIP None <none> 80/TCP 43m
redis-headless-west ClusterIP None <none> 6379/TCP,26379/TCP 28m
nginx-set-2-west ClusterIP 10.43.41.196 <none> 80/TCP 17m
nginx-set-1-west ClusterIP 10.43.109.50 <none> 80/TCP 17m
nginx-set-0-west ClusterIP 10.43.210.204 <none> 80/TCP 17m
redis-node-0-west ClusterIP 10.43.174.118 <none> 6379/TCP,26379/TCP 11m
redis-node-1-west ClusterIP 10.43.141.222 <none> 6379/TCP,26379/TCP 11m
redis-node-2-west ClusterIP 10.43.221.17 <none> 6379/TCP,26379/TCP 11m
# All commands happen in east cluster
# Connect to redis-client
:; kubectl exec --tty -i redis-client \
--namespace default -c redis-client -- bash
# Inside container now, we can try to connect to mirror host
I have no name!@redis-client:/$ redis-cli -h redis-headless-west.default.svc.east.cluster.local -p 6379
# Start sending commands
redis-headless-west.default.svc.east.cluster.local:6379> ping
PONG
redis-headless-west.default.svc.east.cluster.local:6379>
redis-headless-west.default.svc.east.cluster.local:6379> ping
PONG
redis-headless-west.default.svc.east.cluster.local:6379> EXISTS foo
(integer) 0
redis-headless-west.default.svc.east.cluster.local:6379> SET foo 1
OK
redis-headless-west.default.svc.east.cluster.local:6379> EXISTS foo
(integer) 1
redis-headless-west.default.svc.east.cluster.local:6379> exit
I have no name!@redis-client:/$ exitThat concludes my investigation, for now. You'll notice at the end, we got a pong back from redis. We also checked if a key exists, set a value and then checked if it exists again. All of these commands worked for me from a redis-client pod in the source cluster; the cluster where redis wasn't deployed. The redis instance here came from At one point, I got this from sentinel: Also, for some reason redis signals the headless service does not contain the IP of the pod (which is definitely untrue). Could it perhaps be the Bitnami chart? From my side, the target cluster should have no issues with service discovery. The mirror service will just act as an endpoint to the server for your client, so it wouldn't need to do service discovery itself. More than happy to answer more questions here or help you further :) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much, I really appreciate your help. I will try installing redis without the bitnami chart and will tell you the results. |
Beta Was this translation helpful? Give feedback.
-
|
Please do! I'd like to understand what doesn't work in your set-up and see if there are any obvious faults on our side, or any painpoints associated with running statefulset workloads across clusters. |
Beta Was this translation helpful? Give feedback.
-
|
I recreated my setup. I deployed redis to my europe cluster and started a pod with redis-cli in the us cluster. Then I tried to deploy my application, which uses redis and kafka but the application is unable to connect to kafka or redis. I don't know why this happens, I'll keep trying tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
I've been running into the same issue trying to access a Kafka cluster through the multi-cluster extension. As far as I know when connecting to Kafka you first connect with a bootstrap node. This can be any of the nodes. (I believe) the bootstrap node will then reply with the dns records on which all brokers can be reached. Herein lies the problem. The Kafka brokers are usually accessed through a headless service. As an example let's name it
The problem here is that linkerd will post-fix the headless service with the originating cluster name (e.g. Finding a workaroundAfter some hours I was luckily able to find a workaround. In our case we are using the Strimzi operator to deploy a Kafka cluster. Part of the solution is to add an extra listener with the cluster ip type. From Strimzi docs they write:
In the example below I named the extra listener kafka:
version: 3.3.1
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: external
type: cluster-ip
tls: false
port: 9096This exposes three new services in the west cluster:
When connecting to one of these service Kafka still responds with the DNS name not including apiVersion: v1
kind: Service
metadata:
name: main-kafka-external-0
namespace: kafka
spec:
ports:
- port: 9096
externalName: main-kafka-external-0-west.kafka.svc.cluster.local
type: ExternalName
---
apiVersion: v1
kind: Service
metadata:
name: main-kafka-external-1
namespace: kafka
spec:
ports:
- port: 9096
externalName: main-kafka-external-1-west.kafka.svc.cluster.local
type: ExternalName
---
apiVersion: v1
kind: Service
metadata:
name: main-kafka-external-2
namespace: kafka
spec:
ports:
- port: 9096
externalName: main-kafka-external-2-west.kafka.svc.cluster.local
type: ExternalNameNow when the broker tries to redirect the client to It would be nice if we could label service in the origin cluster to define how it should be exposed, as it can make your life difficult when working with workloads like Kafka. Perhaps: mirror.linkerd.io/exported-name-postfix: ""A similar issue are discussed here: #10275 In the meantime I hope this helps others running into the same issue as me. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hello there
I have a problem with the generated name of a mirrored service
Problem
My main cluster is running in the eu, there runs a kafka cluster. Now I created a little cluster in the us, that should access the kafka cluster in the eu. I have installed linkerd with mutlicluster functionality. For most services this works out of the box. But I have a problem with kafka. The kafka cluster consists of the 3 instances.
When a pod in the us wants to connect to kafka, the pod uses the service
kafka-helm-default.kafka.kafka-helm-defaultis the name of the mirrored service andkafkais the namespace. The initial request works but then kafka returns, that an individual kafka instance listens onkafka-helm-0.kafka-helm.kafka. This is the podkafka-helm-0, the servicekafka-helmbut the-defaultis missing because in the eu cluster (which is the default cluster), and the namespacekafkais the same again.So my problem is, that in the eu cluster the name of the service is
kafka-helmand in the us cluster the name iskafka-helm-default. Can I rename the name of the mirrored service that linkerd generates or does the name have to be `-?I did not find anything in the documentation and don't know what else I can do. Sorry for the confusing naming
helmanddefaultwhen I created the cluster I did not know that I will build a multicluster some day.Beta Was this translation helpful? Give feedback.
All reactions