TrafficSplit doesn't failover when all pods in primary svc are scaled down

[kinghrothgar]

I installed SMI via:

linkerd smi install | kubectl apply -f -

I've setup mutlicluster and have it working between clusters cluster-1 and cluster-2. I've mirrored a service from cluster-1 to cluster-2 which resulted in a svc named echo-go-cluster-1. In this cluster-2 I have the following:

apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
  name: echo-go
  namespace: default
spec:
  service: echo-go
  backends:
  - service: echo-go
    weight: 1
  - service: echo-go-cluster-1
    weight: 0

When I scale down the backing pods for echo-go services, curls to the echo-go service 504:

content-length: 0
l5d-proxy-error: HTTP Balancer service in fail-fast
connection: close
l5d-proxy-connection: close
date: Mon, 03 Oct 2022 04:38:51 GMT

If I adjust echo-go-cluster-1 to have a weight of 1 and echo-go to have a weight of 0, it works. It seems like it's not detecting that all the pods behind echo-go are gone and routing traffic to echo-go-cluster-1.

Any ideas?

[kleimkuhler]

I see that you have the TrafficSplit created and configured properly, but are you using linkerd-failover? I don't see any mention of you installing that or applying the required annotations to the TrafficSplit. I'd take a look at the (brief) Getting Started and make sure that you've followed all those steps.