Traffic Splits with GCE Ingress

[dnjp]

Hello! I am trying to get traffic splits working with the GCE Ingress. I got route based metrics working by configuring a BackendConfig for the service like this:

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: app
spec:
  customRequestHeaders:
    headers:
    - "l5d-dst-override:app.default.svc.cluster.local:80"

The Ingress is defined like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.gcp.kubernetes.io/pre-shared-cert: app-ssl
    # ingress.kubernetes.io/custom-request-headers: 'l5d-dst-override: app.default.svc.cluster.local:80' <-- this did not work
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.global-static-ip-name: east-app
  name: app-ingress
  namespace: default
spec:
  rules:
  - host: app.adomain.com
    http:
      paths:
      - backend:
          service:
            name: app
            port:
              number: 80
        path: /*

and the service defined like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/backend-config: '{"ports": {"80":"app"}}'
    cloud.google.com/neg: '{"ingress": true}'
  name: app
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 7000
  selector:
    run: app
  type: NodePort

I created a baseline service to test the traffic split like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    release: baseline
    run: app
  name: app-baseline
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 7000
  selector:
    run: app
    release: baseline
  type: NodePort

and a canary service like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    release: canary
    run: app
  name: app-canary
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 7000
  selector:
    release: canary
    run: app
  type: NodePort

and finally, the traffic split:

apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
spec:
  service: app
  backends:
    - service: app-baseline
      weight: 100
    - service: app-canary
      weight: 0      
metadata:
  name: app
  namespace: default
  labels:
    component: app
    release: canary

However, no metrics are showing in the viz dashboard for this traffic split. If I tap the application with linkerd viz tap deploy/app -o json, the l5d-dst-override header does not show up in the headers array. However, before setting up the BackendConfig route based metrics were not working but after I set it up it does in fact work.

I've obfuscated the service names/routes as they expose vendor information which I don't believe I can share, but here are some screenshots.

Route Metrics:

Traffic Split:

Load Balancer:

I tried switching from stable-2.10.2 to edge-21.6.3 which appears to support adding dstOverrides in the ServiceProfile as an alternative way to create traffic splits. Here's how I changed the ServiceProfile:

spec:
  dstOverrides:
  - authority: <app>.default.svc.cluster.local:80
    weight: 0
  - authority: <app>-baseline.svc.cluster.local:80
    weight: 100

Switching to this version broke route metrics and dstOverrides in the service profile also does not work. However, I've verified the l5d-dst-override header is being set via tap. It is showing the following element in the headers array:

      {
        "name": "l5d-dst-override",
        "valueStr": "<app>.default.svc.cluster.local:80"
      }

I get route metrics again if I switch back to stable. Do traffic splits only work with ingress controllers where you can inject the linkerd sidecar in ingress mode which is not possible with GCE Ingresses or is there another way that Linkerd can determine how to route traffic for a traffic split?

Here is the output of linkerd check:
linkerdcheck.txt

[cpretzer]

... Do traffic splits only work with ingress controllers where you can inject the linkerd sidecar in ingress mode which is not possible with GCE Ingresses or is there another way that Linkerd can determine how to route traffic for a traffic split?

@dnjp If I'm not mistaken, the traffic splitting happens on the client side, so the ingress pod has to be injected with the Linkerd proxy for the traffic split to work.

[dnjp]

That makes sense. The documentation makes it seem like this functionality is supported. We will be trying out the Gateway API in order to support this feature.