httproute not work for some pods after delete the service profile

[wgjak47]

What is the issue?

We are migrating from service profile to httproute, but after deleting the service profile and apply the new http route, the dynamic routing still not work for some pods.

How can it be reproduced?

create deployment canary-example with this python flask server:

from flask import Flask, jsonify

app = Flask(__name__)


@app.route('/')
def check_canary():
    return jsonify({'answer': 'apple'})


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True)

create deployment canary-example-primary with this python flask server:

from flask import Flask, jsonify

app = Flask(__name__)


@app.route('/')
def check_canary():
    return jsonify({'answer': 'pie'})


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True)

we add the linkerd.io/inject: enabled on the pod template annotations of course

create three service（mostly generated by flagger）:

apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      helm.toolkit.fluxcd.io/driftDetection: disabled
      kustomize.toolkit.fluxcd.io/reconcile: disabled
    creationTimestamp: "2024-09-25T09:04:38Z"
    labels:
      app: canary-example
    name: canary-example
    namespace: test-canary
    ownerReferences:
    - apiVersion: flagger.app/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Canary
      name: canary-example
      uid: 2f606031-bfd2-4d88-a7d4-322f74857773
    resourceVersion: "187437670"
    uid: 80ac7209-8946-431a-9e80-ad609010263b
  spec:
    clusterIP: 192.168.146.209
    clusterIPs:
    - 192.168.146.209
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: http
      port: 5000
      protocol: TCP
      targetPort: 5000
    selector:
      app: canary-example-primary
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      helm.toolkit.fluxcd.io/driftDetection: disabled
      kustomize.toolkit.fluxcd.io/reconcile: disabled
    creationTimestamp: "2024-09-25T09:03:38Z"
    labels:
      app: canary-example-canary
    name: canary-example-canary
    namespace: test-canary
    ownerReferences:
    - apiVersion: flagger.app/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Canary
      name: canary-example
      uid: 2f606031-bfd2-4d88-a7d4-322f74857773
    resourceVersion: "25078419"
    uid: 0f59af1a-8a09-4307-8489-f6b7d3febf3a
  spec:
    clusterIP: 192.168.138.77
    clusterIPs:
    - 192.168.138.77
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: http
      port: 5000
      protocol: TCP
      targetPort: 5000
    selector:
      app: canary-example
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      helm.toolkit.fluxcd.io/driftDetection: disabled
      kustomize.toolkit.fluxcd.io/reconcile: disabled
    creationTimestamp: "2024-09-25T09:03:38Z"
    labels:
      app: canary-example-primary
    name: canary-example-primary
    namespace: test-canary
    ownerReferences:
    - apiVersion: flagger.app/v1beta1
      blockOwnerDeletion: true
      controller: true
      kind: Canary
      name: canary-example
      uid: 2f606031-bfd2-4d88-a7d4-322f74857773
    resourceVersion: "25078424"
    uid: b3779025-5302-437c-a5c2-e44b96178c7a
  spec:
    clusterIP: 192.168.57.12
    clusterIPs:
    - 192.168.57.12
    internalTrafficPolicy: Cluster
    ipFamilies:
    - IPv4
    ipFamilyPolicy: SingleStack
    ports:
    - name: http
      port: 5000
      protocol: TCP
      targetPort: 5000
    selector:
      app: canary-example-primary
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""

create a service profile

apiVersion: linkerd.io/v1alpha2
kind: ServiceProfile
metadata:
  name: canary-example.test-canary.svc.cluster.local
  namespace: test-canary
spec:
  routes:
  - condition:
      method: GET
      pathRegex: /
    name: root

create a httproute:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  annotations:
    helm.toolkit.fluxcd.io/driftDetection: disabled
    kustomize.toolkit.fluxcd.io/reconcile: disabled
  creationTimestamp: "2024-09-25T09:04:38Z"
  generation: 20
  name: canary-example
  namespace: test-canary
  ownerReferences:
  - apiVersion: flagger.app/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Canary
    name: canary-example
    uid: 2f606031-bfd2-4d88-a7d4-322f74857773
  resourceVersion: "282600768"
  uid: fae07c19-f3ff-45b0-8793-6a795a4b087c
spec:
  parentRefs:
  - group: core
    kind: Service
    name: canary-example
    namespace: test-canary
    port: 5000
  rules:
  - backendRefs:
    - filters:
      - requestHeaderModifier:
          add:
          - name: l5d-dst-canonical
            value: canary-example-primary.test-canary.svc.cluster.local:5000
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: canary-example-primary
      port: 5000
      weight: 40
    - filters:
      - requestHeaderModifier:
          add:
          - name: l5d-dst-canonical
            value: canary-example-canary.test-canary.svc.cluster.local:5000
        type: RequestHeaderModifier
      group: ""
      kind: Service
      name: canary-example-canary
      port: 5000
      weight: 60
    matches:
    - path:
        type: PathPrefix
        value: /
status:
  parents:
  - conditions:
    - lastTransitionTime: "2025-07-25T10:17:43Z"
      message: ""
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-07-25T10:17:43Z"
      message: ""
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: linkerd.io/policy-controller
    parentRef:
      group: core
      kind: Service
      name: canary-example
      namespace: test-canary
      port: 5000

create a deployment with a load test program on same namespace, set url env: http://canary-example:5000:

❯ cat request_loop.py
import os
import time
import requests
import sys

def main():
    # 从环境变量获取URL
    url = os.getenv('URL')
    
    if not url:
        print("错误: 请设置URL环境变量")
        sys.exit(1)
    
    print(f"开始请求URL: {url}")
    print("按Ctrl+C停止...")
    
    try:
        while True:
            try:
                response = requests.get(url)
                print(f"状态码: {response.status_code}, context: {response.json()} 时间: {time.strftime('%Y-%m-%d %H:%M:%S')}")
            except requests.exceptions.RequestException as e:
                print(f"请求失败: {e}")
            
            time.sleep(1)
    except KeyboardInterrupt:
        print("\n已停止请求")

if __name__ == "__main__":
    main()

after loadtest runing for 5 mins, delete the serviceprofile. you can see the all traffic still go to the canary-example-primary.

and then exec other pods which is meshed, request the http://canary-example.test-canary:5000, we can see the traffic splits to canary-example-primary and canary-example service.

and we restart the loadtest deployment, we found the http route works.

Logs, error output, etc

we found some debug log of linkerd proxy, after delete the serviceprofile, we can this the sidecar change to ClientPolicy, but seems linkerd_distribute::stack didn't update the right httproute info.

[   158.984859s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: tower::buffer::worker: service.ready=true processing request
[   158.984903s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_detect: Detected protocol protocol=Some(HTTP/1) elapsed=16.04µs
[   158.984914s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_proxy_http::server: Creating HTTP service version=HTTP/1
[   158.984918s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_app_outbound::sidecar: Using ServiceProfile
[   158.984929s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_proxy_http::server: Handling as HTTP version=HTTP/1
[   158.984948s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::io: parsed 5 headers
[   158.984953s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::conn: incoming body is empty
[   158.985000s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}: tower::balance::p2c::service: updating from discover
[   158.985017s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: tower::buffer::worker: service.ready=true processing request
[   158.985021s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}: linkerd_proxy_http::classify::channel: state=Some(State { classify: Http(StatusRanges([500..=599])), tx: Sender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x7fa169bda900, tail_position: 0 }, semaphore: Semaphore { semaphore: Semaphore { permits: 10000 }, bound: 10000 }, rx_waker: AtomicWaker, tx_count: 2, rx_fields: "..." } } } })
[   158.985036s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}:orig-proto-upgrade: linkerd_proxy_http::client: method=GET uri=http://canary-example:5000/ version=HTTP/1.1
[   158.985043s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}:orig-proto-upgrade: linkerd_proxy_http::client: headers={"host": "canary-example:5000", "user-agent": "python-requests/2.32.4", "accept-encoding": "gzip, deflate", "accept": "*/*", "l5d-dst-canonical": "canary-example.test-canary.svc.cluster.local:5000"}
[   158.985051s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}:orig-proto-upgrade: linkerd_proxy_http::orig_proto: Upgrading request version=HTTP/1.1 absolute_form=false
[   158.985119s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(285), flags: (0x5: END_HEADERS | END_STREAM) }
[   158.986741s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(285), flags: (0x4: END_HEADERS) }
[   158.986776s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}: linkerd_app_outbound::http::handle_proxy_error_headers: Remote proxy error
[   158.986833s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::io: flushed 146 bytes
[   158.986856s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.65.48:5000}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(285), flags: (0x1: END_STREAM) }
[   158.986883s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_proxy_http::classify::channel: dropping ResponseBody
[   158.986888s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_proxy_http::classify::channel: sending EOS to classify
[   158.986920s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38424 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::io: flushed 24 bytes
[   158.987279s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38408 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::conn: read eof
[   158.987337s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38408 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_proxy_http::server: The client is shutting down the connection res=Ok(())
[   159.055222s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:profiles:controller{addr=linkerd-dst-headless.linkerd.svc.cluster.local:8086}:endpoint{addr=10.80.88.184:8086}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(1) }
[   159.889825s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: tower::buffer::worker: service.ready=true processing request
[   159.889868s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_detect: Detected protocol protocol=Some(HTTP/1) elapsed=17.082µs
[   159.889875s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_proxy_http::server: Creating HTTP service version=HTTP/1
[   159.889879s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_app_outbound::sidecar: Using ClientPolicy routes
[   159.889890s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}: linkerd_proxy_http::server: Handling as HTTP version=HTTP/1
[   159.889908s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::io: parsed 5 headers
[   159.889911s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::conn: incoming body is empty
[   159.889936s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_distribute::stack: New distribution backends=[Concrete { target: Balance(NameAddr { name: Name("canary-example.test-canary.svc.cluster.local"), port: 5000 }, EwmaConfig { default_rtt: 30ms, decay: 10s }), authority: Some(canary-example.test-canary.svc.cluster.local:5000), parent: Http(HttpSidecar { orig_dst: OrigDstAddr(192.168.146.209:5000), version: HTTP/1, routes: Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<linkerd_app_outbound::http::logical::Routes>>, RwLock { data: Profile(Routes { addr: LogicalAddr(canary-example.test-canary.svc.cluster.local:5000), routes: [], targets: [] }) }), version: Version(2), is_closed: false, ref_count_rx: 55 }, version: Version(0) } }), parent_ref: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), backend_ref: BackendRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), failure_accrual: None }]
[   159.889988s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}: tower::balance::p2c::service: updating from discover
[   159.890007s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: tower::buffer::worker: service.ready=true processing request
[   159.890012s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}: linkerd_proxy_http::classify::channel: state=Some(State { classify: Http(StatusRanges([500..=599])), tx: Sender { chan: Tx { inner: Chan { tx: Tx { block_tail: 0x7fa169bdae00, tail_position: 0 }, semaphore: Semaphore { semaphore: Semaphore { permits: 10000 }, bound: 10000 }, rx_waker: AtomicWaker, tx_count: 2, rx_fields: "..." } } } })
[   159.890028s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}:orig-proto-upgrade: linkerd_proxy_http::client: method=GET uri=http://canary-example:5000/ version=HTTP/1.1
[   159.890037s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}:orig-proto-upgrade: linkerd_proxy_http::client: headers={"host": "canary-example:5000", "user-agent": "python-requests/2.32.4", "accept-encoding": "gzip, deflate", "accept": "*/*", "l5d-dst-canonical": "canary-example.test-canary.svc.cluster.local:5000"}
[   159.890045s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}:orig-proto-upgrade: linkerd_proxy_http::orig_proto: Upgrading request version=HTTP/1.1 absolute_form=false
[   159.890102s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}:h2:Connection{peer=Client}: h2::codec::framed_write: send frame=Headers { stream_id: StreamId(93), flags: (0x5: END_HEADERS | END_STREAM) }
[   159.893648s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Headers { stream_id: StreamId(93), flags: (0x4: END_HEADERS) }
[   159.893666s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:41534 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}:h2:Connection{peer=Client}: h2::codec::framed_read: received frame=Data { stream_id: StreamId(93), flags: (0x1: END_STREAM) }
[   159.893694s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http:service{ns=test-canary name=canary-example port=5000}:endpoint{addr=10.80.153.3:5000}: linkerd_app_outbound::http::handle_proxy_error_headers: Remote proxy error
[   159.893721s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_proxy_http::classify::channel: dropping ResponseBody
[   159.893725s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_proxy_http::classify::channel: sending EOS to classify
[   159.893767s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38436 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::io: flushed 170 bytes
[   159.894192s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: hyper::proto::h1::conn: read eof
[   159.894240s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.66.28:38420 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_proxy_http::server: The client is shutting down the connection res=Ok(())

after the deployment restart, I can see linkerd_distribute::stack update with right info:

[     1.712172s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.65.40:40112 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_distribute::stack: New distribution backends=[Backend { route_ref: RouteRef(Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }), concrete: Concrete { target: Balance(NameAddr { name: Name("canary-example-primary.test-canary.svc.cluster.local"), port: 5000 }, EwmaConfig { default_rtt: 30ms, decay: 10s }), authority: Some(canary-example-primary.test-canary.svc.cluster.local:5000), parent: Http(HttpSidecar { orig_dst: OrigDstAddr(192.168.146.209:5000), version: HTTP/1, routes: Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<linkerd_app_outbound::http::logical::Routes>>, RwLock { data: Policy(Http(Params { addr: Socket(192.168.146.209:5000), meta: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), routes: [Route { hosts: [], rules: [Rule { matches: [MatchRequest { path: Some(Prefix("/")), headers: [], query_params: [], method: None }], policy: RoutePolicy { meta: Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }, filters: [], distribution: RandomAvailable([(RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-primary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50), (RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-canary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50)]), request_timeout: None, failure_policy: StatusRanges([500..=599]) } }] }], backends: [Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Default { name: "service" }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }], failure_accrual: None })) }), version: Version(0), is_closed: false, ref_count_rx: 41 }, version: Version(0) } }), parent_ref: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), backend_ref: BackendRef(Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }), failure_accrual: None }, filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-primary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], request_timeout: None }, Backend { route_ref: RouteRef(Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }), concrete: Concrete { target: Balance(NameAddr { name: Name("canary-example-canary.test-canary.svc.cluster.local"), port: 5000 }, EwmaConfig { default_rtt: 30ms, decay: 10s }), authority: Some(canary-example-canary.test-canary.svc.cluster.local:5000), parent: Http(HttpSidecar { orig_dst: OrigDstAddr(192.168.146.209:5000), version: HTTP/1, routes: Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<linkerd_app_outbound::http::logical::Routes>>, RwLock { data: Policy(Http(Params { addr: Socket(192.168.146.209:5000), meta: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), routes: [Route { hosts: [], rules: [Rule { matches: [MatchRequest { path: Some(Prefix("/")), headers: [], query_params: [], method: None }], policy: RoutePolicy { meta: Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }, filters: [], distribution: RandomAvailable([(RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-primary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50), (RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-canary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50)]), request_timeout: None, failure_policy: StatusRanges([500..=599]) } }] }], backends: [Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Default { name: "service" }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }], failure_accrual: None })) }), version: Version(0), is_closed: false, ref_count_rx: 41 }, version: Version(0) } }), parent_ref: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), backend_ref: BackendRef(Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }), failure_accrual: None }, filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-canary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], request_timeout: None }] 

[     1.712172s] DEBUG ThreadId(01) outbound:accept{client.addr=10.80.65.40:40112 server.addr=192.168.146.209:5000}:proxy{addr=192.168.146.209:5000}:http: linkerd_distribute::stack: New distribution backends=[Backend { route_ref: RouteRef(Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }), concrete: Concrete { target: Balance(NameAddr { name: Name("canary-example-primary.test-canary.svc.cluster.local"), port: 5000 }, EwmaConfig { default_rtt: 30ms, decay: 10s }), authority: Some(canary-example-primary.test-canary.svc.cluster.local:5000), parent: Http(HttpSidecar { orig_dst: OrigDstAddr(192.168.146.209:5000), version: HTTP/1, routes: Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<linkerd_app_outbound::http::logical::Routes>>, RwLock { data: Policy(Http(Params { addr: Socket(192.168.146.209:5000), meta: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), routes: [Route { hosts: [], rules: [Rule { matches: [MatchRequest { path: Some(Prefix("/")), headers: [], query_params: [], method: None }], policy: RoutePolicy { meta: Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }, filters: [], distribution: RandomAvailable([(RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-primary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50), (RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-canary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50)]), request_timeout: None, failure_policy: StatusRanges([500..=599]) } }] }], backends: [Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Default { name: "service" }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }], failure_accrual: None })) }), version: Version(0), is_closed: false, ref_count_rx: 41 }, version: Version(0) } }), parent_ref: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), backend_ref: BackendRef(Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }), failure_accrual: None }, filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-primary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], request_timeout: None }, Backend { route_ref: RouteRef(Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }), concrete: Concrete { target: Balance(NameAddr { name: Name("canary-example-canary.test-canary.svc.cluster.local"), port: 5000 }, EwmaConfig { default_rtt: 30ms, decay: 10s }), authority: Some(canary-example-canary.test-canary.svc.cluster.local:5000), parent: Http(HttpSidecar { orig_dst: OrigDstAddr(192.168.146.209:5000), version: HTTP/1, routes: Receiver { shared: Shared { value: RwLock(PhantomData<std::sync::rwlock::RwLock<linkerd_app_outbound::http::logical::Routes>>, RwLock { data: Policy(Http(Params { addr: Socket(192.168.146.209:5000), meta: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), routes: [Route { hosts: [], rules: [Rule { matches: [MatchRequest { path: Some(Prefix("/")), headers: [], query_params: [], method: None }], policy: RoutePolicy { meta: Resource { group: "gateway.networking.k8s.io", kind: "HTTPRoute", name: "canary-example", namespace: "test-canary", section: None, port: None }, filters: [], distribution: RandomAvailable([(RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-primary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50), (RouteBackend { filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-canary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], backend: Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }, request_timeout: None }, 50)]), request_timeout: None, failure_policy: StatusRanges([500..=599]) } }] }], backends: [Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-primary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-primary.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Default { name: "service" }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example.test-canary.svc.cluster.local:5000" }) }, Backend { meta: Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }, queue: Queue { capacity: 100, failfast_timeout: 3s }, dispatcher: BalanceP2c(PeakEwma(PeakEwma { decay: 10s, default_rtt: 30ms }), DestinationGet { path: "canary-example-canary.test-canary.svc.cluster.local:5000" }) }], failure_accrual: None })) }), version: Version(0), is_closed: false, ref_count_rx: 41 }, version: Version(0) } }), parent_ref: ParentRef(Resource { group: "core", kind: "Service", name: "canary-example", namespace: "test-canary", section: None, port: Some(5000) }), backend_ref: BackendRef(Resource { group: "core", kind: "Service", name: "canary-example-canary", namespace: "test-canary", section: None, port: Some(5000) }), failure_accrual: None }, filters: [RequestHeaders(ModifyHeader { add: [("l5d-dst-canonical", "canary-example-canary.test-canary.svc.cluster.local:5000")], set: [], remove: [] })], request_timeout: None }] 

output of linkerd check -o short

❯ linkerd check -o short
linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2025-08-06T08:53:33Z
    see https://linkerd.io/2.14/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
    certificate will expire on 2025-08-06T08:53:33Z
    see https://linkerd.io/2.14/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
    certificate will expire on 2025-08-06T08:53:33Z
    see https://linkerd.io/2.14/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ cli is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-54899858f6-4g8gq (stable-2.14.10)
        * linkerd-destination-54899858f6-77x26 (stable-2.14.10)
        * linkerd-destination-54899858f6-xkscw (stable-2.14.10)
        * linkerd-identity-cd555f6dd-6kq7c (stable-2.14.10)
        * linkerd-identity-cd555f6dd-6pltb (stable-2.14.10)
        * linkerd-identity-cd555f6dd-jstlz (stable-2.14.10)
        * linkerd-proxy-injector-67d777b569-qzpvp (stable-2.14.10)
        * linkerd-proxy-injector-67d777b569-vlkpb (stable-2.14.10)
        * linkerd-proxy-injector-67d777b569-vpbtd (stable-2.14.10)
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints

- Running viz extension check ^C
❯ linkerd check -o short
linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2025-08-06T08:53:33Z
    see https://linkerd.io/2.14/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-webhooks-and-apisvc-tls
-------------------------------
‼ proxy-injector cert is valid for at least 60 days
    certificate will expire on 2025-08-06T08:53:33Z
    see https://linkerd.io/2.14/checks/#l5d-proxy-injector-webhook-cert-not-expiring-soon for hints
‼ sp-validator cert is valid for at least 60 days
    certificate will expire on 2025-08-06T08:53:33Z
    see https://linkerd.io/2.14/checks/#l5d-sp-validator-webhook-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ cli is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-54899858f6-4g8gq (stable-2.14.10)
        * linkerd-destination-54899858f6-77x26 (stable-2.14.10)
        * linkerd-destination-54899858f6-xkscw (stable-2.14.10)
        * linkerd-identity-cd555f6dd-6kq7c (stable-2.14.10)
        * linkerd-identity-cd555f6dd-6pltb (stable-2.14.10)
        * linkerd-identity-cd555f6dd-jstlz (stable-2.14.10)
        * linkerd-proxy-injector-67d777b569-qzpvp (stable-2.14.10)
        * linkerd-proxy-injector-67d777b569-vlkpb (stable-2.14.10)
        * linkerd-proxy-injector-67d777b569-vpbtd (stable-2.14.10)
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints

Environment

• Kubernetes 1.28
• Cloud Provider: Aliyun ACK 1.28
• Alibaba Cloud Linux 3 (based on centos)
• linkerd version: 2.14.10

Possible solution

we found if the pod stop requesting the target for a while, the discovery cache will expired and httproute works.

Additional context

for some reason, we cannot restart all the deployment every time for migrating service profile to Httproute, hope a better solution.

Would you like to work on fixing this bug?

None

[wgjak47]

I have found the possible problem here:
on linkerd/app/outbound/src/sidecar.rs, using service profile or client policy, will return the same httpsidecar key here:

    fn hash<H: std::hash::Hasher>(&self, state: &mut H) {
        self.orig_dst.hash(state);
        self.version.hash(state);
    }

So if a pod keep requesting the target, the idle cache will never expired.

[wgjak47]

I created a PR linkerd/linkerd2-proxy#4064 and test it with our prod environment, this PR could fix this problem. Hope for code review.