Intermittent response_code=0 and 404 in Envoy Gateway data plane logs under hey stress test — causes and troubleshooting

[lideheng6379-del]

I am using the hey tool to perform load testing on an Envoy Gateway. However, I occasionally see response_code=0 and response_code=404 in the data plane logs, while the majority of response codes are 200.

Here are two example log excerpts from the data plane:

09-29 00:16:38.008
response_code:0 bytes_received:0 duration:2 response_code_details:filter_chain_not_found downstream_remote_address:185.244.104.2:33586 bytes_sent:0 response_flags:NR start_time:2025-09-28T16:16:37.713Z downstream_local_address:172.18.1.26:10443

09-29 00:11:17.019
response_code:404 bytes_received:0 duration:2 protocol:HTTP/1.1 upstream_cluster:httproute/upgrade/nginx/rule/0 upstream_host:172.18.16.6:80 authority:43.140.73.225:80 response_code_details:via_upstream x-request-id:e9fed011-0b6d-4dcf-abc8-df94c7198c31 method:GET route_name:httproute/upgrade/nginx/rule/0/match/0/* x-envoy-origin-path:/_ignition/execute-solution downstream_remote_address:79.124.58.198:37670 x-forwarded-for:79.124.58.198 bytes_sent:555 response_flags:- start_time:2025-09-28T16:11:15.815Z downstream_local_address:172.18.1.26:10080 upstream_local_address:172.18.1.26:60626 user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36
Kubernetes deployment YAML configuration:

yaml：
``
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: proxy-config
namespace: upgrade
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 3
container:
resources:
requests:
cpu: 1
memory: 2Gi
tke.cloud.tencent.com/eni-ip: "1"
limits:
cpu: 1
memory: 2Gi
tke.cloud.tencent.com/eni-ip: "1"
pod:
annotations:
tke.cloud.tencent.com/networks: tke-route-eni
envoyService:
annotations:
service.cloud.tencent.com/direct-access: "true"
logging:
level:
default: info

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: gwc
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: proxy-config
namespace: upgrade

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: gw
namespace: upgrade
spec:
gatewayClassName: gwc
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: Same

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: nginx
namespace: upgrade
spec:
parentRefs:
- name: gw
rules:
- backendRefs:
- name: nginx
namespace: test
port: 80
matches:
- path:
type: PathPrefix
value: /

``

[arkodg]

do you see it in the beginning or in the middle of the test ?

[lideheng6379-del]

in the beginning

[arkodg]

this is around eventual consistency, takes around a second for the applied routes to get programmed in the data plane

[lideheng6379-del]

It’s not just happening on the first request, but this situation occurs during every concurrent request. The route configuration has not changed and should have long been applied. So why does this still happen?

[lideheng6379-del]

Yesterday, when I made requests, this issue occurred. Today, when I make requests again, I still get 404 responses, even though the route configuration hasn’t been changed

[arkodg]

Take a look at #2675, you may be hitting h2 connection coalescing