-
Notifications
You must be signed in to change notification settings - Fork 248
Description
Current Behavior
高并发下,网关重试会出现空指针进一步导致worker exit:exit status 2
Expected Behavior
不要退出worker进程
Error Logs
[2025-11-24 16:21:36] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:21:39] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:22:30] [WARN] [master-1556754]1.7639725506145723e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.004379732}
[2025-11-24 16:22:30] [WARN] [master-1556754]1.7639725506146913e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.004519528}
[2025-11-24 16:23:26] [WARN] [master-1556754]1.7639726068055117e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.034523412}
[2025-11-24 16:23:26] [WARN] [master-1556754]1.7639726068056629e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.03469602}
[2025-11-24 16:23:27] [WARN] [master-1556754]1.7639726076038992e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.035989961}
[2025-11-24 16:23:27] [WARN] [master-1556754]1.7639726076084082e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.040509611}
[2025-11-24 16:24:59] [WARN] [master-1556754]1.7639726995577307e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.081778126}
[2025-11-24 16:24:59] [WARN] [master-1556754]1.763972699557873e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.08194328}
[2025-11-24 16:25:01] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:25:04] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:25:04] [ERRO] [worker-1556931]complete timeout
[2025-11-24 16:25:07] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x174bba0]
goroutine 250916 [running]:
github.com/valyala/fasthttp.(*Request).Reset(
0x0)
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/http.go:1125 +0x60
github.com/valyala/fasthttp.ReleaseRequest
(0x0)
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/client.go:1226 +0x18
github.com/eolinker/apinto/node/http-context.(*HttpContext).FastFinish(0xc00095a288)
/mnt/c/computer/code/aesb-opens/apinto/node/http-context/context.go:436
+0x33d
github.com/eolinker/apinto/node/http-context.(*HttpContext).SendTo.func1.1()
/mnt/c/computer/code/aesb-opens/apinto/node/http-context/context.go:253 +0x56
github.com/eolinker/apinto/node/http-context.(*HttpContext).SendTo.func1
(0xc000748100)
/mnt/c/computer/code/aesb-opens/apinto/node/http-context/context.go:290 +0x865
github.com/valyala/fasthttp.NewStreamReader.func1()
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/stream.go:44 +
0x42
created by
github.com/valyala/fasthttp.NewStreamReader in goroutine 247924
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/stream.go:43
+0x227
[2025-11-24 16:25:24] [WARN] [master-1556754]worker exit:exit status 2
[2025-11-24 16:25:24] [ERRO] [master-1556754]worker process quit error: os: process already finished
[2025-11-24 16:25:25] [INFO] [worker-2230531]worker process start...
[2025-11-24 16:25:25] [INFO] [worker-2230531]time spend:NewWorkerServer:22.96378ms
[2025-11-24 16:25:25] [INFO] [worker-2230531]start worker unix server: /tmp/apinto/apinto.worker-2230531.sock
[2025-11-24 16:25:25] [INFO] [worker-2230531]time spend:port-reqiure unix:/tmp/apinto/apinto.worker-2230531.sock:146.68µs
[2025-11-24 16:25:25] [INFO] [master-1556754]time spend:wait [worker] process start::910.744387ms
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen name: 10.11.55.83:8099
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen name: 10.11.55.83:8099
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen name: 10.11.55.83:8099
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen: 3
[2025-11-24 16:25:25] [INFO] [worker-2230531]index: 0 name: template
Steps to Reproduce
1.我们注册了一个服务
2.多线程调用
3.当这个服务访问超时时,网关会重试,重试过程中会出现空指针
4.最后导致worker exit
Environment
v.22.16
go1.24.10 linux/amd64