Skip to content

bug: 高并发下,网关重试会出现空指针进一步导致worker exit #231

@dongsoft

Description

@dongsoft

Current Behavior

高并发下,网关重试会出现空指针进一步导致worker exit:exit status 2

Expected Behavior

不要退出worker进程

Error Logs

[2025-11-24 16:21:36] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:21:39] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:22:30] [WARN] [master-1556754]1.7639725506145723e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.004379732}
[2025-11-24 16:22:30] [WARN] [master-1556754]1.7639725506146913e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.004519528}
[2025-11-24 16:23:26] [WARN] [master-1556754]1.7639726068055117e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.034523412}
[2025-11-24 16:23:26] [WARN] [master-1556754]1.7639726068056629e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.03469602}
[2025-11-24 16:23:27] [WARN] [master-1556754]1.7639726076038992e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.035989961}
[2025-11-24 16:23:27] [WARN] [master-1556754]1.7639726076084082e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.040509611}
[2025-11-24 16:24:59] [WARN] [master-1556754]1.7639726995577307e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "d5ad5cad4c897509", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.081778126}
[2025-11-24 16:24:59] [WARN] [master-1556754]1.763972699557873e+09 warn leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk {"to": "5a169f31b0c8bce", "heartbeat-interval": 0.1, "expected-duration": 0.2, "exceeded-duration": 0.08194328}
[2025-11-24 16:25:01] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:25:04] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
[2025-11-24 16:25:04] [ERRO] [worker-1556931]complete timeout
[2025-11-24 16:25:07] [ERRO] [worker-1556931]http upstream send error: error when dialing 171.214.11.17:80: dialing to the given TCP address timed out
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x174bba0]

goroutine 250916 [running]:
github.com/valyala/fasthttp.(*Request).Reset(
0x0)
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/http.go:1125 +0x60
github.com/valyala/fasthttp.ReleaseRequest
(0x0)
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/client.go:1226 +0x18
github.com/eolinker/apinto/node/http-context.(*HttpContext).FastFinish(0xc00095a288)
/mnt/c/computer/code/aesb-opens/apinto/node/http-context/context.go:436
+0x33d
github.com/eolinker/apinto/node/http-context.(*HttpContext).SendTo.func1.1()

/mnt/c/computer/code/aesb-opens/apinto/node/http-context/context.go:253 +0x56

github.com/eolinker/apinto/node/http-context.(*HttpContext).SendTo.func1
(0xc000748100)

/mnt/c/computer/code/aesb-opens/apinto/node/http-context/context.go:290 +0x865

github.com/valyala/fasthttp.NewStreamReader.func1()
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/stream.go:44 +
0x42
created by
github.com/valyala/fasthttp.NewStreamReader in goroutine 247924
/mnt/c/computer/code/aesb-opens/apinto/vendor/github.com/valyala/fasthttp/stream.go:43
+0x227
[2025-11-24 16:25:24] [WARN] [master-1556754]worker exit:exit status 2
[2025-11-24 16:25:24] [ERRO] [master-1556754]worker process quit error: os: process already finished
[2025-11-24 16:25:25] [INFO] [worker-2230531]worker process start...
[2025-11-24 16:25:25] [INFO] [worker-2230531]time spend:NewWorkerServer:22.96378ms
[2025-11-24 16:25:25] [INFO] [worker-2230531]start worker unix server: /tmp/apinto/apinto.worker-2230531.sock
[2025-11-24 16:25:25] [INFO] [worker-2230531]time spend:port-reqiure unix:/tmp/apinto/apinto.worker-2230531.sock:146.68µs
[2025-11-24 16:25:25] [INFO] [master-1556754]time spend:wait [worker] process start::910.744387ms
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen name: 10.11.55.83:8099
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen name: 10.11.55.83:8099
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen name: 10.11.55.83:8099
[2025-11-24 16:25:25] [INFO] [worker-2230531]mixl listen: 3
[2025-11-24 16:25:25] [INFO] [worker-2230531]index: 0 name: template

Steps to Reproduce

1.我们注册了一个服务
2.多线程调用
3.当这个服务访问超时时,网关会重试,重试过程中会出现空指针
4.最后导致worker exit

Environment

v.22.16

go1.24.10 linux/amd64

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions