Skip to content

provider优雅停机,call需要30秒,consumer10秒后主动中断channel,导致优雅停机失败 #2952

@hjq2016

Description

@hjq2016
  • I have searched the issues of this repository and believe that this is not a duplicate.
  • I have checked the FAQ of this repository and believe that this is not a duplicate.

Environment

  • Dubbo version: 2.6.5
  • Operating System version: CentOS6.9
  • Java version: 1.8

Steps to reproduce this issue

  1. provider server set JVM argument -Ddubbo.service.shutdown.wait=50000
  2. consumer call provider其中一个Service,需要30秒才能返回结果
    3.after call 5秒后,执行 kill pid,将provider优雅停机
    4.consumer收到register center发送的provider下线消息,主动关闭netty的channel连接
    5.provider的DubboServerHandler线程30秒后想返回结果,发现channel关闭,抛出异常;
    6.consumer端配置了60秒超时,得不到返回,抛出timeOut异常。导致请求异常结束

Pls. provide [GitHub address] to reproduce this issue.

Expected Result

What do you expected from the above steps?
期待结果是Service跑完30秒,正常返回,然后优雅停机。consumer端的后续请求切换到可用Service上

Actual Result

What actually happens?

If there is an exception, please attach the exception trace:

com.alibaba.dubbo.remoting.RemotingException: Failed to send message Response [id=213, version=2.0.2, status=20, event=false, error=null, result=RpcResult [返回的content], exception=null]] to /IP:65417, cause: null
	at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:110) ~[dubbo-2.6.5.jar:2.6.5]
	at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:53) ~[dubbo-2.6.5.jar:2.6.5]
	at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:174) ~[dubbo-2.6.5.jar:2.6.5]
	at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:51) ~[dubbo-2.6.5.jar:2.6.5]
	at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:57) [dubbo-2.6.5.jar:2.6.5]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_45]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_45]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: java.nio.channels.ClosedChannelException
	at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:643) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:370) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.channel.Channels.write(Channels.java:632) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) ~[netty-3.2.5.Final.jar:?]
	at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.writeRequested(NettyHandler.java:98) ~[dubbo-2.6.5.jar:2.6.5]
	at org.jboss.netty.channel.Channels.write(Channels.java:611) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.channel.Channels.write(Channels.java:578) ~[netty-3.2.5.Final.jar:?]
	at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) ~[netty-3.2.5.Final.jar:?]
	at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:100) ~[dubbo-2.6.5.jar:2.6.5]
	... 7 more

最后发现问题,是consumer启动参数没有-Ddubbo.service.shutdown.wait=50000。当consumer端收到provider下线消息,主动触发一个优雅关闭链接,默认是10秒。加上参数后,就解决上面的问题。

但是provide的优雅关机居然需要comsumer配置,这可能是不合理的。所以看dubbo开发组怎么解决这个问题。
更多provider日志:
16:19:16.875 [DubboShutdownHook] INFO com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol(432) - [DUBBO] Close dubbo server: /127.0.0.1:20881, dubbo version: 2.6.5, current host: 127.0.0.1

16:19:26.619 [New I/O server worker #1-1] WARN com.alibaba.dubbo.remoting.transport.AbstractServer(205) - [DUBBO] All clients has discontected from /127.0.0.1:20881. You can graceful shutdown now., dubbo version: 2.6.5, current host: 127.0.0.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    type/proposalEverything you want Dubbo have

    Type

    No type

    Projects

    Status

    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions