Connection refused #3322

cutiechi · 2022-08-24T04:09:50Z

cutiechi
Aug 24, 2022

10:30:00.107 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
10:30:00.199 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/55013c63-2b0f-474b-9388-477cc718a49f/2d1075ba-be84-410a-902c-53e5303136bf
10:30:00.283 INFO org.apache.kyuubi.operation.LaunchEngine: Processing admin's query[2d1075ba-be84-410a-902c-53e5303136bf]: INITIALIZED_STATE -> PENDING_STATE, statement: LAUNCH_ENGINE
10:30:00.286 INFO org.apache.kyuubi.session.KyuubiSessionManager: admin's session with SessionHandle [55013c63-2b0f-474b-9388-477cc718a49f] is opened, current opening sessions 1
10:30:00.286 INFO org.apache.kyuubi.operation.LaunchEngine: Processing admin's query[2d1075ba-be84-410a-902c-53e5303136bf]: PENDING_STATE -> RUNNING_STATE, statement: LAUNCH_ENGINE
10:30:00.293 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: Starting
10:30:00.294 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=dol-zookeeper-0.dol-zookeeper.kedacom-project-namespace.svc.cluster.local:12181,dol-zookeeper-1.dol-zookeeper.kedacom-project-namespace.svc.cluster.local:12181,dol-zookeeper-2.dol-zookeeper.kedacom-project-namespace.svc.cluster.local:12181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@157d7e0e
10:30:00.357 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server dol-zookeeper-1.dol-zookeeper.kedacom-project-namespace.svc.cluster.local/10.244.69.107:12181. Will not attempt to authenticate using SASL (unknown error)
10:30:00.359 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to dol-zookeeper-1.dol-zookeeper.kedacom-project-namespace.svc.cluster.local/10.244.69.107:12181, initiating session
10:30:00.361 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server dol-zookeeper-1.dol-zookeeper.kedacom-project-namespace.svc.cluster.local/10.244.69.107:12181, sessionid = 0x182a9a2a4de0001, negotiated timeout = 40000
10:30:00.362 INFO org.apache.curator.framework.state.ConnectionStateManager: State change: CONNECTED
10:30:00.462 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/55013c63-2b0f-474b-9388-477cc718a49f/3689198a-47b2-4ca3-9923-6afce4f24ed1
10:30:00.463 INFO org.apache.kyuubi.ha.client.ServiceDiscovery: Get service instance:10.244.94.199:43871 and version:Some(1.5.0-incubating) under /kyuubi/bigdata/kyuubi_1.5.0-incubating_SERVER_SPARK_SQL/hive/default
10:30:00.466 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [admin:10.244.94.243] SessionHandle [55013c63-2b0f-474b-9388-477cc718a49f] - Starting to wait the launch engine operation finished
10:30:00.481 INFO org.apache.curator.framework.imps.CuratorFrameworkImpl: backgroundOperationsLoop exiting
10:30:00.486 INFO org.apache.zookeeper.ZooKeeper: Session: 0x182a9a2a4de0001 closed
10:30:00.560 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down for session: 0x182a9a2a4de0001
10:30:00.571 INFO org.apache.kyuubi.operation.LaunchEngine: Processing admin's query[2d1075ba-be84-410a-902c-53e5303136bf]: RUNNING_STATE -> ERROR_STATE, statement: LAUNCH_ENGINE, time taken: 0.285 seconds
10:30:00.574 ERROR org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Error executing statement: 
java.util.concurrent.ExecutionException: org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:264)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:105)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:97)
	at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:104)
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:97)
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 13 more

	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
	at java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$waitForEngineLaunched$1(KyuubiSessionImpl.scala:131) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$waitForEngineLaunched$1$adapted(KyuubiSessionImpl.scala:127) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at scala.Option.foreach(Option.scala:407) ~[scala-library-2.12.15.jar:?]
	at org.apache.kyuubi.session.KyuubiSessionImpl.waitForEngineLaunched(KyuubiSessionImpl.scala:127) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.runOperation(KyuubiSessionImpl.scala:117) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.AbstractSession.$anonfun$executeStatement$1(AbstractSession.scala:122) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.AbstractSession.withAcquireRelease(AbstractSession.scala:75) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.AbstractSession.executeStatement(AbstractSession.scala:119) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.service.AbstractBackendService.executeStatement(AbstractBackendService.scala:66) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$executeStatement(KyuubiServer.scala:142) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$executeStatement$1(BackendServiceMetric.scala:62) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:111) ~[kyuubi-metrics_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.server.BackendServiceMetric.executeStatement(BackendServiceMetric.scala:62) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.server.BackendServiceMetric.executeStatement$(BackendServiceMetric.scala:55) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.server.KyuubiServer$$anon$1.executeStatement(KyuubiServer.scala:142) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.service.TFrontendService.ExecuteStatement(TFrontendService.scala:232) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) ~[hive-service-rpc-3.1.2.jar:3.1.2]
	at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) ~[hive-service-rpc-3.1.2.jar:3.1.2]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:36) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) ~[libthrift-0.9.3.jar:0.9.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: org.apache.kyuubi.KyuubiSQLException: Error operating UNKNOWN_OPERATION: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266)
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:264)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:105)
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:97)
	at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:104)
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:97)
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
	... 13 more

	at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69) ~[kyuubi-common_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:81) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.operation.KyuubiOperation$$anonfun$onError$1.applyOrElse(KyuubiOperation.scala:62) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38) ~[scala-library-2.12.15.jar:?]
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:54) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	... 3 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:264) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:105) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:97) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:104) ~[kyuubi-ha_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:97) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	... 3 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
	at java.net.Socket.connect(Socket.java:609) ~[?:?]
	at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:266) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) ~[libthrift-0.9.3.jar:0.9.3]
	at org.apache.kyuubi.client.KyuubiSyncThriftClient$.createClient(KyuubiSyncThriftClient.scala:264) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:105) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1$adapted(KyuubiSessionImpl.scala:97) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.ha.client.ZooKeeperClientProvider$.withZkClient(ZooKeeperClientProvider.scala:104) ~[kyuubi-ha_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:97) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:51) ~[kyuubi-server_2.12-1.5.0-incubating.jar:1.5.0-incubating]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	... 3 more
10:30:00.668 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [55013c63-2b0f-474b-9388-477cc718a49f]
10:30:00.671 INFO org.apache.kyuubi.session.KyuubiSessionManager: SessionHandle [55013c63-2b0f-474b-9388-477cc718a49f] is closed, current opening sessions 0
10:30:00.673 INFO org.apache.kyuubi.operation.LaunchEngine: Processing admin's query[2d1075ba-be84-410a-902c-53e5303136bf]: ERROR_STATE -> CLOSED_STATE, statement: LAUNCH_ENGINE
10:30:00.675 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [55013c63-2b0f-474b-9388-477cc718a49f]

lianneli · 2022-09-11T07:42:24Z

lianneli
Sep 11, 2022

I find it would happen when the spark driver recorded in zk is un reachable. It may because of accessable of network, or the driver was dead already.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Connection refused #3322

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Connection refused #3322

Uh oh!

cutiechi Aug 24, 2022

Replies: 1 comment

Uh oh!

lianneli Sep 11, 2022

cutiechi
Aug 24, 2022

lianneli
Sep 11, 2022