Is there any way set multiple engine.type with a bind.host and bind.port? #6210

ysmintor · 2024-03-27T02:52:54Z

ysmintor
Mar 27, 2024

I am a newbie in Kyuubi, I am using kyuubi-default.conf with kyuubi.frontend.bind.host, kyuubi.frontend.thrift.binary.bind.port 10009, kyuubi.engine. type=SPARK_SQL, but I also want to use FLINK_SQL. I thought adding #kyuubi.engine.type=FLINK_SQL to the beeline connection URL would work, but it doesn't work, prompting that the port is already bound (which I assume is the default SPARK_SQL usage).

2024-03-27 10:46:32.710 INFO main org.apache.flink.yarn.cli.FlinkYarnSessionCli: Found Yarn properties file under /tmp/.yarn-properties-root.
2024-03-27 10:46:32.756 WARN main org.apache.flink.core.plugin.PluginConfig: The plugins directory [plugins] does not exist.
2024-03-27 10:46:32.796 INFO main org.apache.flink.table.gateway.service.context.DefaultContext: Execution config: {execution.savepoint.ignore-unclaimed-state=false, execution.attached=true, execution.savepoint-restore-mode=NO_CLAIM, execution.shutdown-on-attached-exit=false, pipeline.jars=[], pipeline.classpaths=[], $internal.deployment.config-dir=/opt/flink/conf}
2024-03-27 10:46:32.814 INFO main org.apache.kyuubi.util.ThreadUtils: FlinkSQLSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 60000 ms
2024-03-27 10:46:32.818 INFO main org.apache.kyuubi.engine.flink.operation.FlinkSQLOperationManager: Service[FlinkSQLOperationManager] is initialized.
2024-03-27 10:46:32.820 INFO main org.apache.kyuubi.engine.flink.session.FlinkSQLSessionManager: Service[FlinkSQLSessionManager] is initialized.
2024-03-27 10:46:32.820 INFO main org.apache.kyuubi.engine.flink.FlinkSQLBackendService: Service[FlinkSQLBackendService] is initialized.
2024-03-27 10:46:32.868 ERROR main org.apache.kyuubi.engine.flink.FlinkTBinaryFrontendService: java.net.BindException: Address already in use
2024-03-27 10:46:32.868 ERROR main org.apache.kyuubi.engine.flink.FlinkSQLEngine: Fatal error occurs, thus stopping the engines
org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on /10.10.10.187:10009.
        at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:122)
        at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
        at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
        at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine.initialize(FlinkSQLEngine.scala:46)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.$anonfun$startEngine$2(FlinkSQLEngine.scala:125)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.$anonfun$startEngine$2$adapted(FlinkSQLEngine.scala:124)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.startEngine(FlinkSQLEngine.scala:124)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.main(FlinkSQLEngine.scala:103)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine.main(FlinkSQLEngine.scala)
Caused by: java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:555)
        at java.base/sun.nio.ch.Net.bind(Net.java:544)
        at java.base/sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:648)
        at java.base/java.net.ServerSocket.bind(ServerSocket.java:388)
        at java.base/java.net.ServerSocket.<init>(ServerSocket.java:274)
        at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:100)
        ... 14 more
Error: org.apache.kyuubi.KyuubiSQLException:
The engine application has been terminated. Please check the engine log.
ApplicationInfo: (
name -> null,
state -> NOT_FOUND,
url -> null,
id -> null,
error -> null
)

        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$9(EngineRef.scala:282)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$9$adapted(EngineRef.scala:271)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$8(EngineRef.scala:271)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$8$adapted(EngineRef.scala:260)
        at scala.Option.foreach(Option.scala:407)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:260)
        at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:166)
        at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:178)
        at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:183)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:317)
        at scala.Option.getOrElse(Option.scala:189)
        at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:317)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:159)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:133)
        at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:133)
        at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49)
        at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:133)
        at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:60)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiSQLException: org.apache.kyuubi.KyuubiException: Failed to initialize frontend service on /10.133.151.187:10009.
        at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:122)
        at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1(CompositeService.scala:40)
        at org.apache.kyuubi.service.CompositeService.$anonfun$initialize$1$adapted(CompositeService.scala:40)
        at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:58)
        at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:51)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.kyuubi.service.CompositeService.initialize(CompositeService.scala:40)
        at org.apache.kyuubi.service.Serverable.initialize(Serverable.scala:48)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine.initialize(FlinkSQLEngine.scala:46)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.$anonfun$startEngine$2(FlinkSQLEngine.scala:125)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.$anonfun$startEngine$2$adapted(FlinkSQLEngine.scala:124)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.startEngine(FlinkSQLEngine.scala:124)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine$.main(FlinkSQLEngine.scala:103)
        at org.apache.kyuubi.engine.flink.FlinkSQLEngine.main(FlinkSQLEngine.scala)
Caused by: java.net.BindException: Address already in use
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:555)
        at java.base/sun.nio.ch.Net.bind(Net.java:544)
        at java.base/sun.nio.ch.NioSocketImpl.bind(NioSocketImpl.java:648)
        at java.base/java.net.ServerSocket.bind(ServerSocket.java:388)
        at java.base/java.net.ServerSocket.<init>(ServerSocket.java:274)
        at org.apache.kyuubi.service.TBinaryFrontendService.initialize(TBinaryFrontendService.scala:100)
 See more: /opt/kyuubi/work/kentyao/kyuubi-flink-sql-engine.log.5
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
        at org.apache.kyuubi.engine.ProcBuilder.$anonfun$start$1(ProcBuilder.scala:234)
        at java.base/java.lang.Thread.run(Thread.java:842)
.
FYI: The last 10 line(s) of log are:
2024-03-27 10:46:32.693 INFO main org.apache.flink.configuration.GlobalConfiguration: Loading configuration property: state.backend, filesystem
2024-03-27 10:46:32.693 INFO main org.apache.flink.configuration.GlobalConfiguration: Loading configuration property: high-availability.zookeeper.client.acl, open
2024-03-27 10:46:32.693 INFO main org.apache.flink.configuration.GlobalConfiguration: Loading configuration property: state.checkpoints.dir, hdfs://mycluster/flink/checkpoints
2024-03-27 10:46:32.710 INFO main org.apache.flink.yarn.cli.FlinkYarnSessionCli: Found Yarn properties file under /tmp/.yarn-properties-root.
2024-03-27 10:46:32.756 WARN main org.apache.flink.core.plugin.PluginConfig: The plugins directory [plugins] does not exist.
2024-03-27 10:46:32.796 INFO main org.apache.flink.table.gateway.service.context.DefaultContext: Execution config: {execution.savepoint.ignore-unclaimed-state=false, execution.attached=true, execution.savepoint-restore-mode=NO_CLAIM, execution.shutdown-on-attached-exit=false, pipeline.jars=[], pipeline.classpaths=[], $internal.deployment.config-dir=/opt/flink/conf}
2024-03-27 10:46:32.814 INFO main org.apache.kyuubi.util.ThreadUtils: FlinkSQLSessionManager-exec-pool: pool size: 100, wait queue size: 100, thread keepalive time: 60000 ms
2024-03-27 10:46:32.818 INFO main org.apache.kyuubi.engine.flink.operation.FlinkSQLOperationManager: Service[FlinkSQLOperationManager] is initialized.
2024-03-27 10:46:32.820 INFO main org.apache.kyuubi.engine.flink.session.FlinkSQLSessionManager: Service[FlinkSQLSessionManager] is initialized.
2024-03-27 10:46:32.820 INFO main org.apache.kyuubi.engine.flink.FlinkSQLBackendService: Service[FlinkSQLBackendService] is initialized.
        at org.apache.kyuubi.KyuubiSQLException$.apply(KyuubiSQLException.scala:69)
        at org.apache.kyuubi.engine.ProcBuilder.getError(ProcBuilder.scala:281)
        at org.apache.kyuubi.engine.ProcBuilder.getError$(ProcBuilder.scala:270)
        at org.apache.kyuubi.engine.flink.FlinkProcessBuilder.getError(FlinkProcessBuilder.scala:39)
        ... 25 more (state=,code=0)

pan3793 · 2024-03-27T04:09:22Z

pan3793
Mar 27, 2024
Collaborator

Should be a bug, the kyuubi.frontend.thrift.binary.bind.port should be filtered out before passing args to Flink engines.

5 replies

pan3793 Mar 27, 2024
Collaborator

As a temporary workaround, I think you can unset kyuubi.frontend.thrift.binary.bind.port in kyuubi-defaults.conf to let it fallback to the default value, for Server, is 10009, for engine, is 0

ysmintor Mar 27, 2024
Author

Do you mean "for engine is 0 " is to set kyuubi.engine.type=0? @pan3793

pan3793 Mar 27, 2024
Collaborator

I mean the port

pan3793 Mar 27, 2024
Collaborator

0 means finding a random available port to bind

ysmintor Mar 27, 2024
Author

@pan3793 But previously I need to set kyuubi.frontend.thrift.binary.bind.port=18009. but unset kyuubi.frontend.thrift.binary.bind.port , the default is 10009. I do need a specific port to bind. So is there a way not a random port to bind? or could you show the related configuration parameters here?

ysmintor · 2024-03-27T05:40:55Z

ysmintor
Mar 27, 2024
Author

I have found a similar bug there #5957.
So currently I tried to replace kyuubi.frontend.thrift.binary.bind.port to deprecated kyuubi.frontend.bind.port.

1 reply

pan3793 Mar 27, 2024
Collaborator

won't work, please re-read my previous reply

ysmintor · 2024-03-28T03:34:52Z

ysmintor
Mar 28, 2024
Author

Currently, in kyuubi 1.9.0, I can only solve it like the post in #5957 (comment) with yarn application mode. However, yarn session mode will encounter errors which I think is a bug.

0 replies

Is there any way set multiple engine.type with a bind.host and bind.port? #6210

Uh oh!

ysmintor Mar 27, 2024

Replies: 3 comments · 6 replies

Uh oh!

pan3793 Mar 27, 2024 Collaborator

Uh oh!

pan3793 Mar 27, 2024 Collaborator

Uh oh!

ysmintor Mar 27, 2024 Author

Uh oh!

pan3793 Mar 27, 2024 Collaborator

Uh oh!

pan3793 Mar 27, 2024 Collaborator

Uh oh!

Uh oh!

ysmintor Mar 27, 2024 Author

Uh oh!

ysmintor Mar 27, 2024 Author

Uh oh!

pan3793 Mar 27, 2024 Collaborator

Uh oh!

ysmintor Mar 28, 2024 Author

ysmintor
Mar 27, 2024

Replies: 3 comments 6 replies

pan3793
Mar 27, 2024
Collaborator

pan3793 Mar 27, 2024
Collaborator

ysmintor Mar 27, 2024
Author

pan3793 Mar 27, 2024
Collaborator

pan3793 Mar 27, 2024
Collaborator

ysmintor Mar 27, 2024
Author

ysmintor
Mar 27, 2024
Author

pan3793 Mar 27, 2024
Collaborator

ysmintor
Mar 28, 2024
Author