kyuubi server out of memory

[stgztsw]

kyuubi server 设置了4G的内存，使用了几天之后，kyuubi server 出现了out of memory的问题，请问是内存设置不够吗？还有内存有泄漏的问题

Exception in thread "KyuubiTBinaryFrontendHandler-Pool: Thread-178507" java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:719)
        at org.apache.commons.io.monitor.FileAlterationMonitor.start(FileAlterationMonitor.java:141)
        at com.tangshiwei.kyuubi.CustomKyuubiAuth.<init>(CustomKyuubiAuth.java:70)
        at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.kyuubi.service.authentication.AuthenticationProviderFactory$.getAuthenticationProviderForServer(AuthenticationProviderFactory.scala:64)
        at org.apache.kyuubi.service.authentication.AuthenticationProviderFactory$.getAuthenticationProvider(AuthenticationProviderFactory.scala:35)
        at org.apache.kyuubi.service.authentication.PlainSASLHelper$PlainServerCallbackHandler.handle(PlainSASLHelper.scala:68)
        at org.apache.kyuubi.service.authentication.PlainSASLServer.evaluateResponse(PlainSASLServer.scala:74)
        at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283)
        at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:719)
        at org.apache.commons.io.monitor.FileAlterationMonitor.start(FileAlterationMonitor.java:141)
        at com.tangshiwei.kyuubi.CustomKyuubiAuth.<init>(CustomKyuubiAuth.java:70)
        at sun.reflect.GeneratedConstructorAccessor13.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.Class.newInstance(Class.java:442)
        at org.apache.kyuubi.service.authentication.AuthenticationProviderFactory$.getAuthenticationProviderForServer(AuthenticationProviderFactory.scala:64)
        at org.apache.kyuubi.service.authentication.AuthenticationProviderFactory$.getAuthenticationProvider(AuthenticationProviderFactory.scala:35)
        at org.apache.kyuubi.service.authentication.PlainSASLHelper$PlainServerCallbackHandler.handle(PlainSASLHelper.scala:68)
        at org.apache.kyuubi.service.authentication.PlainSASLServer.evaluateResponse(PlainSASLServer.scala:74)

[pan3793]

Which version you are using?

[stgztsw]

Which version you are using?

1.5.1

[ulysses-you]

Not sure this oom is caused by the code stack which you put. It will be helpful if you can provide the heap dump

[pan3793]

There is no known memory leak issue in v1.5.1.

java.lang.OutOfMemoryError: unable to create new native thread

The error message indicates that you are run out of open files, please use lsof to check which and how many files are opened by the kyuubi process.

[akizminet]

I had the same issue when starting many spark engines in yarn-client mode.
Increase max number of processes in /etc/security/limits.conf fixed it.

kyuubi         soft    nproc            16384
kyuubi         hard    nproc            16384

[kmsantanu]

can you please tell me what is kyuubi here.
is it process name or user name where you install kyubi

[akizminet]

@kmsantanu It is the user name for running kyuubi server