Containers startup throttling

[chary1112004]

This issue happens for us while deploying quarkus application to kubernetes (eks). It impacts while we upgrade kubernetes while moving pods to new worker nodes and services in many tenants keep crashing as it don't have resource in startup time

When service start, it consumes a lot of cpu ~1000m ~ 1CPU. With this consume then if 1 node contains 4 CPUs then there will be only 4 services could start at the time and other keep crashing and retry until there is free space cpu to reserve.

data-service-86995fcbc-vhg4w             1142m        288Mi

After service in running time, cpu is quite low ~ 25m

data-service-86995fcbc-vhg4w             25m          308Mi

Any solution to resolve this issue would be very much appreciated.

Thanks!

[quarkus-bot[bot]]

/cc @Sgitario (kubernetes), @geoand (kubernetes), @iocanel (kubernetes)

[maxandersen]

Is this in jvm or native mode?

Have you profiled what parts of the startup that is taking time?

Both core quarkus and User code is involved so without having some profile data it's hard to know where the issue is.

[geoand]

In addition to this, it's known that the JVM does a lot of work when the process first boots.

Have you read this and this already?

[chary1112004]

@maxandersen we are running in jvm mode.

We just did profile in local, not into container part. I will try to remove all code belong to user code and just keep quarkus code and give feedback to you.

Another point that is we have installed top command inside container to verify which process is consuming cpu. Then it is only java that is consuming 200% .

@geoand thanks for the links. I will check about that.