Backend Stream taking up a lot of resources

[yatharthranjan]

Just a question. Is this kind of behaviour normal for streams ?

Here is a screen grab of htop

[yatharthranjan]

Could be because of this setting - max.request.size: 3500042
Maybe will be fixed by the reservoir sampling aggregator when a new version is released.

Although this should not relate with the high CPU consumption shown above

[yatharthranjan]

On rosalind its even worse. The CPU consumption goes upto 1200%

[yatharthranjan]

Also a screen grab of top from within the container -

[yatharthranjan]

I restricted the CPU usage using docker-compose by adding the following parameter to streams service in docker-compose file (suggested by @afolarin ) -

cpus: 2.0

This limits the CPU utilization of the container to 200%. That works fine for rosalind. Don't know how it affects the streams app though. I see a lot of rebalancing. Will wait and see if the streams start failing

[blootsvoets]

Good temporary fix. My hunch is that CPU usage is dominated by garbage collection, indeed because the streams take too much memory. Once the new sampling method in RADAR-Backend is deployed, memory requirements (11.6 GB according to htop!) and cpu usage should go down.

[yatharthranjan]

We have a lot of memory on rosalind. Do you think if we increase the Min GC size then the CPU usage will go down?

[yatharthranjan]

Reopening this although the cpu utilization has been dropped to half but it is still not ideal. Getting around 600-700% CPU utilization for now compared to 1200% previously.
Adding the GC stats for streams JVM

[yatharthranjan]

Here are the CPU profiling results for the streams app -
Looks like jackson lib is using a lot of CPU.

[blootsvoets]

Increasing commit.interval.ms should help then, as this controls how often a state store is flushed (to JSON). Perhaps this value should depend on the time window: if 1 week, not more than 1 flush per hour should be needed. For 10 min, perhaps 1 min would suffice? Note that for the integration tests, we want this value low because it will take forever to complete otherwise.