Logstash - JVM Config

[theduke]

Hi there.

I'm quite sceptical about the Logstash performance results.

I'm assuming it's a JVM config issue. Logstash can become very slow without enough memory.

Since it seems you are using the default config from apt, I think the memory will be limited to 1GB.

While the high memory usage is certainly a drawback of Logstash, no-one runs it in production with such a limited capacity.

For a fairer comparison, you'll probably want to give it at least 4GB of memory.

Some info here:
https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

[binarylogic]

Thanks @theduke! We'll try that and see if that helps. Once we collect new data we'll report back.

It is interesting to me that it would require more memory for such simple use cases. Especially the tcp_to_blackhole_performance test and tcp_to_tcp_performance test.

[theduke]

That's the JVM for you. Fast but very memory hungry ;)

I haven't used Logstash in a while, but I remember running into severe performance issues even with very low message counts. Oonly to discover that the problem was insufficient memory. If the engine doesn't have enough, it can slow to a crawl.

[MOZGIII]

JVM tries to preallocate and take ownership of extra memory, and it doesn't release it often. Vector is also designed to take advantage of all the system memory, however it uses a different design for memory usage and releases memory much better.

Under these circumstances, for the high-load use case, memory usage stats the way we measure it might not be representative (i.e. fair) - we have to max out the memory demand somehow to balance out the runtime differences, but then the readings that we'll get is simply that all system memory is consumed 📈 That said, some people might still find those stats useful, especially for scenarios where the load is not too high and the agent deployment variant is used on a not-so-big VM.

So, depending on the scenario we want to test, the JVM config will be different. Using the defaults provided out of the box might be the most useful way to compare. We run test harness on a reasonable sized VMs (c5.large and similar), so there is system memory to take if there's a demand for it.