Support Cluster Size Presets

[williamtrelawny]

It would be useful to have presets established for users to deploy clusters based on our generally accepted sizing guides, which exist both in our documentation as well as additional internal resources as well. Note that the below suggestions deviate from these guides because of the resource-efficient nature of containers. The original guides were written for VM deployments.

Presets would adjust the following values for both Graylog and Datanode in the Chart. MongoDB can be excluded for the time being as our Chart currently only supports a 3-pod deployment which is suitable for almost all use cases.

Replicas:

• graylog.replicas
• datanode.replicas

CPU:

• graylog.custom.resources.limits.cpu
• datanode.custom.resources.limits.cpu

Memory:

• graylog.config.serverJavaOpts
• datanode.config.javaOpts
• graylog.custom.resources.limits.memory
• datanode.custom.resources.limits.memory

I also think we should use the common "t-shirt" sizing terminology instead of the "core/conventional" terms used in docs, mainly because docs doesn't get as granular as our internal sizing guide does, so we'd need more terms anyway.

For example (components are represented as a tuple of the form (n of replicas, CPU/server, RAM/server [GB]):

Preset	Graylog	Datanode	Note
(default)	(2, 1, 1)	(3, 0.5, 3.5)	Default values if no preset is used
poc	(1, 1, 1)	(1, 0.5, 3.5)	For POC use in testing on single-node non-prod environments
xs	(1, 2, 4)	(1, 1, 4)	For <10GB/day log volume
small	(1, 2, 4)	(2, 2, 8)	For 10-20GB/day log volume
medium	(2, 4, 8)	(2, 4, 16)	For 20-50GB/day log volume
large	(2, 4, 8)	(3, 4, 16)	For 50-100GB/day log volume
xl	(3, 4, 8)	(4, 4, 16)	For 100-300GB/day log volume
xxl	(4, 4, 8)	(6, 4, 16)	For 300-500GB/day log volume

These will all need testing of course, which is going to require a lot of time.

[monrax]

just checking: are we sure we want limits

{graylog|datanode}.custom.resources.limits.cpu

instead of requests

{graylog|datanode}.custom.resources.requests.cpu

for both CPU and memory resources?

Note

requests: you must have at least this much cowbell to run the application.

limits: if the application reaches this point, performance/availability will suffer

• cpu: if the container tries to use more than the specified cpu limit, k8s throttles it.
• memory: if the container tries to use more than the specified memory limit, k8s kills the container (dropping any ingestion, parsing, etc. being perfomed at that time).

[williamtrelawny]

We could do both but in my mind limits is more important than requests, at least for configurability. I think the requests values would be the same regardless of the preset used, right?

[monrax]

From my perspective, when I run

helm install graylog ./graylog --set preset=small

I kinda expect a small cluster that fits in my small setup, not a cluster that will be more limited during peak processing times.

For example, let's say I wanna try Graylog in my cluster but it only has 1 vCPU and 2 GB of memory left in the one k8s node available. If I run helm install with the default values for replicas, cpu, and memory the graylog pod will be stuck at "Pending" until some other workload is removed or more resources are added. Now, a demo/poc cluster can have 1 vCPU + 1 GB of memory since it won't be ingesting/processing that much data anyway. So, if I have the ability to run helm install with the poc preset, I can get the graylog pod scheduled there instead of leaving it stuck at "Pending".

As you mentioned, this also applies to the JVM options:

• heap request: -Xms (should be higher than container requests to also account for JVM memory overhead)
• heap limit: -Xmx (should be lower than container limit to also account for JVM memory overhead)

All presets should be equal to or above the absolute minimum system requirements if we want the application to run. I think of k8s container requirements as the prescribed requirements to allow a given configuration to run, with all tradeoffs considered (like the poc case above, where we know the application runs, just not as performant as a normal deployment should, but good enough for the use case).

Other than playing nice with the other workloads and not hogging the system resources during peak processing (protecting node stability), I'm not sure of the use case for limits. However, assuming we can live with the possibility of false failures (and downtime data gaps) due to spikes in memory usage, maintaining node stability might still be a good enough reason to set container/JVM limits. Perhaps we should set both requests and limits with a given preset? What would those be?

[williamtrelawny]

Closing since #25 was merged 🚀