Determine minimum and maximum resources need per bitcoind pod

[josibake]

In order to correctly provision a cluster, we need to know total number of tanks and how many resources are required per tank. As a starting point, 0.5 ~ 1 vcpu per tank and 1 gb ram per tank seems to be reasonable. I haven't validated this but we may be able to give the cluster better auto-scaling hints if we directly specify these resource limits on the pod, e.g. each pod requests a minimum amount of CPU.

Deliverable

1. Validated numbers for tank resources (probably best obtained through a monitoring tool)
2. Document detailing cluster provisioning and specifying a formula for determining the total size needed for the cluster
3. Potentially, updates to the deployment resources, specifying the limits explicitly

[pinheadmz]

It may depend on the scenario as well. I had a 100 vCPU cluster with only 64 tanks + logging and started getting "critical CPU level" warnings from K8s at around 70 tx/s. However at the lightning attackathon last april I had 100 tanks + lnd + circuitbreaker + simln humming along at around 50% CPU levels.

[josibake]

Yeah, I think starting with minimums would help with the autoscaling, e.g., what is the minimum CPU / RAM a "typical" tank will need. I think there is also a way to take existing pods and autoscale based on an increased workload in already scheduled pods, which would be relevant for scenarios that are more intensive than the "normal" usage, but that might be overkill for us at this point. I'm mainly looking for a way to have the cluster bark at you before you start doing anything if you're staring with too little resources.

[adamjonas]

ref #456

[josibake]

Given that this can be easily specified in the yaml configs, going to close this for now. As @pinheadmz mentioned, a lot of this will be scenario dependent, and I think what we currently have for our defaults seems reasonable.