Load Testing on K8s

[ranchodeluxe]

Depends On

#3

Goal and Background

Since we have autoscaling set up, we now want to come up with some more "real" workflows against tipg/titler/stac to test load and autoscaling and providing metrics about this on k8s. We need to really understand the workflows and the special cases around them that might give us different problematic autoscaling. We have a few existing examples of load testing these services from VEDA and from Aimee's examples of zarr visualization:

1. https://github.com/NASA-IMPACT/veda-load-test
2. https://github.com/developmentseed/tile-benchmarking/blob/main/03-e2e/caching-strategies-report.ipynb

AC

1. review existing load testing and understand the workflows we want to emulate
2. pick some datasets (good real world examples), load them
3. and write some load tests (let's use artillery) that are less naive than hey tool (which is still good to use here and there). NOTE: we'll probably have to think here about how to monitor the API server on Control Plane node and make sure our metrics tooling is scaling itself properly
4. do some of kind of write up (part of this is documentation for eoapi-k8s users) to talk about the challenges we saw and the metrics we got