KEB memory consumption

[jaroslaw-pieszka]

During long running tests we noticed that memory consumption increases over time.
We reduced the footprint and flattened the trend curve but still there increase occurs.

When using pprof and massive provisioning on k3d cluster you can see that most of allocations occur in libraries e.g. in prometheus client:

Initial memory:

After massive provisioning (10000 provisionings)

Investigate if there is a way to reduce memory used by prometheus client.

Consider that:

• we can expose the same metrics with lower footprint, smarter usage of libraries (removal of old cache keys, regeneration of maps etc)
• we cannot avoid it, the footprint increases then reaches plateau and there is nothing to worry about
• we keep metrics without real need, there is no need to expose the same samples for subsequent scrapings
• we can use db calls instead of memory cache if we accept trade off.
• other possibilities