Improve performance when querying from Grafana

[jnywong]

Context

Depending on the volume of data requested, the performance of the backend when interfacing with Grafana can be flaky.

For example, fetching the same visualisation twice with the same time range and variable selections seems to re-trigger a request and response cycle, when ideally the response should be cached.

Proposal

We have identified that the compute usage query, with a time resolution/step of 5m, is the bottleneck.

• ensure the prometheus compute usage query is cached and retrievable from Grafana
• configure a prometheus recording rule to store results from the compute usage query so that it's readily available for fetching from Grafana

Definition of Done

• Deploy cost monitoring system to VEDA cluster
  • [nasa-veda] Deploy cost monitoring system  infrastructure#6814
• Dashboard panels take at most 10 seconds to load, especially when readjusting time ranges and variable selectors

[jnywong]

Closed by #53

Follow up issue #54