Collect GPU usage metrics with prometheus

We use [prometheus node exporter](https://github.com/prometheus/node_exporter),
deployed as part of our prometheus chart, to collect metrics about
CPU and memory usage.

This deploys NVIDIA's [dcgm-exporter](https://github.com/NVIDIA/dcgm-exporter)
which collects information about GPU usage.

As we work towards more cost monitoring and usage monitoring,
collecting this information should allow us to help users get more
bang for the buck from their GPU use. Since we only collect information
after the exporters are deployed, this starts the information collection
process even if it's not directly visible to end users.

Works towards https://2i2c.productboard.com/entity-detail/features/30046512,
initially requested as part of https://2i2c.freshdesk.com/a/tickets/2545.

Author: yuvipanda

helm-charts/support/Chart.yaml

@@ -59,3 +59,9 @@ dependencies:
     version: "0.0.1-0.dev.git.72.hadbe1d4"
     repository: https://2i2c.org/gcp-filestore-backups
     condition: gcpFilestoreBackups.enabled
+
+  # Provide metrics about GPU usage
+  # https://github.com/NVIDIA/dcgm-exporter
+  - name: dcgm-exporter
+    version: 3.6.1
+    repository: https://nvidia.github.io/dcgm-exporter/helm-charts

helm-charts/support/values.schema.yaml

@@ -42,6 +42,9 @@ properties:
   global:
     type: object
     additionalProperties: true
+  dcgm-exporter:
+    type: object
+    additionalProperties: true
 
   # These provide values for objects we create, so we validate their schema
   # to the best of our ability.

helm-charts/support/values.yaml

@@ -494,6 +494,20 @@ cryptnono:
 aws-ce-grafana-backend:
   enabled: false
 
+dcgm-exporter:
+  serviceMonitor:
+    enabled: false
+  podAnnotations:
+    prometheus.io/path: "/metrics"
+    prometheus.io/port: "12121"
+    prometheus.io/scrape: "true"
+  tolerations:
+  - key: nvidia.com/gpu
+    operator: Equal
+    value: present
+    effect: NoSchedule
+
+
 # Configuration of templates provided directly by this chart
 # -------------------------------------------------------------------------------
 #