harvest exporter Storagegrid exporting additional metrics

[santosh725]

hello,

I run harvest exporter-Storagegrid on K8. Apart from the two default objects provided I am trying to extract additional metrics. but unable to do so. Any help would be greatly appreciated.

here is how I am setting up my configuration.

default.yaml
/opt/harvest/conf/storagegrid # more default.yaml

collector: StorageGrid

schedule:

• data: 5m

objects:
Tenant: tenant.yaml
Prometheus: storagegrid_metrics.yaml
Health: health.yaml

health.yaml

/opt/harvest/conf/storagegrid/11.6.0 # more health.yaml
name: Health
query: api/v3/grid/health
object: storagegrid

schedule:

• data: 1m

counters:

• alarms.critical => alarms_critical
• alarms.major => alarms_major
• alarms.minor => alarms_minor
• nodes.connected => nodes_connected
• nodes.administratively-down => nodes_admin_down
• nodes.unknown => nodes_unknown

export_options:
include_all_labels: true

(hyphen are showing up as bullet points )

[cgrinds]

hi @santosh725 this is caused by the StorageGRID collector not supporting the template option only_cluster_instance. That parameter is needed when there is no way to identify unique instances from the REST response. That's the case with this API since it returns data like this:

{
  "alarms": {
    "critical": 0,
    "major": 0,
    "minor": 0,
    "notice": 0
  },
  "alerts": {
    "critical": 0,
    "major": 0,
    "minor": 0
  },
  "nodes": {
    "connected": 2,
    "administratively-down": 0,
    "unknown": 3
  }
}

Nothing in there about node, cluster, etc. so Harvest was rejecting it. Once I added support for only_cluster_instance to the StorageGRID colellector and fixed the template query (grid/health instead of api/v3/grid/health), I'm able to retrieve the SG metrics like so:

curl -s 'http://localhost:14002/metrics' | grep -Ev "#|metadata_" | sort

sg_health_alarms_critical{cluster="SG5712FP",datacenter="rtp"} 0
sg_health_alarms_major{cluster="SG5712FP",datacenter="rtp"} 0
sg_health_alarms_minor{cluster="SG5712FP",datacenter="rtp"} 0
sg_health_nodes_admin_down{cluster="SG5712FP",datacenter="rtp"} 0
sg_health_nodes_connected{cluster="SG5712FP",datacenter="rtp"} 2
sg_health_nodes_unknown{cluster="SG5712FP",datacenter="rtp"} 3

Here's the template I used,

name:     Health
query:    grid/health
object:   sg_health
api:      v3

schedule:
  - data: 1m

counters:
  - alarms.critical             => alarms_critical
  - alarms.major                => alarms_major
  - alarms.minor                => alarms_minor
  - nodes.connected             => nodes_connected
  - nodes.administratively-down => nodes_admin_down
  - nodes.unknown               => nodes_unknown

only_cluster_instance: true

[santosh725]

Thank you for the response. Is there any service that I would need to restart inside my pod after updating my template ?

[cgrinds]

You need two things for this template to work:

1. the template above
2. a new build since there is a small code change required to make only_cluster_instance work with the StorageGRID collector.

That code is in review and once it's published to nightly, you can pull it from there.

[santosh725]

Thank you for the quick turnaround. Should I use the same tag 23.11.0-1 to pull the new image from your nightly run. here the end goal is to push these metrics to Datadog and then create alerting on it. Currently with the two objects that are provided I am seeing only 39 metrics. I plan to add additional useful templates( i was testing with the above template health.yaml to get alerts, alarms, node status) and create alerting on our system. Hopefully the fix you are talking about will let me add additional templates .

[cgrinds]

You will need to use the nightly tag.

If you have other templates, feel free to share them and I'll check.

[cgrinds]

The code change for PR #2542 has been reviewed and committed. I've kicked off a nightly build that runs everything through CI, and once everything passes, the image will be pushed to nightly in roughly 30m. Let us know how these changes work for you.

[santosh725]

So i did pull the new nightly tagged image. But alas no luck.

[cgrinds]

The nightly build includes the code change but not the template. Did you add the template after pulling and bouncing the poller?

If so, can you share the poller log output? You should see something like this:

2023-12-08T15:00:38-05:00 INF collector/collector.go:575 > Collected Poller=sg-tudor apiMs=40 calcMs=0 collector=StorageGrid:Health exportMs=0 instances=1 instancesExported=1 metrics=6 metricsExported=18 parseMs=0 pluginMs=0 pollMs=40 zBegin=1702065638580

[santosh725]

So I deployed my pod with the nightly tagged image. Then I logged into one of my pod and then updated the default.yaml (/opt/harvest/conf/storagegrid) with the new object. Then I added the corresponding template in this location /opt/harvest/conf/storagegrid/11.6.0 ( not sure if this matters but I am running 11.7 version on my Grid).

name:   Health
query:  grid/health
object: storagegrid
api:    v3

schedule:
  - data: 1m

counters:
  - alarms.critical               => alarms_critical
  - alarms.major                  => alarms_major
  - alarms.minor                  => alarms_minor
  - nodes.connected               => nodes_connected
  - nodes.administratively-down   => nodes_admin_down
  - nodes.unknown                 => nodes_unknown

only_cluster_instance: true

`After updating it these are logs from my pod. All I see are the collectors from the previous objects in my default.yaml.

As for the logs, is there an email I can send them over to ?

[cgrinds]

11.6.0 is fine even when running 11.7.
Please email logs to ng-harvest-files@netapp.com.

[cgrinds]

@santosh725 replied in email to say that #2542, plus the template shared above worked for him. Closing