[NABox] Modify Collection Interval for ZapiPerf Volume Performance Template

[rahulguptajss]

Note: Below steps are applicable for NABox

Prerequisite: ssh to NABox via root

Step 1: Create custom_mycustom_conf.yaml

Navigate to the zapiperf directory:

cd /opt/harvest2-conf/conf/zapiperf

Create custom_mycustom_conf.yaml:

vi custom_mycustom_conf.yaml

Update the file with the following content:

objects:
  Volume:
    - custom_volume.yaml

Step 2: Create custom_volume.yaml

Navigate to the cdot/9.8.0 directory:

cd /opt/harvest2-conf/conf/zapiperf/cdot/9.8.0

Create custom_volume.yaml:

vi custom_volume.yaml

Update the file with the following content:

schedule:
  - data: 5s

Step 3: Change Prometheus Collection Interval for Prometheus Volume Metrics

Note: NABox upgrade will overwrite this file.

Navigate to the Prometheus directory:

cd /usr/local/nabox/files/prometheus

Backup prometheus.yml:

cp prometheus.yml prometheus_bkup.yml

Open prometheus.yml in a text editor:

vi prometheus.yml

Update the file with the following content:

# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:
  - "alert_rules.yml"
  - "ems_alert_rules.yml"

# Alert Manager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'alertmanager:9093'

scrape_configs:
- job_name: cadvisor
  scrape_interval: 15s
  static_configs:
  - targets:
    - cadvisor:8080

- job_name: node-exporter
  scrape_interval: 15s
  static_configs:
  - targets:
    - node-exporter:9100

- job_name: "prometheus_fast"
  scrape_interval: 5s
  metric_relabel_configs:
    - source_labels: [__name__]
      regex: 'volume_.*'  # Keep metrics that start with 'volume_'
      action: keep
  file_sd_configs:
  - files:
    - '/scrapers/scrapers.yml'

- job_name: "harvest2"
  metric_relabel_configs:
    - source_labels: [__name__]
      regex: 'volume_.*'  # Drop metrics that start with 'volume_'
      action: drop
  file_sd_configs:
  - files:
    - '/scrapers/scrapers.yml'

Step 4: Restart NABox containers

dc restart

Step 5: Validate Volume Performance Collection in Harvest

Wait for atleast 5 minutes. To ensure that the ZapiPerf:Volume metrics are being collected every 5 seconds. You can validate this by using the following commands:

docker logs nabox-harvest2 &> logs.txt
grep -w 'ZapiPerf:Volume' logs.txt

These commands will redirect the logs from the nabox-harvest2 Docker container to a file named logs.txt, and then search for the term ZapiPerf:Volume within that file.

In the screenshot below, you can see that for the poller A250-41-42-43, the collection interval for ZapiPerf:volume is now 5 seconds.

Step 6: Verify Volume Performance Metrics in Prometheus

Confirm that the volume performance metrics are being scraped 12 times per minute by prometheus. This would mean that the scrape interval has been reduced to 5 seconds by Prometheus.

To verify this, run the following query:

count_over_time(volume_avg_latency[1m])

You can run this query on the following page: https://NABox/prometheus.