Modify Collection Interval and apply filter for ZapiPerf WorkloadVolume Performance Template

[rahulguptajss]

Below docs are with reference to Feature

⚠️ Note:

1. Below steps are applicable for NABox.
2. Ensure that workload templates are enabled in NABox. To verify their existence, you can search for the qos_latency metric in Prometheus. If this metric exists, it indicates that the QoS templates are indeed enabled.
3. These steps modify the existing template (collecting workloads every 3 minutes) to exclude workloads that need to be collected more frequently. They add a new template to collect the workloads specified in the filter expression, which need to be collected every 5 seconds.
4. These steps are applicable only to the nightly builds currently. The filter feature will be included in the 24.02 release.

Prerequisite: ssh to NABox via root

Step 1: Create custom_my_workload.yaml

Navigate to the zapiperf directory:

cd /opt/harvest2-conf/conf/zapiperf

Create custom_my_workload.yaml:

vi custom_my_workload.yaml

Update the file with the following content:

objects:
  WorkloadVolume:
    - custom_workload_volume.yaml
  WorkloadVolumeCritical:
    - custom_workload_volume_critical.yaml

Step 2: Create custom_workload_volume.yaml

Navigate to the cdot/9.8.0 directory:

cd /opt/harvest2-conf/conf/zapiperf/cdot/9.8.0

Create custom_workload_volume.yaml:

vi custom_workload_volume.yaml

Update the file with the following content.

⚠️ Note: You should replace the query used in workload-name with your specific anti-pattern filter query. For instance, in the example below, we aim to exclude workloads containing the names 'fg2' and 'vol_dummy'. These represent the workloads that we wish to collect rapidly, and therefore, we aim to exclude them from the regular collection schedule.

For more details, refer to the workload filter templates.

counters:
  - filter:
      - workload-name: "!*fg2*,!*vol_dummy*"

Step 3: Create custom_workload_volume_critical.yaml

Navigate to the zapiperf directory:

cd /opt/harvest2-conf/conf/zapiperf/cdot/9.8.0

Create custom_workload_volume_critical.yaml:

vi custom_workload_volume_critical.yaml

Update the file with the following content:

⚠️ Note: You should replace the query used in workload-name with your specific filter query. For instance, in the example below, we aim to filter Workload performance instances by workload-name where the name contains fg2 or vol_dummy
For more details, refer to the workload filter templates.

name:                     WorkloadVolumeCritical
query:                    workload_volume
object:                   qos

client_timeout:           1m30s

schedule:
  - counter:  20m
  - instance: 10m
  - data:      5s

instance_key:             name

counters:
  - concurrency
  - instance_name
  - instance_uuid
  - latency
  - ops
  - other_ops
  - read_data
  - read_io_type
  - read_latency
  - read_ops
  - sequential_reads
  - sequential_writes
  - total_data
  - write_data
  - write_latency
  - write_ops
  - filter:
      - workload-name: "*fg2*|*vol_dummy*"


override:
  - read_io_type_base: delta

qos_labels:
  - vserver       => svm
  - volume
  - qtree
  - lun
  - file
  - policy-group
  - wid
  - workload-name => workload

export_options:
  instance_keys:
    - file
    - lun
    - policy_group
    - qtree
    - svm
    - volume
    - wid
    - workload

Step 4: Modify Prometheus Collection Interval for WorkloadVolume Metrics

⚠️ Note: An NABox upgrade will overwrite the prometheus.yml file. Ensure you have a backup.
These configuration changes instruct Prometheus to scrape all qos_ metrics (except qos_details_) every 5 seconds. However, only the filtered workloads specified in the custom_workload_volume_critical.yaml template are actually collected at this frequency by Harvest. Harvest collects the remaining workloads every 3 minutes.

1. Navigate to the Prometheus directory:

   cd /usr/local/nabox/files/prometheus

2. Create a backup of the prometheus.yml file:

   cp prometheus.yml prometheus_bkup.yml

3. Open prometheus.yml in a text editor:

   vi prometheus.yml

4. Make the following changes:

   • Remove these lines:

     - job_name: 'harvest2'
       file_sd_configs:
       - files:
         - '/scrapers/scrapers.yml'

   • Add the following lines:

     - job_name: "prometheus_qos_critical"
       scrape_interval: 5s
       metric_relabel_configs:
         - source_labels: [__name__]
           regex: 'qos_[^d].*'  # # Keep metrics that start with 'qos_' that are not 'qos_detail' metrics
           action: keep
       file_sd_configs:
       - files:
         - '/scrapers/scrapers.yml'
     
     - job_name: "harvest2"
       metric_relabel_configs:
         - source_labels: [__name__]
           regex: 'qos_[^d].*'  # Drop metrics that start with qos_ and include qos_detail metrics
           action: drop
       file_sd_configs:
       - files:
         - '/scrapers/scrapers.yml'

Step 5: Restart NABox containers

dc restart

Step 5: Validate WorkloadVolumeCritical Performance Collection in Harvest

Wait for at least ~5 minutes to ensure that the ZapiPerf:WorkloadVolumeCritical metrics are being collected every 5 seconds. You can validate this by using the following commands:

docker logs nabox-harvest2 &> logs.txt
grep -w 'ZapiPerf:WorkloadVolumeCritical' logs.txt

These commands will redirect the logs from the nabox-harvest2 Docker container to a file named logs.txt, and then search for the term ZapiPerf:WorkloadVolumeCritical within that file.

In the screenshot below, you can see that the poller umeng-aff300-01-02's collection interval for ZapiPerf:WorkloadVolumeCritical is now 5 seconds and exporting 4 instance as a result of filter expression.

2024-01-08T06:54:27Z INF collector/collector.go:575 > Collected Poller=umeng-aff300-01-02 apiMs=23 calcMs=0 collector=ZapiPerf:WorkloadVolumeCritical exportMs=0 instances=4 instancesExported=4 metrics=108 metricsExported=96 parseMs=2 pluginMs=0 pollMs=25 skips=0 zBegin=1704696867272
2024-01-08T06:54:32Z INF collector/collector.go:575 > Collected Poller=umeng-aff300-01-02 apiMs=447 calcMs=0 collector=ZapiPerf:WorkloadVolumeCritical exportMs=1 instances=4 instancesExported=4 metrics=108 metricsExported=96 parseMs=2 pluginMs=0 pollMs=450 skips=0 zBegin=1704696872272
2024-01-08T06:54:37Z INF collector/collector.go:575 > Collected Poller=umeng-aff300-01-02 apiMs=23 calcMs=0 collector=ZapiPerf:WorkloadVolumeCritical exportMs=0 instances=4 instancesExported=4 metrics=108 metricsExported=96 parseMs=2 pluginMs=0 pollMs=25 skips=0 zBegin=1704696877273
2024-01-08T06:54:42Z INF collector/collector.go:575 > Collected Poller=umeng-aff300-01-02 apiMs=67 calcMs=0 collector=ZapiPerf:WorkloadVolumeCritical exportMs=0 instances=4 instancesExported=4 metrics=108 metricsExported=96 parseMs=1 pluginMs=0 pollMs=68 skips=0 zBegin=1704696882274

Step 6: Verify WorkloadVolumeCritical Performance Metrics in Prometheus

Confirm that the WorkloadVolumeCritical performance metrics are being scraped 12 times per minute by Prometheus. This would mean that the scrape interval has been reduced to 5 seconds by Prometheus.

To verify this, run the following query:

count_over_time(qos_latency{workload="vol_dummy-wid29555"}[1m])

You can run this query on the following page: https://NABox/prometheus.

{cluster="umeng-aff300-01-02", datacenter="dc-1", instance="nabox-harvest2:12990", job="harvest2", policy_group="User-Best-Effort", svm="test_vserver", volume="vol_dummy", wid="29555", workload="vol_dummy-wid29555"} | 12