RFE: Calculate change point on each metric series

[afcollins]

Problem:
One metric in a test config is detecting a changepoint. If I add an additional metric to the same config to focus the changepoint detection on one subset of the same data, it affects whether a changepoint is detected in the original data set.

When I use this config against OCP 4.19 data:

tests :
  - name : payload-cluster-density-v2
    index: {{ es_metadata_index }}
    benchmarkIndex: {{ es_benchmark_index }}
    metadata:
      platform: AWS
      clusterType: self-managed
      masterNodesType: m6a.xlarge
      masterNodesCount: 3
      workerNodesType: m6a.xlarge
      workerNodesCount: 6
      benchmark.keyword: cluster-density-v2
      ocpVersion: {{ version }}
      networkType: OVNKubernetes
      jobType: periodic

    metrics :
    - name: multusCPU
      metricName : containerCPU
      labels.namespace.keyword: openshift-multus
      metric_of_interest: value
      agg:
        value: cpu
        agg_type: avg
      labels:
        - "[Jira: multus]"
      direction: 1                                                                                                                                                                                                                          

I get a changepoint on run fc90b400-ee81-4fd1-afbf-dd1408eab767.

When I add a metric to focus on a only whereabouts pods, it appears to filter out uuids which don't contain whereabouts pods for all metrics.

tests :
  - name : payload-cluster-density-v2
    index: {{ es_metadata_index }}
    benchmarkIndex: {{ es_benchmark_index }}
    metadata:
      platform: AWS
      clusterType: self-managed
      masterNodesType: m6a.xlarge
      masterNodesCount: 3
      workerNodesType: m6a.xlarge
      workerNodesCount: 6
      benchmark.keyword: cluster-density-v2
      ocpVersion: {{ version }}
      networkType: OVNKubernetes
      jobType: periodic

    metrics :
    - name: multusCPU
      metricName : containerCPU
      labels.namespace.keyword: openshift-multus
      metric_of_interest: value
      agg:
        value: cpu
        agg_type: avg
      labels:
        - "[Jira: multus]"
      direction: 1

    - name: multusCPU-whereabouts-only
      metricName : containerCPU
      labels.namespace.keyword: openshift-multus
      labels.pod: whereabouts-reconciler
      metric_of_interest: value
      agg:
        value: cpu
        agg_type: avg
      labels:
        - "[Jira: multus]"
      direction: 1

output:

<testsuites>
        <testsuite name="payload-cluster-density-v2 nightly compare" failures="0" tests="2">
                <testcase name="[Jira: multus] multusCPU_avg regression detection" timestamp="1744664801"/>
                <testcase name="[Jira: multus] multusCPU-whereabouts-only_avg regression detection" timestamp="1744664801"/>
        </testsuite>
</testsuites>

The resulting dataset CSVs are different lengths:

% wc multus_{,without_}changepoint/data-payload-cluster-density-v2.csv
      31      31    7946 multus_changepoint/data-payload-cluster-density-v2.csv
      24      24    6606 multus_without_changepoint/data-payload-cluster-density-v2.csv

And I only see the changepoint uuid in the first config.
This uuid was filtered out of the dataset retrieved by the second config because this run doesn't contain any whereabouts-reconciler pods.

% grep -c fc90b400 multus_{,without_}changepoint/data-payload-cluster-density-v2.csv
multus_changepoint/data-payload-cluster-density-v2.csv:1
multus_without_changepoint/data-payload-cluster-density-v2.csv:0

[afcollins]

What seems like is happening here is the UUID list is being modified in place based on the most restrictive filter matches.

What I expect to happen is that the full set of all UUIDs is used for each metric.

In the output logs here, it appears that each metric is being queried as I expect, but maybe the data set is restricted when it is passed to Hunter, and hence why we don't detect the changepoint.

[jtaleric]

In the output logs here, it appears that each metric is being queried as I expect, but maybe the data set is restricted when it is passed to Hunter, and hence why we don't detect the changepoint.

This is the intended functionality... When we match on a specific metric, we only compare the matching results. However, I can see where for triage or investigations we would want to mix in matching/non-matching.

This would seem more like an RFE.

[afcollins]

Thank you for responding. I updated the title.

This makes sense to me we would do it to create the table output so it does not have empty cells, but I was confused with the junit output appearing to evaluate each metric individually.

Anyway, an RFE nonetheless.

[afcollins]

After more thinking about this, especially for the junit test report output, I think we need this behavior.

That is: Running otava for each metric column separately. Instead of truncating the dataset based on the minimum denominator.

One reason for this is introducing a new metric for change point detection. With the current behavior, we would see the data set be truncated for all metrics until we have enough runs with the new metric.

It may be a transient issue for 15 runs, but I believe removing this dependency will empower us more to add metrics we want to track.