Add guidance for CICD per pipeline run metrics (#2618)

Co-authored-by: Joao Grassi <[email protected]>
Co-authored-by: Liudmila Molkova <[email protected]>

Author: 3 people

.chloggen/1184-per-cicd-run-metrics.yaml

@@ -0,0 +1,22 @@
+# Use this changelog template to create an entry for release notes.
+#
+# If your change doesn't affect end users you should instead start
+# your pull request title with [chore] or use the "Skip Changelog" label.
+
+# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
+change_type: enhancement
+
+# The name of the area of concern in the attributes-registry, (e.g. http, cloud, db)
+component: cicd
+
+# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
+note: Add guidance on per pipeline run metrics
+
+# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
+# The values here must be integers.
+issues: [1184]
+
+# (Optional) One or more lines of additional information to render under the primary note.
+# These lines will be padded with 2 spaces and then inserted directly into the document.
+# Use pipe (|) for multiline entries.
+subtext:

docs/cicd/cicd-metrics.md

@@ -9,6 +9,7 @@ linkTitle: Metrics
 <!-- toc -->
 
 - [CICD Metrics](#cicd-metrics)
+  - [Guidance on per pipeline run metrics](#guidance-on-per-pipeline-run-metrics)
   - [Metric: `cicd.pipeline.run.duration`](#metric-cicdpipelinerunduration)
   - [Metric: `cicd.pipeline.run.active`](#metric-cicdpipelinerunactive)
   - [Metric: `cicd.worker.count`](#metric-cicdworkercount)
@@ -35,6 +36,24 @@ The conventions described in this section are specific to Continuous Integration
 **Disclaimer:** These are initial CICD metrics and attributes
 but more may be added in the future.
 
+### Guidance on per pipeline run metrics
+
+It might be useful to gather metrics specific to each pipeline run.<br>
+Examples of such metrics include:
+
+* host metrics like cpu, memory, disk of the worker on which a pipeline run executes
+* container metrics like cpu, memory requests and limits
+* runtime metrics of any processes executed as part of the pipeline run like JVM metrics
+
+These metrics could be used to correlate build failures with environment issues like overload or out-of-memory.
+They could also be used in pipeline-level aggregations to inform adjustements of allocated worker resources.
+
+When reporting host, container, runtime, and other metrics in CI/CD pipelines, instrumentation authors and application developers SHOULD use applicable [CICD and VCS resource conventions][cicdres] and MAY also associate all or specific metrics with `cicd.pipeline.run` to correlate them with run information.
+
+The [`cicd.pipeline.run`](/docs/resource/cicd.md#cicd-pipeline-run) entity identifies a pipeline run. Associating it with metrics inherently causes high cardinality and may increase costs with some metric storage backends. Thus, `cicd.pipeline.run` entity MUST be opt-in.
+
+[cicdres]: /docs/resource/cicd.md (CICD and VCS resource conventions)
+
 ### Metric: `cicd.pipeline.run.duration`
 
 This metric is [recommended][MetricRecommended].