Skip to content

Commit a91541f

Browse files
skopacz1skopacz1
authored
Merge pull request #73313 from max-cx/export-all-data
OBSDOCS-866: (TRACING-3922) Configure OTEL to scrape in-cluster monitoring stack
2 parents eca09b5 + 3d79cde commit a91541f

File tree

4 files changed

+135
-12
lines changed

4 files changed

+135
-12
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2876,8 +2876,8 @@ Topics:
28762876
File: otel-configuration-of-instrumentation
28772877
- Name: Sending traces and metrics to the Collector
28782878
File: otel-sending-traces-and-metrics-to-otel-collector
2879-
- Name: Sending metrics to the monitoring stack
2880-
File: otel-config-send-metrics-monitoring-stack
2879+
- Name: Configuring metrics for the monitoring stack
2880+
File: otel-configuring-metrics-for-monitoring-stack
28812881
- Name: Forwarding traces to a TempoStack
28822882
File: otel-forwarding
28832883
- Name: Configuring the Collector metrics

‎modules/otel-config-receive-metrics-monitoring-stack.adoc

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-configuring-metrics-for-monitoring-stack.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="configuration-for-receiving-metrics-from-the-monitoring-stack_{context}"]
7+
= Configuration for receiving metrics from the monitoring stack
8+
9+
A configured OpenTelemetry Collector custom resource (CR) can set up the Prometheus receiver to scrape metrics from the in-cluster monitoring stack.
10+
11+
.Example of the OpenTelemetry Collector CR for scraping metrics from the in-cluster monitoring stack
12+
[source,yaml]
13+
----
14+
apiVersion: rbac.authorization.k8s.io/v1
15+
kind: ClusterRoleBinding
16+
metadata:
17+
name: otel-collector
18+
roleRef:
19+
apiGroup: rbac.authorization.k8s.io
20+
kind: ClusterRole
21+
name: cluster-monitoring-view # <1>
22+
subjects:
23+
- kind: ServiceAccount
24+
name: otel-collector
25+
namespace: observability
26+
---
27+
kind: ConfigMap
28+
apiVersion: v1
29+
metadata:
30+
name: cabundle
31+
namespce: observability
32+
annotations:
33+
service.beta.openshift.io/inject-cabundle: "true" # <2>
34+
---
35+
apiVersion: opentelemetry.io/v1alpha1
36+
kind: OpenTelemetryCollector
37+
metadata:
38+
name: otel
39+
namespace: observability
40+
spec:
41+
volumeMounts:
42+
- name: cabundle-volume
43+
mountPath: /etc/pki/ca-trust/source/service-ca
44+
readOnly: true
45+
volumes:
46+
- name: cabundle-volume
47+
configMap:
48+
name: cabundle
49+
mode: deployment
50+
config: |
51+
receivers:
52+
prometheus: # <3>
53+
config:
54+
scrape_configs:
55+
- job_name: 'federate'
56+
scrape_interval: 15s
57+
scheme: https
58+
tls_config:
59+
ca_file: /etc/pki/ca-trust/source/service-ca/service-ca.crt
60+
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
61+
honor_labels: false
62+
params:
63+
'match[]':
64+
- '{__name__="<metric_name>"}' # <4>
65+
metrics_path: '/federate'
66+
static_configs:
67+
- targets:
68+
- "prometheus-k8s.openshift-monitoring.svc.cluster.local:9091"
69+
exporters:
70+
debug: # <5>
71+
verbosity: detailed
72+
service:
73+
pipelines:
74+
metrics:
75+
receivers: [prometheus]
76+
processors: []
77+
exporters: [debug]
78+
----
79+
<1> Assigns the `cluster-monitoring-view` cluster role to the service account of the OpenTelemetry Collector so that it can access the metrics data.
80+
<2> Injects the OpenShift service CA for configuring the TLS in the Prometheus receiver.
81+
<3> Configures the Prometheus receiver to scrape the federate endpoint from the in-cluster monitoring stack.
82+
<4> Uses the Prometheus query language to select the metrics to be scraped. See the in-cluster monitoring documentation for more details and limitations of the federate endpoint.
83+
<5> Configures the debug exporter to print the metrics to the standard output.

‎observability/otel/otel-config-send-metrics-monitoring-stack.adoc renamed to ‎modules/otel-config-send-metrics-monitoring-stack.adoc

Lines changed: 19 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,23 @@
1-
:_mod-docs-content-type: ASSEMBLY
2-
[id="otel-configuration-for-sending-metrics-to-the-monitoring-stack"]
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/otel/otel-configuring-metrics-for-monitoring-stack.adoc
4+
5+
:_mod-docs-content-type: REFERENCE
6+
[id="configuration-for-sending-metrics-to-the-monitoring-stack_{context}"]
37
= Configuration for sending metrics to the monitoring stack
4-
include::_attributes/common-attributes.adoc[]
5-
:context: otel-configuration-for-sending-metrics-to-the-monitoring-stack
68

7-
The OpenTelemetry Collector custom resource (CR) can be configured to create a Prometheus `ServiceMonitor` CR for scraping the Collector's pipeline metrics and the enabled Prometheus exporters.
9+
One of two following custom resources (CR) configures the sending of metrics to the monitoring stack:
10+
11+
* OpenTelemetry Collector CR
12+
* Prometheus `PodMonitor` CR
13+
14+
A configured OpenTelemetry Collector CR can create a Prometheus `ServiceMonitor` CR for scraping the Collector's pipeline metrics and the enabled Prometheus exporters.
815

9-
.Example of the OpenTelemetry Collector custom resource with the Prometheus exporter
16+
.Example of the OpenTelemetry Collector CR with the Prometheus exporter
1017
[source,yaml]
1118
----
19+
apiVersion: opentelemetry.io/v1alpha1
20+
kind: OpenTelemetryCollector
1221
spec:
1322
mode: deployment
1423
observability:
@@ -31,9 +40,9 @@ spec:
3140
----
3241
<1> Configures the Operator to create the Prometheus `ServiceMonitor` CR to scrape the Collector's internal metrics endpoint and Prometheus exporter metric endpoints. The metrics will be stored in the OpenShift monitoring stack.
3342

34-
Alternatively, a manually created Prometheus `PodMonitor` can provide fine control, for example removing duplicated labels added during Prometheus scraping.
43+
Alternatively, a manually created Prometheus `PodMonitor` CR can provide fine control, for example removing duplicated labels added during Prometheus scraping.
3544

36-
.Example of the `PodMonitor` custom resource that configures the monitoring stack to scrape the Collector metrics
45+
.Example of the `PodMonitor` CR that configures the monitoring stack to scrape the Collector metrics
3746
[source,yaml]
3847
----
3948
apiVersion: monitoring.coreos.com/v1
@@ -43,7 +52,7 @@ metadata:
4352
spec:
4453
selector:
4554
matchLabels:
46-
app.kubernetes.io/name: `<cr_name>-collector` # <1>
55+
app.kubernetes.io/name: <cr_name>-collector # <1>
4756
podMetricsEndpoints:
4857
- port: metrics # <2>
4958
- port: promexporter # <3>
@@ -60,6 +69,6 @@ spec:
6069
- action: labeldrop
6170
regex: job
6271
----
63-
<1> The name of the OpenTelemetry Collector custom resource.
72+
<1> The name of the OpenTelemetry Collector CR.
6473
<2> The name of the internal metrics port for the OpenTelemetry Collector. This port name is always `metrics`.
6574
<3> The name of the Prometheus exporter port for the OpenTelemetry Collector.

‎observability/otel/otel-configuring-metrics-for-monitoring-stack.adoc

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
[id="otel-configuring-metrics-for-monitoring-stack"]
3+
= Configuring metrics for the monitoring stack
4+
include::_attributes/common-attributes.adoc[]
5+
:context: otel-configuring-metrics-for-monitoring-stack
6+
7+
toc::[]
8+
9+
As a cluster administrator, you can configure the OpenTelemetry Collector custom resource (CR) to perform the following tasks:
10+
11+
* Create a Prometheus `ServiceMonitor` CR for scraping the Collector’s pipeline metrics and the enabled Prometheus exporters.
12+
13+
* Configure the Prometheus receiver to scrape metrics from the in-cluster monitoring stack.
14+
15+
include::modules/otel-config-send-metrics-monitoring-stack.adoc[leveloffset=+1]
16+
include::modules/otel-config-receive-metrics-monitoring-stack.adoc[leveloffset=+1]
17+
18+
[id="additional-resources_otel-configuring-metrics-for-monitoring-stack"]
19+
== Additional resources
20+
21+
// * xref:../monitoring/accessing-third-party-monitoring-apis.adoc#monitoring-querying-metrics-by-using-the-federation-endpoint-for-prometheus[Querying metrics by using the federation endpoint for Prometheus]
22+
23+
//* xref:../monitoring/accessing-third-party-monitoring-apis.adoc#monitoring-querying-metrics-by-using-the-federation-endpoint-for-prometheus_accessing-third-party-monitoring-apis[Querying metrics by using the federation endpoint for Prometheus]
24+
25+
//* xref:../monitoring/accessing-third-party-monitoring-apis.adoc#monitoring-querying-metrics-by-using-the-federation-endpoint-for-prometheus[Querying metrics by using the federation endpoint for Prometheus]
26+
27+
//* xref:../monitoring/accessing-third-party-monitoring-apis.adoc#monitoring-querying-metrics-by-using-the-federation-endpoint-for-prometheus_accessing-monitoring-apis-by-using-the-cli[Querying metrics by using the federation endpoint for Prometheus]
28+
29+
//* xref:../monitoring/accessing-third-party-monitoring-apis.adoc#accessing-third-party-monitoring-apis_monitoring-querying-metrics-by-using-the-federation-endpoint-for-prometheus[Querying metrics by using the federation endpoint for Prometheus]
30+
31+
* xref:../monitoring/accessing-third-party-monitoring-apis.adoc#monitoring-querying-metrics-by-using-the-federation-endpoint-for-prometheus_accessing-monitoring-apis-by-using-the-cli[Querying metrics by using the federation endpoint for Prometheus]

0 commit comments

Comments
 (0)