Skip to content

Commit 7aeaa79

Browse files
gabriel-rhgabriel-rh
authored
Merge pull request #84486 from gabriel-rh/OBSDOCS-1161
OBSDOCS-1161 add content for validating that the monitoring stack is working
2 parents f6042dd + 3f1f135 commit 7aeaa79

3 files changed

+146
-0
lines changed

‎modules/monitoring-creating-a-monitoringstack-object-for-cluster-observability-operator.adoc

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,3 +59,64 @@ $ oc -n ns1-coo get monitoringstack
5959
NAME AGE
6060
example-coo-monitoring-stack 81m
6161
----
62+
63+
. Run the following comand to retrieve information about the active targets from Prometheus and filter the output to list only targets labeled with `app=prometheus-coo-example-app`. This verifies which targets are discovered and actively monitored by Prometheus with this specific label.
64+
+
65+
[source,terminal]
66+
----
67+
$ oc -n oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[].discoveredLabels | select(.__meta_kubernetes_endpoints_label_app=="prometheus-coo-example-app")'
68+
----
69+
+
70+
.Example output
71+
[source,json]
72+
----
73+
{
74+
"__address__": "10.129.2.25:8080",
75+
"__meta_kubernetes_endpoint_address_target_kind": "Pod",
76+
"__meta_kubernetes_endpoint_address_target_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
77+
"__meta_kubernetes_endpoint_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
78+
"__meta_kubernetes_endpoint_port_name": "web",
79+
"__meta_kubernetes_endpoint_port_protocol": "TCP",
80+
"__meta_kubernetes_endpoint_ready": "true",
81+
"__meta_kubernetes_endpoints_annotation_endpoints_kubernetes_io_last_change_trigger_time": "2024-11-05T11:24:09Z",
82+
"__meta_kubernetes_endpoints_annotationpresent_endpoints_kubernetes_io_last_change_trigger_time": "true",
83+
"__meta_kubernetes_endpoints_label_app": "prometheus-coo-example-app",
84+
"__meta_kubernetes_endpoints_labelpresent_app": "true",
85+
"__meta_kubernetes_endpoints_name": "prometheus-coo-example-app",
86+
"__meta_kubernetes_namespace": "ns1-coo",
87+
"__meta_kubernetes_pod_annotation_k8s_ovn_org_pod_networks": "{\"default\":{\"ip_addresses\":[\"10.129.2.25/23\"],\"mac_address\":\"0a:58:0a:81:02:19\",\"gateway_ips\":[\"10.129.2.1\"],\"routes\":[{\"dest\":\"10.128.0.0/14\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"172.30.0.0/16\",\"nextHop\":\"10.129.2.1\"},{\"dest\":\"100.64.0.0/16\",\"nextHop\":\"10.129.2.1\"}],\"ip_address\":\"10.129.2.25/23\",\"gateway_ip\":\"10.129.2.1\",\"role\":\"primary\"}}",
88+
"__meta_kubernetes_pod_annotation_k8s_v1_cni_cncf_io_network_status": "[{\n \"name\": \"ovn-kubernetes\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.129.2.25\"\n ],\n \"mac\": \"0a:58:0a:81:02:19\",\n \"default\": true,\n \"dns\": {}\n}]",
89+
"__meta_kubernetes_pod_annotation_openshift_io_scc": "restricted-v2",
90+
"__meta_kubernetes_pod_annotation_seccomp_security_alpha_kubernetes_io_pod": "runtime/default",
91+
"__meta_kubernetes_pod_annotationpresent_k8s_ovn_org_pod_networks": "true",
92+
"__meta_kubernetes_pod_annotationpresent_k8s_v1_cni_cncf_io_network_status": "true",
93+
"__meta_kubernetes_pod_annotationpresent_openshift_io_scc": "true",
94+
"__meta_kubernetes_pod_annotationpresent_seccomp_security_alpha_kubernetes_io_pod": "true",
95+
"__meta_kubernetes_pod_controller_kind": "ReplicaSet",
96+
"__meta_kubernetes_pod_controller_name": "prometheus-coo-example-app-5d8cd498c7",
97+
"__meta_kubernetes_pod_host_ip": "10.0.128.2",
98+
"__meta_kubernetes_pod_ip": "10.129.2.25",
99+
"__meta_kubernetes_pod_label_app": "prometheus-coo-example-app",
100+
"__meta_kubernetes_pod_label_pod_template_hash": "5d8cd498c7",
101+
"__meta_kubernetes_pod_labelpresent_app": "true",
102+
"__meta_kubernetes_pod_labelpresent_pod_template_hash": "true",
103+
"__meta_kubernetes_pod_name": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
104+
"__meta_kubernetes_pod_node_name": "ci-ln-8tt8vxb-72292-6cxjr-worker-a-wdfnz",
105+
"__meta_kubernetes_pod_phase": "Running",
106+
"__meta_kubernetes_pod_ready": "true",
107+
"__meta_kubernetes_pod_uid": "054c11b6-9a76-4827-a860-47f3a4596871",
108+
"__meta_kubernetes_service_label_app": "prometheus-coo-example-app",
109+
"__meta_kubernetes_service_labelpresent_app": "true",
110+
"__meta_kubernetes_service_name": "prometheus-coo-example-app",
111+
"__metrics_path__": "/metrics",
112+
"__scheme__": "http",
113+
"__scrape_interval__": "30s",
114+
"__scrape_timeout__": "10s",
115+
"job": "serviceMonitor/ns1-coo/prometheus-coo-example-monitor/0"
116+
}
117+
----
118+
+
119+
[NOTE]
120+
====
121+
The above example uses link:https://jqlang.github.io/jq/[`jq` command-line JSON processor] to format the output for convenience.
122+
====

‎modules/monitoring-validating-a-monitoringstack-for-cluster-observability-operator.adoc

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * observability/cluster-observability-operator/configuring-the-cluster-observability-operator-to-monitor-a-service.adoc
4+
5+
:_mod-docs-content-type: PROCEDURE
6+
[id="monitoring-validating-a-monitoringstack-for-cluster-observability-operator_{context}"]
7+
= Validating the monitoring stack
8+
9+
To validate that the monitoring stack is working correctly, access the example service and then view the gathered metrics.
10+
11+
.Prerequisites
12+
13+
* You have access to the cluster as a user with the `cluster-admin` cluster role or as a user with administrative permissions for the namespace.
14+
* You have installed the {coo-full}.
15+
* You have deployed the `prometheus-coo-example-app` sample service in the `ns1-coo` namespace.
16+
* You have created a `ServiceMonitor` object named `prometheus-coo-example-monitor` in the `ns1-coo` namespace.
17+
* You have created a `MonitoringStack` object named `example-coo-monitoring-stack` in the `ns1-coo` namespace.
18+
19+
.Procedure
20+
21+
. Create a route to expose the example `prometheus-coo-example-app` service. From your terminal, run the command:
22+
+
23+
[source,terminal]
24+
----
25+
$ oc expose svc prometheus-coo-example-app
26+
----
27+
. Access the route from your browser, or command line, to generate metrics.
28+
29+
. Execute a query on the Prometheus pod to return the total HTTP requests metric:
30+
+
31+
[source,terminal]
32+
----
33+
$ oc -n ns1-coo exec -c prometheus prometheus-example-coo-monitoring-stack-0 -- curl -s 'http://localhost:9090/api/v1/query?query=http_requests_total'
34+
----
35+
+
36+
.Example output (formatted using `jq` for convenience)
37+
[source,json]
38+
----
39+
{
40+
"status": "success",
41+
"data": {
42+
"resultType": "vector",
43+
"result": [
44+
{
45+
"metric": {
46+
"__name__": "http_requests_total",
47+
"code": "200",
48+
"endpoint": "web",
49+
"instance": "10.129.2.25:8080",
50+
"job": "prometheus-coo-example-app",
51+
"method": "get",
52+
"namespace": "ns1-coo",
53+
"pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
54+
"service": "prometheus-coo-example-app"
55+
},
56+
"value": [
57+
1730807483.632,
58+
"3"
59+
]
60+
},
61+
{
62+
"metric": {
63+
"__name__": "http_requests_total",
64+
"code": "404",
65+
"endpoint": "web",
66+
"instance": "10.129.2.25:8080",
67+
"job": "prometheus-coo-example-app",
68+
"method": "get",
69+
"namespace": "ns1-coo",
70+
"pod": "prometheus-coo-example-app-5d8cd498c7-9j2gj",
71+
"service": "prometheus-coo-example-app"
72+
},
73+
"value": [
74+
1730807483.632,
75+
"0"
76+
]
77+
}
78+
]
79+
}
80+
}
81+
----

‎observability/cluster_observability_operator/configuring-the-cluster-observability-operator-to-monitor-a-service.adoc

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,7 @@ include::modules/monitoring-specifying-how-a-service-is-monitored-by-cluster-obs
2525

2626
// Create a MonitoringStack object to discover the service monitor
2727
include::modules/monitoring-creating-a-monitoringstack-object-for-cluster-observability-operator.adoc[leveloffset=+1]
28+
29+
// Validate a MonitoringStack
30+
include::modules/monitoring-validating-a-monitoringstack-for-cluster-observability-operator.adoc[leveloffset=+1]
31+

0 commit comments

Comments
 (0)