Merge pull request #51059 from bburt-rh/RHDEVDOCS-4472-new-sample-code-for-how-to-view-telemetry-metrics

RHDEVDOCS-4472 -Add query to view metrics collected by Telemetry

Author: bburt-rh

modules/telemetry-showing-data-collected-from-the-cluster.adoc

@@ -6,28 +6,168 @@
 [id="showing-data-collected-from-the-cluster_{context}"]
 = Showing data collected by Telemetry
 
-You can see the cluster and components time series data captured by Telemetry.
+You can view the cluster and components time series data captured by Telemetry.
 
 .Prerequisites
 
-* Install the OpenShift CLI (`oc`).
-* You must log in to the cluster with a user that has either the `cluster-admin` role or the `cluster-monitoring-view` role.
+* You have installed the {product-title} CLI (`oc`).
+* You have access to the cluster as a user with the `cluster-admin` role or the `cluster-monitoring-view` role.
 
 .Procedure
 
-. Find the URL for the Prometheus service that runs in the {product-title} cluster:
-+
-[source,terminal]
-----
-$ oc get route prometheus-k8s -n openshift-monitoring -o jsonpath="{.spec.host}"
-----
+. Log in to a cluster.
 
-. Navigate to the URL.
-
-. Enter this query in the *Expression* input box and press *Execute*:
+. Run the following command, which queries a cluster's Prometheus service and returns the full set of time series data captured by Telemetry:
 +
+[source,terminal]
 ----
-{__name__=~"cluster:usage:.*|count:up0|count:up1|cluster_version|cluster_version_available_updates|cluster_operator_up|cluster_operator_conditions|cluster_version_payload|cluster_installer|cluster_infrastructure_provider|cluster_feature_set|instance:etcd_object_counts:sum|ALERTS|code:apiserver_request_total:rate:sum|cluster:capacity_cpu_cores:sum|cluster:capacity_memory_bytes:sum|cluster:cpu_usage_cores:sum|cluster:memory_usage_bytes:sum|openshift:cpu_usage_cores:sum|openshift:memory_usage_bytes:sum|workload:cpu_usage_cores:sum|workload:memory_usage_bytes:sum|cluster:virt_platform_nodes:sum|cluster:node_instance_type_count:sum|cnv:vmi_status_running:count|cluster:vmi_request_cpu_cores:sum|node_role_os_version_machine:cpu_capacity_cores:sum|node_role_os_version_machine:cpu_capacity_sockets:sum|subscription_sync_total|olm_resolution_duration_seconds|csv_succeeded|csv_abnormal|cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum|cluster:kubelet_volume_stats_used_bytes:provisioner:sum|ceph_cluster_total_bytes|ceph_cluster_total_used_raw_bytes|ceph_health_status|job:ceph_osd_metadata:count|job:kube_pv:count|job:ceph_pools_iops:total|job:ceph_pools_iops_bytes:total|job:ceph_versions_running:count|job:noobaa_total_unhealthy_buckets:sum|job:noobaa_bucket_count:sum|job:noobaa_total_object_count:sum|noobaa_accounts_num|noobaa_total_usage|console_url|cluster:network_attachment_definition_instances:max|cluster:network_attachment_definition_enabled_instance_up:max|cluster:ingress_controller_aws_nlb_active:sum|insightsclient_request_send_total|cam_app_workload_migrations|cluster:apiserver_current_inflight_requests:sum:max_over_time:2m|cluster:alertmanager_integrations:max|cluster:telemetry_selected_series:count|openshift:prometheus_tsdb_head_series:sum|openshift:prometheus_tsdb_head_samples_appended_total:sum|monitoring:container_memory_working_set_bytes:sum|namespace_job:scrape_series_added:topk3_sum1h|namespace_job:scrape_samples_post_metric_relabeling:topk3|monitoring:haproxy_server_http_responses_total:sum|rhmi_status|cluster_legacy_scheduler_policy|cluster_master_schedulable|che_workspace_status|che_workspace_started_total|che_workspace_failure_total|che_workspace_start_time_seconds_sum|che_workspace_start_time_seconds_count|cco_credentials_mode|cluster:kube_persistentvolume_plugin_type_counts:sum|visual_web_terminal_sessions_total|acm_managed_cluster_info|cluster:vsphere_vcenter_info:sum|cluster:vsphere_esxi_version_total:sum|cluster:vsphere_node_hw_version_total:sum|openshift:build_by_strategy:sum|rhods_aggregate_availability|rhods_total_users|instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile|instance:etcd_mvcc_db_total_size_in_bytes:sum|instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile|instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum|instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile|jaeger_operator_instances_storage_types|jaeger_operator_instances_strategies|jaeger_operator_instances_agent_strategies|appsvcs:cores_by_product:sum|nto_custom_profiles:count|openshift_csi_share_configmap|openshift_csi_share_secret|openshift_csi_share_mount_failures_total|openshift_csi_share_mount_requests_total",alertstate=~"firing|"}
+$ curl -G -k -H "Authorization: Bearer $(oc whoami -t)" \
+https://$(oc get route prometheus-k8s-federate -n \
+openshift-monitoring -o jsonpath="{.spec.host}")/federate \
+--data-urlencode 'match[]={__name__=~"cluster:usage:.*"}' \
+--data-urlencode 'match[]={__name__="count:up0"}' \
+--data-urlencode 'match[]={__name__="count:up1"}' \
+--data-urlencode 'match[]={__name__="cluster_version"}' \
+--data-urlencode 'match[]={__name__="cluster_version_available_updates"}' \
+--data-urlencode 'match[]={__name__="cluster_version_capability"}' \
+--data-urlencode 'match[]={__name__="cluster_operator_up"}' \
+--data-urlencode 'match[]={__name__="cluster_operator_conditions"}' \
+--data-urlencode 'match[]={__name__="cluster_version_payload"}' \
+--data-urlencode 'match[]={__name__="cluster_installer"}' \
+--data-urlencode 'match[]={__name__="cluster_infrastructure_provider"}' \
+--data-urlencode 'match[]={__name__="cluster_feature_set"}' \
+--data-urlencode 'match[]={__name__="instance:etcd_object_counts:sum"}' \
+--data-urlencode 'match[]={__name__="ALERTS",alertstate="firing"}' \
+--data-urlencode 'match[]={__name__="code:apiserver_request_total:rate:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:capacity_cpu_cores:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:capacity_memory_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:cpu_usage_cores:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:memory_usage_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="openshift:cpu_usage_cores:sum"}' \
+--data-urlencode 'match[]={__name__="openshift:memory_usage_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="workload:cpu_usage_cores:sum"}' \
+--data-urlencode 'match[]={__name__="workload:memory_usage_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:virt_platform_nodes:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:node_instance_type_count:sum"}' \
+--data-urlencode 'match[]={__name__="cnv:vmi_status_running:count"}' \
+--data-urlencode 'match[]={__name__="cluster:vmi_request_cpu_cores:sum"}' \
+--data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_cores:sum"}' \
+--data-urlencode 'match[]={__name__="node_role_os_version_machine:cpu_capacity_sockets:sum"}' \
+--data-urlencode 'match[]={__name__="subscription_sync_total"}' \
+--data-urlencode 'match[]={__name__="olm_resolution_duration_seconds"}' \
+--data-urlencode 'match[]={__name__="csv_succeeded"}' \
+--data-urlencode 'match[]={__name__="csv_abnormal"}' \
+--data-urlencode 'match[]={__name__="cluster:kube_persistentvolumeclaim_resource_requests_storage_bytes:provisioner:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:kubelet_volume_stats_used_bytes:provisioner:sum"}' \
+--data-urlencode 'match[]={__name__="ceph_cluster_total_bytes"}' \
+--data-urlencode 'match[]={__name__="ceph_cluster_total_used_raw_bytes"}' \
+--data-urlencode 'match[]={__name__="ceph_health_status"}' \
+--data-urlencode 'match[]={__name__="odf_system_raw_capacity_total_bytes"}' \
+--data-urlencode 'match[]={__name__="odf_system_raw_capacity_used_bytes"}' \
+--data-urlencode 'match[]={__name__="odf_system_health_status"}' \
+--data-urlencode 'match[]={__name__="job:ceph_osd_metadata:count"}' \
+--data-urlencode 'match[]={__name__="job:kube_pv:count"}' \
+--data-urlencode 'match[]={__name__="job:odf_system_pvs:count"}' \
+--data-urlencode 'match[]={__name__="job:ceph_pools_iops:total"}' \
+--data-urlencode 'match[]={__name__="job:ceph_pools_iops_bytes:total"}' \
+--data-urlencode 'match[]={__name__="job:ceph_versions_running:count"}' \
+--data-urlencode 'match[]={__name__="job:noobaa_total_unhealthy_buckets:sum"}' \
+--data-urlencode 'match[]={__name__="job:noobaa_bucket_count:sum"}' \
+--data-urlencode 'match[]={__name__="job:noobaa_total_object_count:sum"}' \
+--data-urlencode 'match[]={__name__="odf_system_bucket_count", system_type="OCS", system_vendor="Red Hat"}' \
+--data-urlencode 'match[]={__name__="odf_system_objects_total", system_type="OCS", system_vendor="Red Hat"}' \
+--data-urlencode 'match[]={__name__="noobaa_accounts_num"}' \
+--data-urlencode 'match[]={__name__="noobaa_total_usage"}' \
+--data-urlencode 'match[]={__name__="console_url"}' \
+--data-urlencode 'match[]={__name__="cluster:ovnkube_master_egress_routing_via_host:max"}' \
+--data-urlencode 'match[]={__name__="cluster:network_attachment_definition_instances:max"}' \
+--data-urlencode 'match[]={__name__="cluster:network_attachment_definition_enabled_instance_up:max"}' \
+--data-urlencode 'match[]={__name__="cluster:ingress_controller_aws_nlb_active:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:min"}' \
+--data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:max"}' \
+--data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:avg"}' \
+--data-urlencode 'match[]={__name__="cluster:route_metrics_controller_routes_per_shard:median"}' \
+--data-urlencode 'match[]={__name__="cluster:openshift_route_info:tls_termination:sum"}' \
+--data-urlencode 'match[]={__name__="insightsclient_request_send_total"}' \
+--data-urlencode 'match[]={__name__="cam_app_workload_migrations"}' \
+--data-urlencode 'match[]={__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}' \
+--data-urlencode 'match[]={__name__="cluster:alertmanager_integrations:max"}' \
+--data-urlencode 'match[]={__name__="cluster:telemetry_selected_series:count"}' \
+--data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_series:sum"}' \
+--data-urlencode 'match[]={__name__="openshift:prometheus_tsdb_head_samples_appended_total:sum"}' \
+--data-urlencode 'match[]={__name__="monitoring:container_memory_working_set_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="namespace_job:scrape_series_added:topk3_sum1h"}' \
+--data-urlencode 'match[]={__name__="namespace_job:scrape_samples_post_metric_relabeling:topk3"}' \
+--data-urlencode 'match[]={__name__="monitoring:haproxy_server_http_responses_total:sum"}' \
+--data-urlencode 'match[]={__name__="rhmi_status"}' \
+--data-urlencode 'match[]={__name__="status:upgrading:version:rhoam_state:max"}' \
+--data-urlencode 'match[]={__name__="state:rhoam_critical_alerts:max"}' \
+--data-urlencode 'match[]={__name__="state:rhoam_warning_alerts:max"}' \
+--data-urlencode 'match[]={__name__="rhoam_7d_slo_percentile:max"}' \
+--data-urlencode 'match[]={__name__="rhoam_7d_slo_remaining_error_budget:max"}' \
+--data-urlencode 'match[]={__name__="cluster_legacy_scheduler_policy"}' \
+--data-urlencode 'match[]={__name__="cluster_master_schedulable"}' \
+--data-urlencode 'match[]={__name__="che_workspace_status"}' \
+--data-urlencode 'match[]={__name__="che_workspace_started_total"}' \
+--data-urlencode 'match[]={__name__="che_workspace_failure_total"}' \
+--data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_sum"}' \
+--data-urlencode 'match[]={__name__="che_workspace_start_time_seconds_count"}' \
+--data-urlencode 'match[]={__name__="cco_credentials_mode"}' \
+--data-urlencode 'match[]={__name__="cluster:kube_persistentvolume_plugin_type_counts:sum"}' \
+--data-urlencode 'match[]={__name__="visual_web_terminal_sessions_total"}' \
+--data-urlencode 'match[]={__name__="acm_managed_cluster_info"}' \
+--data-urlencode 'match[]={__name__="cluster:vsphere_vcenter_info:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:vsphere_esxi_version_total:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:vsphere_node_hw_version_total:sum"}' \
+--data-urlencode 'match[]={__name__="openshift:build_by_strategy:sum"}' \
+--data-urlencode 'match[]={__name__="rhods_aggregate_availability"}' \
+--data-urlencode 'match[]={__name__="rhods_total_users"}' \
+--data-urlencode 'match[]={__name__="instance:etcd_disk_wal_fsync_duration_seconds:histogram_quantile",quantile="0.99"}' \
+--data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="instance:etcd_network_peer_round_trip_time_seconds:histogram_quantile",quantile="0.99"}' \
+--data-urlencode 'match[]={__name__="instance:etcd_mvcc_db_total_size_in_use_in_bytes:sum"}' \
+--data-urlencode 'match[]={__name__="instance:etcd_disk_backend_commit_duration_seconds:histogram_quantile",quantile="0.99"}' \
+--data-urlencode 'match[]={__name__="jaeger_operator_instances_storage_types"}' \
+--data-urlencode 'match[]={__name__="jaeger_operator_instances_strategies"}' \
+--data-urlencode 'match[]={__name__="jaeger_operator_instances_agent_strategies"}' \
+--data-urlencode 'match[]={__name__="appsvcs:cores_by_product:sum"}' \
+--data-urlencode 'match[]={__name__="nto_custom_profiles:count"}' \
+--data-urlencode 'match[]={__name__="openshift_csi_share_configmap"}' \
+--data-urlencode 'match[]={__name__="openshift_csi_share_secret"}' \
+--data-urlencode 'match[]={__name__="openshift_csi_share_mount_failures_total"}' \
+--data-urlencode 'match[]={__name__="openshift_csi_share_mount_requests_total"}' \
+--data-urlencode 'match[]={__name__="cluster:velero_backup_total:max"}' \
+--data-urlencode 'match[]={__name__="cluster:velero_restore_total:max"}' \
+--data-urlencode 'match[]={__name__="eo_es_storage_info"}' \
+--data-urlencode 'match[]={__name__="eo_es_redundancy_policy_info"}' \
+--data-urlencode 'match[]={__name__="eo_es_defined_delete_namespaces_total"}' \
+--data-urlencode 'match[]={__name__="eo_es_misconfigured_memory_resources_info"}' \
+--data-urlencode 'match[]={__name__="cluster:eo_es_data_nodes_total:max"}' \
+--data-urlencode 'match[]={__name__="cluster:eo_es_documents_created_total:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:eo_es_documents_deleted_total:sum"}' \
+--data-urlencode 'match[]={__name__="pod:eo_es_shards_total:max"}' \
+--data-urlencode 'match[]={__name__="eo_es_cluster_management_state_info"}' \
+--data-urlencode 'match[]={__name__="imageregistry:imagestreamtags_count:sum"}' \
+--data-urlencode 'match[]={__name__="imageregistry:operations_count:sum"}' \
+--data-urlencode 'match[]={__name__="log_logging_info"}' \
+--data-urlencode 'match[]={__name__="log_collector_error_count_total"}' \
+--data-urlencode 'match[]={__name__="log_forwarder_pipeline_info"}' \
+--data-urlencode 'match[]={__name__="log_forwarder_input_info"}' \
+--data-urlencode 'match[]={__name__="log_forwarder_output_info"}' \
+--data-urlencode 'match[]={__name__="cluster:log_collected_bytes_total:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:log_logged_bytes_total:sum"}' \
+--data-urlencode 'match[]={__name__="cluster:kata_monitor_running_shim_count:sum"}' \
+--data-urlencode 'match[]={__name__="platform:hypershift_hostedclusters:max"}' \
+--data-urlencode 'match[]={__name__="platform:hypershift_nodepools:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_bucket_claims:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_buckets_claims:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_resources:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_namespace_resources:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_unhealthy_namespace_buckets:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_namespace_buckets:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_accounts:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_usage:max"}' \
+--data-urlencode 'match[]={__name__="namespace:noobaa_system_health_status:max"}' \
+--data-urlencode 'match[]={__name__="ocs_advanced_feature_usage"}' \
+--data-urlencode 'match[]={__name__="os_image_url_override:sum"}'
 ----
-+
-This query replicates the request that Telemetry makes against a running {product-title} cluster's Prometheus service and returns the full set of time series captured by Telemetry.