Merge pull request #268951 from rashmichandrashekar/rashmi/operator-docs

Updates for operator release

Author: Stacyrch140

articles/azure-monitor/containers/media/prometheus-metrics-troubleshoot/image-pod-monitor-config.png

@@ -15,7 +15,8 @@ This article provides instructions on customizing metrics scraping for a Kuberne
 Four different configmaps can be configured to provide scrape configuration and other settings for the metrics add-on. All config-maps should be applied to `kube-system` namespace for any cluster.
 
 > [!NOTE]
-> None of the four configmaps exist by default in the cluster when Managed Prometheus is enabled. Depending on what needs to be customized, you need to deploy any or all of these four configmaps with the same name specified, in `kube-system` namespace. AMA-Metrics pods will pick up these configmaps after you deploy them to `kube-system` namespace, and will restart in 2-3 minutes to apply the configuration settings specified in the configmap(s).
+> None of the four configmaps exist by default in the cluster when Managed Prometheus is enabled. Depending on what needs to be customized, you need to deploy any or all of these four configmaps with the same name specified, in `kube-system` namespace. AMA-Metrics pods will pick up these configmaps after you deploy them to `kube-system` namespace, and will restart in 2-3 minutes to apply the configuration settings specified in the configmap(s). 
+
 
 1. [`ama-metrics-settings-configmap`](https://aka.ms/azureprometheus-addon-settings-configmap)
    This config map has below simple settings that can be configured. You can take the configmap from the above git hub repo, change the settings are required and apply/deploy the configmap to `kube-system` namespace for your cluster
@@ -25,15 +26,20 @@ Four different configmaps can be configured to provide scrape configuration and
       * metric keep-lists - this setting is used to control which metrics are listed to be allowed from each default target and to change the default behavior
       * scrape intervals for default/pre-definetargets. `30 secs` is the default scrape frequency and it can be changed per default target using this configmap
       * debug-mode - turning this ON helps to debug missing metric/ingestion issues - see more on [troubleshooting](prometheus-metrics-troubleshoot.md#debug-mode)
-2. [`ama-metrics-prometheus-config`](https://aka.ms/azureprometheus-addon-rs-configmap) (**Recommended**)
+2. [`ama-metrics-prometheus-config`](https://aka.ms/azureprometheus-addon-rs-configmap)
    This config map can be used to provide Prometheus scrape config for addon replica. Addon runs a singleton replica, and any cluster level services can be discovered and scraped by providing scrape jobs in this configmap. You can take the sample configmap from the above git hub repo, add scrape jobs that you  would need and apply/deploy the config map to `kube-system` namespace for your cluster.
+   **Although this is supported, please note that the recommended way of scraping custom targets is using [custom resources](prometheus-metrics-scrape-configuration.md#custom-resource-definitions)**
 3. [`ama-metrics-prometheus-config-node`](https://aka.ms/azureprometheus-addon-ds-configmap) (**Advanced**)
     This config map can be used to provide Prometheus scrape config for addon DaemonSet that runs on every **Linux** node in the cluster, and any node level targets on each node can be scraped by providing scrape jobs in this configmap. When you use this configmap, you can use `$NODE_IP` variable in your scrape config, which gets substituted by corresponding  node's ip address in DaemonSet pod running on each node. This way you get access to scrape anything that runs on that node from the metrics addon DaemonSet. **Please be careful when you use discoveries in scrape config in this node level config map, as every node in the cluster will setup & discover the target(s) and will collect redundant metrics**.
     You can take the sample configmap from the above git hub repo, add scrape jobs that you  would need and apply/deploy the config map to `kube-system` namespace for your cluster
 4. [`ama-metrics-prometheus-config-node-windows`](https://aka.ms/azureprometheus-addon-ds-configmap-windows) (**Advanced**)
     This config map can be used to provide Prometheus scrape config for addon DaemonSet that runs on every **Windows** node in the cluster, and node level targets on each node can be scraped by providing scrape jobs in this configmap. When you use this configmap, you can use `$NODE_IP` variable in your scrape config, which will be substituted by corresponding  node's ip address in DaemonSet pod running on each node. This way you get access to scrape anything that runs on that node from the metrics addon DaemonSet. **Please be careful when you use discoveries in scrape config in this node level config map, as every node in the cluster will setup & discover the target(s) and will collect redundant metrics**.
     You can take the sample configmap from the above git hub repo, add scrape jobs that you  would need and apply/deploy the config map to `kube-system` namespace for your cluster
 
+## Custom Resource Definitions
+The Azure Monitor metrics add-on supports scraping Prometheus metrics using Prometheus - Pod Monitors and Service Monitors, similar to the OSS Prometheus operator. Enabling the add-on will deploy the Pod and Service Monitor custom resource definitions to allow you to create your own custom resources. 
+Follow the instructions to [create and apply custom resources](prometheus-metrics-scrape-crd.md) on your cluster.
+
 ## Metrics add-on settings configmap
 
 The [ama-metrics-settings-configmap](https://aka.ms/azureprometheus-addon-settings-configmap) can be downloaded, edited, and applied to the cluster to customize the out-of-the-box features of the metrics add-on.
@@ -142,17 +148,21 @@ and apply the YAML using the following command: `kubectl apply -f .\ama-metrics-
 
 ## Configure custom Prometheus scrape jobs
 
-You can configure the metrics add-on to scrape targets other than the default ones by using the same configuration format as the [Prometheus configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file).
-
-Follow the instructions to [create, validate, and apply the configmap](prometheus-metrics-scrape-validate.md) for your cluster.
-
+You can scrape Prometheus metrics using Prometheus - Pod Monitors and Service Monitors(**Recommended**), similar to the OSS Prometheus operator.
+Follow the instructions to [create and apply custom resources](prometheus-metrics-scrape-crd.md) on your cluster.
 
+Additionally, you can follow the instructions to [create, validate, and apply the configmap](prometheus-metrics-scrape-validate.md) for your cluster.
+The configuration format is similar to [Prometheus configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#configuration-file). 
 
 ## Prometheus configuration tips and examples
 
 Learn some tips from examples in this section.
 
-### Configuration file for custom scrape config
+### [Configuration using CRD for custom scrape config](#tab/CRDConfig)
+Use the [Pod and Service Monitor templates](https://github.com/Azure/prometheus-collector/tree/main/otelcollector/customresources) and follow the API specification to create your custom resources([PodMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#podmonitor) and [Service Monitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitor)). **Note** that the only change required to the existing OSS CRs for being picked up by the Managed Prometheus is the API group - **azmonitoring.coreos.com/v1**. See [here](prometheus-metrics-scrape-crd.md) to learn more
+
+
+### [Configuration file for custom scrape config](#tab/ConfigFile)
 
 The configuration format is the same as the [Prometheus configuration file](https://aka.ms/azureprometheus-promioconfig). Currently, the following sections are supported:
 
@@ -172,10 +182,143 @@ Any other unsupported sections must be removed from the config before they're ap
 
 See the [Apply config file](prometheus-metrics-scrape-validate.md#deploy-config-file-as-configmap) section to create a configmap from the Prometheus config.
 
+---
+
 > [!NOTE]
 > When custom scrape configuration fails to apply because of validation errors, default scrape configuration continues to be used.
 
+> If you want to use global settings that apply to all the scrape jobs, and only have [Custom Resources](prometheus-metrics-scrape-crd.md) you would still need to create a configmap with just the global settings(Settings for each of these in the custom resources will override the ones in the global section)
+
+
 ## Scrape configs
+### [Scrape Configs using CRD](#tab/CRDScrapeConfig)
+Currently, the supported methods of target discovery for custom resources are pod and service monitor
+
+#### Pod and Service Monitors
+Targets discovered using pod and service monitors have different `__meta_*` labels depending on what monitor is used. You can use the labels in the `relabelings` section to filter targets or replace labels for the targets.
+
+See the [Pod and Service Monitor examples](https://github.com/Azure/prometheus-collector/tree/main/otelcollector/deploy/example-custom-resources) of pod and service monitors.
+
+### Relabelings
+The `relabelings` section is applied at the time of target discovery and applies to each target for the job. The following examples show ways to use `relabelings`.
+
+#### Add a label
+Add a new label called `example_label` with the value `example_value` to every metric of the job. Use `__address__` as the source label only because that label always exists and adds the label for every target of the job.
+
+```yaml
+relabelings:
+- sourceLabels: [__address__]
+  targetLabel: example_label
+  replacement: 'example_value'
+```
+
+#### Use Pod or Service Monitor labels
+
+Targets discovered using pod and service monitors have different `__meta_*` labels depending on what monitor is used. The `__*` labels are dropped after discovering the targets. To filter by using them at the metrics level, first keep them using `relabelings` by assigning a label name. Then use `metricRelabelings` to filter.
+
+```yaml
+# Use the kubernetes namespace as a label called 'kubernetes_namespace'
+relabelings:
+- sourceLabels: [__meta_kubernetes_namespace]
+  action: replace
+  targetLabel: kubernetes_namespace
+
+# Keep only metrics with the kubernetes namespace 'default'
+metricRelabelings:
+- sourceLabels: [kubernetes_namespace]
+  action: keep
+  regex: 'default'
+```
+
+#### Job and instance relabeling
+
+You can change the `job` and `instance` label values based on the source label, just like any other label.
+
+```yaml
+# Replace the job name with the pod label 'k8s app'
+relabelings:
+- sourceLabels: [__meta_kubernetes_pod_label_k8s_app]
+  targetLabel: job
+
+# Replace the instance name with the node name. This is helpful to replace a node IP
+# and port with a value that is more readable
+relabelings:
+- sourceLabels: [__meta_kubernetes_node_name]]
+  targetLabel: instance
+```
+
+### Metric Relabelings
+
+Metric relabelings are applied after scraping and before ingestion. Use the `metricRelabelings` section to filter metrics after scraping. The following examples show how to do so.
+
+#### Drop metrics by name
+
+```yaml
+# Drop the metric named 'example_metric_name'
+metricRelabelings:
+- sourceLabels: [__name__]
+  action: drop
+  regex: 'example_metric_name'
+```
+
+#### Keep only certain metrics by name
+
+```yaml
+# Keep only the metric named 'example_metric_name'
+metricRelabelings:
+- sourceLabels: [__name__]
+  action: keep
+  regex: 'example_metric_name'
+```
+
+```yaml
+# Keep only metrics that start with 'example_'
+metricRelabelings:
+- sourceLabels: [__name__]
+  action: keep
+  regex: '(example_.*)'
+```
+
+#### Rename metrics
+Metric renaming isn't supported.
+
+#### Filter metrics by labels
+
+```yaml
+# Keep metrics only where example_label = 'example'
+metricRelabelings:
+- sourceLabels: [example_label]
+  action: keep
+  regex: 'example'
+```
+
+```yaml
+# Keep metrics only if `example_label` equals `value_1` or `value_2`
+metricRelabelings:
+- sourceLabels: [example_label]
+  action: keep
+  regex: '(value_1|value_2)'
+```
+
+```yaml
+# Keep metrics only if `example_label_1 = value_1` and `example_label_2 = value_2`
+metricRelabelings:
+- sourceLabels: [example_label_1, example_label_2]
+  separator: ';'
+  action: keep
+  regex: 'value_1;value_2'
+```
+
+```yaml
+# Keep metrics only if `example_label` exists as a label
+metricRelabelings:
+- sourceLabels: [example_label_1]
+  action: keep
+  regex: '.+'
+```
+
+
+### [Scrape Configs using Config file](#tab/ConfigFileScrapeConfig)
 Currently, the supported methods of target discovery for a [scrape config](https://aka.ms/azureprometheus-promioconfig-scrape) are either [`static_configs`](https://aka.ms/azureprometheus-promioconfig-static) or [`kubernetes_sd_configs`](https://aka.ms/azureprometheus-promioconfig-sdk8s) for specifying or discovering targets.
 
 #### Static config
@@ -313,6 +456,8 @@ metric_relabel_configs:
   regex: '.+'
 ```
 
+---
+
 ### TLS based scraping
 
 If you have a Prometheus instance served with TLS and you want to scrape metrics from it, you need to set scheme to `https` and set the TLS settings in your configmap or respective CRD. You can use the `tls_config` configuration property inside a custom scrape job to configure the TLS settings either using a CRD or a configmap. You need to provide a CA certificate to validate API server certificate with. The CA certificate is used to verify the authenticity of the server's certificate when Prometheus connects to the target over TLS. It helps ensure that the server's certificate is signed by a trusted authority.

articles/azure-monitor/containers/media/prometheus-metrics-troubleshoot/image-pod-service-monitor.png

@@ -0,0 +1,118 @@
+---
+title: Create and apply Pod and Service Monitors for Prometheus metrics in Azure Monitor
+description: Describes how to create and apply pod and service monitors to scrape Prometheus metrics in Azure Monitor to Kubernetes cluster.
+ms.topic: conceptual
+ms.date: 3/13/2024
+ms.reviewer: aul
+---
+# Custom Resource Definitions
+The enablement of managed prometheus automatically deploys the custom resource definitions (CRD) for [pod monitors](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/deploy/addon-chart/azure-monitor-metrics-addon/templates/ama-metrics-podmonitor-crd.yaml) and [service monitors](https://github.com/Azure/prometheus-collector/blob/main/otelcollector/deploy/addon-chart/azure-monitor-metrics-addon/templates/ama-metrics-servicemonitor-crd.yaml). These custom resource definitions are the same custom resource definitions (CRD) as [OSS Pod monitors](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.PodMonitor) and [OSS service monitors](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitor) for Prometheus, except for a change in the group name. If you have existing Prometheus CRDs and custom resources on your cluster, these CRDs won't conflict with the CRDs created by the add-on. At the same time, the managed Prometheus addon does not pick up the CRDs created for the OSS Prometheus. This separation is intentional for the purposes of isolation of scrape jobs.
+
+### Create a Pod or Service Monitor
+Use the [Pod and Service Monitor templates](https://github.com/Azure/prometheus-collector/tree/main/otelcollector/customresources) and follow the API specification to create your custom resources([PodMonitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#podmonitor) and [Service Monitor](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#monitoring.coreos.com/v1.ServiceMonitor)). **Note** that the only change required to the existing OSS CRs(Custom Resources) for being picked up by the Managed Prometheus is the API group - **azmonitoring.coreos.com/v1**.
+>Note - Please make sure to use the **labelLimit, labelNameLengthLimit and labelValueLengthLimit** specified in the templates so that they are not dropped during processing.
+
+Your pod and service monitors should look like the following examples:
+
+#### Example Pod Monitor -
+
+```yaml
+# Note the API version is azmonitoring.coreos.com/v1 instead of monitoring.coreos.com/v1
+apiVersion: azmonitoring.coreos.com/v1
+kind: PodMonitor
+
+# Can be deployed in any namespace
+metadata:
+  name: reference-app
+  namespace: app-namespace
+spec:
+  labelLimit: 63
+  labelNameLengthLimit: 511
+  labelValueLengthLimit: 1023
+
+  # The selector specifies which pods to filter for
+  selector:
+
+    # Filter by pod labels
+    matchLabels:
+      environment: test
+    matchExpressions:
+      - key: app
+        operator: In
+        values: [app-frontend, app-backend]
+
+    # [Optional] Filter by pod namespace
+    namespaceSelector:
+      matchNames: [app-frontend, app-backend]
+
+  # [Optional] Labels on the pod with these keys will be added as labels to each metric scraped
+  podTargetLabels: [app, region, environment]
+
+  # Multiple pod endpoints can be specified. Port requires a named port.
+  podMetricsEndpoints:
+    - port: metrics
+```
+#### Example Service Monitor - 
+```yaml
+# Note the API version is azmonitoring.coreos.com/v1 instead of monitoring.coreos.com/v1
+apiVersion: azmonitoring.coreos.com/v1
+kind: ServiceMonitor
+
+# Can be deployed in any namespace
+metadata:
+  name: reference-app
+  namespace: app-namespace
+spec:
+  labelLimit: 63
+  labelNameLengthLimit: 511
+  labelValueLengthLimit: 1023
+
+  # The selector filters endpoints by service labels.
+  selector:
+    matchLabels:
+      app: reference-app
+
+  # Multiple endpoints can be specified. Port requires a named port.
+  endpoints:
+  - port: metrics
+```
+
+### Deploy a Pod or Service Monitor
+You can then deploy the pod or service monitor using kubectl apply.
+
+
+When applied, any errors in the custom resources should show up and the pod or service monitors should fail to apply.  
+A successful pod monitor creation looks like the following - 
+```bash
+podmonitor.azmonitoring.coreos.com/my-pod-monitor created
+```
+
+### Examples
+#### Create a sample application
+Deploy a sample application exposing prometheus metrics to be configured by pod/service monitor.
+
+```bash
+kubectl apply -f https://github.com/Azure/prometheus-collector/blob/main/internal/referenceapp/prometheus-reference-app.yaml
+```
+
+#### Create a pod monitor and/or service monitor to scrape metrics 
+Deploy a pod monitor that is configured to scrape metrics from the example application from the previous step.
+
+##### Pod Monitor
+```bash
+kubectl apply -f https://github.com/Azure/prometheus-collector/blob/main/otelcollector/deploy/example-custom-resources/pod-monitor/pod-monitor-reference-app.yaml
+```
+
+##### Service Monitor
+```bash
+kubectl apply -f https://github.com/Azure/prometheus-collector/blob/main/otelcollector/deploy/example-custom-resources/service-monitor/service-monitor-reference-app.yaml
+```
+
+### Troubleshooting
+When the pod or service monitors are successfully applied, if you want to make sure that the pod or service monitor targets get picked up by the addon, follow the instructions [here](prometheus-metrics-troubleshoot.md#prometheus-interface) for general troubleshooting of custom resources and also to ensure the targets show up in 127.0.0.1/targets.
+
+  :::image type="content" source="media/prometheus-metrics-troubleshoot/image-pod-service-monitor.png" alt-text="Screenshot showing targets for pod/service monitor" lightbox="media/prometheus-metrics-troubleshoot/image-pod-service-monitor.png":::
+
+## Next steps
+
+- [Learn more about collecting Prometheus metrics](../essentials/prometheus-metrics-overview.md).