Prometheus Metrics Exporting to Azure Monitor

Prometheus To Azure Monitor

A Service to export queryable Prometheus Metrics to Azure Monitor for aggregation, persistence, and alerting.

Sample Jackson exposed Prometheus Metrics.

Fabrikate Deployment:

References for deploying through Fabrikate:

Helm chart
Sample Fabrikate HLD
- Modify /config/common.yaml to change prometheus queries.
- Modify /config/prod.yaml or whichever env component to add service principal auth components needed to write to Azure Metrics. (requirements listed below)
OSS Repository for source code

Example Service:

Prometheus Exporter for Azure Monitor

A solution built for a customer by Devdatta Waghdhare and Vinicius Souza. Permission has been given to open source this into a generic service/library. We need to look into steps needed to audit this first. Where can we find resources for this? Are there other tools?:

Steps to deploy this service to a cluster with Prometheus running:

Requirements:
- Active configured Prometheus Service
- Azure RBAC role with Contributor and Monitoring Metrics Publisher roles for targeted Resource(eg. AzureCluster)
Steps:
- Configure Prometheus URL, custom metric namespace and queried metrics in values.yaml and cronjob schedule.
- In deployment directory, deploy via script: sh install_helm_chart.sh -a <AzureResourceID> -b <AzureADClientID> -c <AzureADClientSecret> -d <AzureADTenantID> -e <AzureMonitorRegion> -f <tag> -g <AksNamespace> -h <PromToAzureMonitorCMD> -i <RepoImageName> -n <ChartName> -m <helmLog:Boolean> -o <deleteBeforeUpdate:Boolean> where:
  - <AzureResourceID>
    - Full resource ID
      - az aks show --resource-group <resource group> --name <cluster name> --query "id" --output table
    - /subscriptions/<SubscriptionID>/resourcegroups/<resourcegroup>/providers/Microsoft.ContainerService/managedClusters/<ClusterName>
  - <AzureADClientID>
    - Azure AD Application ID of RBAC role
    - <ApplicationID>
  - <AzureADClientSecret>
    - Azure AD Application Key
      - Settings -> Keys -> Create password/key and get value
    - <password key value string>
  - <AzureADTenantID>
    - Aka Directory ID
      - In portal: Azure Active Directory -> Properties -> Directory ID
    - <DirectoryID>
  - <AzureMonitorRegion>
    - Region where the resource exists. Must be a Azure Monitor supported region
    - eg. westus2
  - <tag>
    - Docker image tag name
    - eg. v0.1
  - <AksNamespace>
    - Kubernetes Name space for this service
    - eg. prom2azure
  - <PromToAzureMonitorCMD>
    - Command to run with cron job
    - ie. pull or push
  - <RepoImageName>
    - Docker/Azure container repository and image name
    - eg. mtarng/prom2azure
  - <ChartName>
    - Helm chart name
    - eg. prom2azurechart
  - <helmLog:Boolean>
    - Helm install debug dry-run
    - ie. true or false
  - <deleteBeforeUpdate:Boolean>
    - Helm delete before update
    - ie. true or false

Troubleshooting

If the docker image is unable to be built or run, try adding this to the Dockerfile:

RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

Prometheus

Azure Monitor

Custom Metrics in Azure Monitor

Current limitations as defined by Documentation

Identified bug that prevents query/metric names with commas

REST API Documentation

Supported Regions

Quotas and Limits

Category	Limit
Active time series/subscriptions/region*	50,000
Dimension keys per metric	10
String length for metric namespaces, metric names, dimension keys, and dimension values	256 characters

*An active time series is defined as any unique combination of metric, dimension key, or dimension value that has had metric values published in the past 12 hours.

Data is limited at 1 minute intervals.

Alerting via Azure Monitor may take up to 1.5 minutes.