Prometheus Metrics Exporting to Azure Monitor

Prometheus To Azure Monitor

A Service to export queryable Prometheus Metrics to Azure Monitor for aggregation, persistence, and alerting.

Sample Jackson exposed Prometheus Metrics.

Example Service:

Prometheus Exporter for Azure Monitor

A solution built for a customer (Inside Sales) by Devdatta Waghdhare and Vinicius Souza. Permission has been given to open source this into a generic service/library. We need to look into steps needed to audit this first. Where can we find resources for this? Are there other tools?:

• https://docs.opensource.microsoft.com/tools/cg.html
• https://docs.opensource.microsoft.com/
• https://opensource.microsoft.com/resources/release
• https://opensource.microsoft.com/
• https://microsoft.sharepoint.com/teams/msopensource/Pages/default.aspx
• https://cloudblogs.microsoft.com/opensource/
• https://docs.opensource.microsoft.com/
• https://microsoft.sharepoint.com/teams/OpenSourceMicrosoft
• Adjacent Teams discussion

Steps to deploy this service to a cluster with Prometheus running:

• Requirements:
  • Active configured Prometheus Service
  • Azure RBAC role with Contributor and Monitoring Metrics Publisher roles for targeted Resource(eg. AzureCluster)
• Steps:
  • Configure Prometheus URL, custom metric namespace and queried metrics in values.yaml and cronjob schedule.
  • In deployment directory, deploy via script: sh install_helm_chart.sh -a <AzureResourceID> -b <AzureADClientID> -c <AzureADClientSecret> -d <AzureADTenantID> -e <AzureMonitorRegion> -f <tag> -g <AksNamespace> -h <PromToAzureMonitorCMD> -i <RepoImageName> -n <ChartName> -m <helmLog:Boolean> -o <deleteBeforeUpdate:Boolean> where:
    • <AzureResourceID>
      • Full resource ID
        • az aks show --resource-group <resource group> --name <cluster name> --query "id" --output table
      • /subscriptions/<SubscriptionID>/resourcegroups/<resourcegroup>/providers/Microsoft.ContainerService/managedClusters/<ClusterName>
    • <AzureADClientID>
      • Azure AD Application ID of RBAC role
      • <ApplicationID>
    • <AzureADClientSecret>
      • Azure AD Application Key
        • Settings -> Keys -> Create password/key and get value
      • <password key value string>
    • <AzureADTenantID>
      • Aka Directory ID
        • In portal: Azure Active Directory -> Properties -> Directory ID
      • <DirectoryID>
    • <AzureMonitorRegion>
      • Region where the resource exists. Must be a Azure Monitor supported region
      • eg. westus2
    • <tag>
      • Docker image tag name
      • eg. v0.1
    • <AksNamespace>
      • Kubernetes Name space for this service
      • eg. prom2azure
    • <PromToAzureMonitorCMD>
      • Command to run with cron job
      • ie. pull or push
    • <RepoImageName>
      • Docker/Azure container repository and image name
      • eg. mtarng/prom2azure
    • <ChartName>
      • Helm chart name
      • eg. prom2azurechart
    • <helmLog:Boolean>
      • Helm install debug dry-run
      • ie. true or false
    • <deleteBeforeUpdate:Boolean>
      • Helm delete before update
      • ie. true or false

Troubleshooting

If the docker image is unable to be built or run, try adding this to the Dockerfile:

RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2

Prometheus

• Querying Basics
• Querying Examples

Azure Monitor

Custom Metrics in Azure Monitor

Current limitations as defined by Documentation

REST API Documentation

Supported Regions

Quotas and Limits

Category	Limit
Active time series/subscriptions/region*	50,000
Dimension keys per metric	10
String length for metric namespaces, metric names, dimension keys, and dimension values	256 characters

*An active time series is defined as any unique combination of metric, dimension key, or dimension value that has had metric values published in the past 12 hours.

Custom Metrics

Azure Monitor stores all metrics at one-minute granularity intervals.

Azure Monitor Rest API

Azure Monitor Rest API Reference

Azure Keyvault

Kubernetes-KeyVault-FlexVolume