Skip to content

Commit 2656268

Browse files
committed
new article GPU-aware cluster monitoring
1 parent bf258ef commit 2656268

File tree

1 file changed

+42
-0
lines changed

1 file changed

+42
-0
lines changed
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
---
2+
title: Configure GPU monitoring with Azure Monitor for containers | Microsoft Docs
3+
description: This article describes how you can configure monitoring Kubernetes clusters with NVIDIA and AMD GPU enabled nodes with Azure Monitor for containers.
4+
ms.topic: conceptual
5+
ms.date: 03/09/2020
6+
---
7+
8+
# Configure GPU monitoring with Azure Monitor for containers
9+
10+
Starting with agent version *ciprod03022019*, Azure monitor for containers integrated agent now supports monitoring GPU (graphical processing units) usage on GPU-aware Kubernetes cluster nodes, and monitor pods/containers requesting and using GPU resources.
11+
12+
## Supported GPU Vendors
13+
14+
Azure Monitor for Containers supports monitoring GPU clusters from following GPU vendors:
15+
16+
- [NVIDIA](https://developer.nvidia.com/kubernetes-gpu)
17+
18+
- [AMD](https://github.com/RadeonOpenCompute/k8s-device-plugin)
19+
20+
Azure Monitor for containers automatically starts monitoring GPU usage on nodes, and GPU requesting pods and workloads by collecting the following metrics at 60sec intervals and storing them in the **InsightMetrics** table:
21+
22+
|Metric name |Metric dimension (tags) |Description |
23+
|------------|------------------------|------------|
24+
|containerGpuDutyCycle |container.azm.ms/clusterId, container.azm.ms/clusterName, containerName, gpuId, gpuModel, gpuVendor|Percentage of time over the past sample period (60 seconds) during which GPU was busy/actively processing for a container. Duty cycle is a number between 1 and 100. |
25+
|containerGpuLimits |container.azm.ms/clusterId, container.azm.ms/clusterName, containerName |Each container can specify limits as one or more GPUs. It is not possible to request or limit a fraction of a GPU. |
26+
|containerGpuRequests |container.azm.ms/clusterId, container.azm.ms/clusterName, containerName |Each container can request one or more GPUs. It is not possible to request or limit a fraction of a GPU.|
27+
|containerGpumemoryTotalBytes |container.azm.ms/clusterId, container.azm.ms/clusterName, containerName, gpuId, gpuModel, gpuVendor |Amount of GPU Memory in bytes available to use for a specific container. |
28+
|containerGpumemoryUsedBytes |container.azm.ms/clusterId, container.azm.ms/clusterName, containerName, gpuId, gpuModel, gpuVendor |Amount of GPU Memory in bytes used by a specific container. |
29+
|nodeGpuAllocatable |container.azm.ms/clusterId, container.azm.ms/clusterName, gpuVendor |Number of GPUs in a node that can be used by Kubernetes. |
30+
|nodeGpuCapacity |container.azm.ms/clusterId, container.azm.ms/clusterName, gpuVendor |Total Number of GPUs in a node. |
31+
32+
## GPU performance charts
33+
34+
Azure Monitor for containers includes pre-configured charts for the metrics listed earlier in the table as a GPU workbook for every cluster. You can find the GPU workbook directly from an AKS cluster by selecting **Workbooks** from the left-hand pane, and from the **View Workbooks** drop-down list in the Insight.
35+
36+
## Next steps
37+
38+
Use GPUs for compute-intensive workloads on Azure Kubernetes Service (AKS)
39+
40+
GPU Optimized VM SKUs in Microsoft Azure
41+
42+
GPU support in Kubernetes

0 commit comments

Comments
 (0)