Skip to content

Commit 9b97431

Browse files
PMEds28PMEds28
authored
Merge pull request #106061 from avirishuv/add-doc-vmss-automatic-instance-repairs
add document for VMSS automatic instance repairs
2 parents 1a3aee1 + ead3e89 commit 9b97431

File tree

2 files changed

+249
-0
lines changed

2 files changed

+249
-0
lines changed

‎articles/virtual-machine-scale-sets/TOC.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -247,6 +247,8 @@
247247
href: virtual-machine-scale-sets-networking.md
248248
- name: Automatic OS upgrades
249249
href: virtual-machine-scale-sets-automatic-upgrade.md
250+
- name: Automatic instance repairs (preview)
251+
href: virtual-machine-scale-sets-automatic-instance-repairs.md
250252
- name: Instance protection
251253
href: virtual-machine-scale-sets-instance-protection.md
252254
- name: Scale-In Policy

‎articles/virtual-machine-scale-sets/virtual-machine-scale-sets-automatic-instance-repairs.md

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
---
2+
title: Automatic instance repairs with Azure virtual machine scale sets
3+
description: Learn how to configure automatic repairs policy for VM instances in a scale set
4+
author: avirishuv
5+
manager: vashan
6+
tags: azure-resource-manager
7+
8+
ms.service: virtual-machine-scale-sets
9+
ms.workload: infrastructure-services
10+
ms.tgt_pltfrm: vm
11+
ms.topic: conceptual
12+
ms.date: 02/28/2020
13+
ms.author: avverma
14+
15+
---
16+
# Preview: Automatic instance repairs for Azure virtual machine scale sets
17+
18+
Enabling automatic instance repairs for Azure virtual machine scale sets helps achieve high availability for applications by maintaining a set of healthy instances. If an instance in the scale set is found to be unhealthy as reported by [Application Health extension](./virtual-machine-scale-sets-health-extension.md) or [Load balancer health probes](../load-balancer/load-balancer-custom-probe-overview.md), then this feature automatically performs instance repair by deleting the unhealthy instance and creating a new one to replace it.
19+
20+
> [!NOTE]
21+
> This preview feature is provided without a service level agreement, and it's not recommended for production workloads.
22+
23+
## Requirements for using automatic instance repairs
24+
25+
**Opt in for the automatic instance repairs preview**
26+
27+
Use either the REST API or Azure PowerShell to opt in for the automatic instance repairs preview. These steps will register your subscription for the preview feature. Note this is only a one-time setup required for using this feature. If your subscription is already registered for automatic instance repairs preview, then you do not need to register again.
28+
29+
Using REST API
30+
31+
1. Register for the feature using [Features - Register](/rest/api/resources/features/register)
32+
33+
```
34+
POST on '/subscriptions/{subscriptionId}/providers/Microsoft.Features/providers/Microsoft.Compute/features/RepairVMScaleSetInstancesPreview/register?api-version=2015-12-01'
35+
```
36+
37+
```json
38+
{
39+
"properties": {
40+
"state": "Registering"
41+
},
42+
"id": "/subscriptions/{subscriptionId}/providers/Microsoft.Features/providers/Microsoft.Compute/features/RepairVMScaleSetInstancesPreview",
43+
"type": "Microsoft.Features/providers/features",
44+
"name": "Microsoft.Compute/RepairVMScaleSetInstancesPreview"
45+
}
46+
```
47+
48+
2. Wait for a few minutes for the *State* to change to *Registered*. You can use the following API to confirm this.
49+
50+
```
51+
GET on '/subscriptions/{subscriptionId}/providers/Microsoft.Features/providers/Microsoft.Compute/features/RepairVMScaleSetInstancesPreview?api-version=2015-12-01'
52+
```
53+
54+
```json
55+
{
56+
"properties": {
57+
"state": "Registered"
58+
},
59+
"id": "/subscriptions/{subscriptionId}/providers/Microsoft.Features/providers/Microsoft.Compute/features/RepairVMScaleSetInstancesPreview",
60+
"type": "Microsoft.Features/providers/features",
61+
"name": "Microsoft.Compute/RepairVMScaleSetInstancesPreview"
62+
}
63+
```
64+
65+
3. Once the *State* has changed to *Registered*, then run the following.
66+
67+
```
68+
POST on '/subscriptions/{subscriptionId}/providers/Microsoft.Compute/register?api-version=2015-12-01'
69+
```
70+
71+
Using Azure PowerShell
72+
73+
1. Register for the feature using cmdlet [Register-AzureRmResourceProvider](/powershell/module/azurerm.resources/register-azurermresourceprovider) followed by [Register-AzureRmProviderFeature](/powershell/module/azurerm.resources/register-azurermproviderfeature)
74+
75+
```azurepowershell-interactive
76+
Register-AzureRmResourceProvider `
77+
-ProviderNamespace Microsoft.Compute
78+
79+
Register-AzureRmProviderFeature `
80+
-ProviderNamespace Microsoft.Compute `
81+
-FeatureName RepairVMScaleSetInstancesPreview
82+
```
83+
84+
2. Wait for a few minutes for the *RegistrationState* to change to *Registered*. You can use the following cmdlet to confirm this.
85+
86+
```azurepowershell-interactive
87+
Get-AzureRmProviderFeature `
88+
-ProviderNamespace Microsoft.Compute `
89+
-FeatureName RepairVMScaleSetInstancesPreview
90+
```
91+
92+
The response should be as follows.
93+
94+
| FeatureName  | ProviderName | RegistrationState |
95+
|---------------------------------------|-------------------------|-------------------------|
96+
| RepairVMScaleSetInstancesPreview | Microsoft.Compute | Registered |
97+
98+
3. Once the *RegistrationState* to change to *Registered*, then run the following cmdlet.
99+
100+
```azurepowershell-interactive
101+
Register-AzureRmResourceProvider `
102+
-ProviderNamespace Microsoft.Compute
103+
```
104+
105+
**Enable application health monitoring for scale set**
106+
107+
The scale set should have application health monitoring for instances enabled. This can be done using either [Application Health extension](./virtual-machine-scale-sets-health-extension.md) or [Load balancer health probes](../load-balancer/load-balancer-custom-probe-overview.md). Only one of these can be enabled at a time. The application health extension or the load balancer probes ping the application endpoint configured on virtual machine instances to determine the application health status. This health status is used by the scale set orchestrator to monitor instance health and perform repairs when required.
108+
109+
**Configure endpoint to provide health status**
110+
111+
Before enabling automatic instance repairs policy, ensure that the scale set instances have application endpoint configured to emit the application health status. When an instance returns status 200 (OK) on this application endpoint, then the instance is marked as “Healthy”. In all other cases, the instance is marked “Unhealthy”, including the following scenarios:
112+
113+
- When there is no application endpoint configured inside the virtual machine instances to provide application health status
114+
- When the application endpoint is incorrectly configured
115+
- When the application endpoint is not reachable
116+
117+
For instances marked as “Unhealthy”, automatic repairs are triggered by the scale set. Ensure the application endpoint is correctly configured before enabling the automatic repairs policy in order to avoid unintended instance repairs, while the endpoint is getting configured.
118+
119+
**Enable single placement group**
120+
121+
This preview is currently available only for scale sets deployed as single placement group. The property *singlePlacementGroup* should be set to *true* for your scale set to use automatic instance repairs feature. Learn more about [placement groups](./virtual-machine-scale-sets-placement-groups.md#placement-groups).
122+
123+
**API version**
124+
125+
Automatic repairs policy is supported for compute API version 2018-10-01 or higher.
126+
127+
**Restrictions on resource or subscription moves**
128+
129+
As part of this preview, resource or subscription moves are currently not supported for scale sets when automatic repairs policy is enabled.
130+
131+
**Restriction for service fabric scale sets**
132+
133+
This preview feature is currently not supported for service fabric scale sets.
134+
135+
## How do automatic instance repairs work?
136+
137+
Automatic instance repair feature relies on health monitoring of individual instances in a scale set. VM instances in a scale set can be configured to emit application health status using either the [Application Health extension](./virtual-machine-scale-sets-health-extension.md) or [Load balancer health probes](../load-balancer/load-balancer-custom-probe-overview.md). If an instance is found to be unhealthy, then the scale set performs repair action by deleting the unhealthy instance and creating a new one to replace it. This feature can be enabled in the virtual machine scale set model by using the *automaticRepairsPolicy* object.
138+
139+
### Batching
140+
141+
The automatic instance repair operations are performed in batches. At any given time, no more than 5% of the instances in the scale set are repaired through the automatic repairs policy. This helps avoid simultaneous deletion and re-creation of a large number of instances if found unhealthy at the same time.
142+
143+
### Grace period
144+
145+
When an instance goes through a state change operation because of a PUT, PATCH or POST action performed on the scale set (for example reimage, redeploy, update, etc.), then any repair action on that instance is performed only after waiting for the grace period. Grace period is the amount of time to allow the instance to return to healthy state. The grace period starts after the state change has completed. This helps avoid any premature or accidental repair operations. The grace period is honored for any newly created instance in the scale set (including the one created as a result of repair operation). Grace period is specified in minutes in ISO 8601 format and can be set using the property *automaticRepairsPolicy.gracePeriod*. Grace period can range between 30 minutes and 90 minutes, and has a default value of 30 minutes.
146+
147+
The automatic instance repairs process works as follows:
148+
149+
1. [Application Health extension](./virtual-machine-scale-sets-health-extension.md) or [Load balancer health probes](../load-balancer/load-balancer-custom-probe-overview.md) ping the application endpoint inside each virtual machine in the scale set to get application health status for each instance.
150+
2. If the endpoint responds with a status 200 (OK), then the instance is marked as “Healthy”. In all the other cases (including if the endpoint is unreachable), the instance is marked “Unhealthy”.
151+
3. When an instance is found to be unhealthy, the scale set triggers a repair action by deleting the unhealthy instance and creating a new one to replace it.
152+
4. Instance repairs are performed in batches. At any given time, no more than 5% of the total instances in the scale set are repaired. If a scale set has fewer than 20 instances, the repairs are done for one unhealthy instance at a time.
153+
5. The above process continues until all unhealthy instance in the scale set are repaired.
154+
155+
## Instance protection and automatic repairs
156+
157+
If an instance in a scale set is protected by applying the *[Protect from scale-set actions protection policy](./virtual-machine-scale-sets-instance-protection.md#protect-from-scale-set-actions)*, then automatic repairs are not performed on that instance.
158+
159+
## Enabling automatic repairs policy when creating a new scale set
160+
161+
For enabling automatic repairs policy while creating a new scale set, ensure that all the [requirements](#requirements-for-using-automatic-instance-repairs) for opting in to this feature are met. The application endpoint should be correctly configured for scale set instances to avoid triggering unintended repairs while the endpoint is getting configured. For newly created scale sets, any instance repairs are performed only after waiting for the duration of grace period. To enable the automatic instance repair in a scale set, use *automaticRepairsPolicy* object in the virtual machine scale set model.
162+
163+
### REST API
164+
165+
The following example shows how to enable automatic instance repair in a scale set model. Use API version 2018-10-01 or higher.
166+
167+
```
168+
PUT or PATCH on '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachineScaleSets/{vmScaleSetName}?api-version=2019-07-01'
169+
```
170+
171+
```json
172+
{
173+
"properties": {
174+
"automaticRepairsPolicy": {
175+
"enabled": "true",
176+
"gracePeriod": "PT30M"
177+
}
178+
}
179+
}
180+
```
181+
182+
### Azure PowerShell
183+
184+
The automatic instance repair feature can be enabled while creating a new scale set by using the [New-AzVmssConfig](/powershell/module/az.compute/new-azvmssconfig) cmdlet. This sample script walks through the creation of a scale set and associated resources using the configuration file: [Create a complete virtual machine scale set](./scripts/powershell-sample-create-complete-scale-set.md). You can configure automatic instance repairs policy by adding the parameters *EnableAutomaticRepair* and *AutomaticRepairGracePeriod* to the configuration object for creating the scale set. The following example enables the feature with a grace period of 30 minutes.
185+
186+
```azurepowershell-interactive
187+
New-AzVmssConfig `
188+
-Location "EastUS" `
189+
-SkuCapacity 2 `
190+
-SkuName "Standard_DS2" `
191+
-UpgradePolicyMode "Automatic" `
192+
-EnableAutomaticRepair $true `
193+
-AutomaticRepairGracePeriod "PT30M"
194+
```
195+
196+
## Enabling automatic repairs policy when updating an existing scale set
197+
198+
Before enabling automatic repairs policy in an existing scale set, ensure that all the [requirements](#requirements-for-using-automatic-instance-repairs) for opting in to this feature are met. The application endpoint should be correctly configured for scale set instances to avoid triggering unintended repairs while the endpoint is getting configured. To enable the automatic instance repair in a scale set, use *automaticRepairsPolicy* object in the virtual machine scale set model.
199+
200+
### REST API
201+
202+
The following example enables the policy with grace period of 40 minutes. Use API version 2018-10-01 or higher.
203+
204+
```
205+
PUT or PATCH on '/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Compute/virtualMachineScaleSets/{vmScaleSetName}?api-version=2019-07-01'
206+
```
207+
208+
```json
209+
{
210+
"properties": {
211+
"automaticRepairsPolicy": {
212+
"enabled": "true",
213+
"gracePeriod": "PT40M"
214+
}
215+
}
216+
}
217+
```
218+
219+
### Azure PowerShell
220+
221+
Use the [Update-AzVmss](/powershell/module/az.compute/update-azvmss) cmdlet to modify the configuration of automatic instance repair feature in an existing scale set. The following example updates the grace period to 40 minutes.
222+
223+
```azurepowershell-interactive
224+
Update-AzVmss `
225+
-ResourceGroupName "myResourceGroup" `
226+
-VMScaleSetName "myScaleSet" `
227+
-EnableAutomaticRepair $true `
228+
-AutomaticRepairGracePeriod "PT40M"
229+
```
230+
231+
## Troubleshoot
232+
233+
**Failure to enable automatic repairs policy**
234+
235+
If you get a ‘BadRequest’ error with a message stating “Could not find member ‘automaticRepairsPolicy’ on object of type ‘properties’”, then check the API version used for virtual machine scale set. API version 2018-10-01 or higher is required for this feature.
236+
237+
**Instance not getting repaired even when policy is enabled**
238+
239+
The instance could be in grace period. This is the amount of time to wait after any state change on the instance before performing repairs. This is to avoid any premature or accidental repairs. The repair action should happen once the grace period is completed for the instance.
240+
241+
**Viewing application health status for scale set instances**
242+
243+
You can use the [Get Instance View API](/rest/api/compute/virtualmachinescalesetvms/getinstanceview) for instances in a virtual machine scale set to view the application health status. With Azure PowerShell, you can use the cmdlet [Get-AzVmssVM](/powershell/module/az.compute/get-azvmssvm) with the *-InstanceView* flag. The application health status is provided under the property *vmHealth*.
244+
245+
## Next steps
246+
247+
Learn how to configure [Application Health extension](./virtual-machine-scale-sets-health-extension.md) or [Load balancer health probes](../load-balancer/load-balancer-custom-probe-overview.md) for your scale sets.

0 commit comments

Comments
 (0)