Skip to content

Commit abda463

Browse files
v-dirichardsv-dirichards
authored
Merge pull request #229689 from AbhishekMallick01/Mar-7-2023-AKS
AKS post release updates
2 parents e14a879 + 43f3205 commit abda463

6 files changed

+206
-16
lines changed

‎articles/backup/azure-kubernetes-service-backup-overview.md

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
---
2+
title: Azure Kubernetes Service backup - Overview
3+
description: This article gives you an understanding about Azure Kubernetes Service (AKS) backup, the cloud-native process to back up and restore the containerized applications and data running in AKS clusters.
4+
ms.topic: conceptual
5+
ms.service: backup
6+
ms.date: 03/14/2023
7+
author: jyothisuri
8+
ms.author: jsuri
9+
---
10+
11+
# Overview of Azure Kubernetes Service backup using Azure Backup (preview)
12+
13+
[Azure Kubernetes Service (AKS)](/azure/aks/intro-kubernetes) backup is a simple, cloud-native process to back up and restore the containerized applications and data running in AKS clusters. You can configure scheduled backup for cluster state and application data (persistent volumes - CSI driver-based Azure Disks). The solution provides granular control to choose a specific namespace or an entire cluster to back up or restore by storing backups locally in a blob container and as disk snapshots. With AKS backup, you can unlock end-to-end scenarios - operational recovery, cloning developer/test environments, or cluster upgrade scenarios.
14+
15+
AKS backup integrates with Backup center (with other backup management capabilities) to provide a single pane of glass that helps you govern, monitor, operate, and analyze backups at scale.
16+
17+
## How does AKS backup work?
18+
19+
AKS backup enables you to back up your Kubernetes workloads and persistent volumes deployed in AKS clusters. The solution requires a [**Backup Extension**](/azure/azure-arc/kubernetes/conceptual-extensions) to be installed in the AKS cluster. Backup vault communicates to the Backup Extension to perform backup and restore related operations. You can configure scheduled backups for your clusters as per your backup policy and can restore the backups to the original or an alternate cluster within the same subscription and region. The extension also allows you to enable granular controls to choose a specific namespace or an entire cluster as a backup/restore configuration while performing the specific operation.
20+
21+
>[!Note]
22+
>- You must install Backup Extension in the AKS cluster to enable backups and restores. With the extension installation, a User Identity is created in the AKS cluster's managed resource group (Extension Identity), which gets assigned a set of permissions to access the storage account with the backups stored in the blob container.
23+
>
24+
>- An AKS cluster can have only one Backup Extension installed at a time.
25+
>
26+
>- Currently, AKS backup allows storing backups in Operational Tier. Operational Tier is a local data store and backups aren't moved to a vault but are stored in your own tenant. However, the Backup vault still serves as the unit for managing backups.
27+
28+
The backup solution enables backups for your Kubernetes workloads deployed in the cluster and the data stored in the persistent volume. Currently, the solution only supports persistent volumes of CSI driver-based Azure Disks. During backups, other *PV* types (such as File Share and Blobs) are skipped by the solution. The Kubernetes workloads are stored in a blob container and the Disk-based persistent volumes are backed up as Disk snapshots.
29+
30+
## Backup
31+
32+
To configure backup for AKS cluster, first you need to create a *Backup vault*. The vault gives you a consolidated view of the backups configured across different workloads. AKS backup supports only Operational Tier backup.
33+
Note: Copying backups to the Vault Tier is currently not supported. So, the Backup vault storage redundancy setting (LRS/GRS) doesn't apply to the backups stored in Operational Tier.
34+
35+
AKS backup automatically triggers scheduled backup job that copies the cluster resources to a blob container and creates an incremental snapshot of the disk-based persistent volumes as per the backup frequency. Older backups are deleted as per the retention duration specified by the backup policy.
36+
37+
>[!Note]
38+
>AKS backup allows creating multiple backup instances for a single AKS cluster. You can create multiple backup Instances with different backup configurations, as required. However, each backup instance of an AKS cluster should be created with a different backup policy, either in the same or in a different Backup vault.
39+
40+
## Backup management
41+
42+
Once the backup configuration for an AKS cluster is complete, a backup instance is created in the Backup vault. You can view the backup instance for the cluster under the Backup section in the AKS portal. You can perform any Backup-related operations for the Instance, such as initiating restores, monitoring, stopping protection, and so on, through its corresponding backup instance.
43+
44+
AKS backup also integrates directly with Backup center to help you manage the protection of all your storage accounts centrally along with all other backup supported workloads. The Backup center is a single pane of glass for all your backup requirements, such as monitoring jobs and state of backups and restores, ensuring compliance and governance, analyzing backup usage, and performing operations pertaining to back up and restore of data.
45+
46+
AKS backup uses Managed Identity to access other Azure resources. To configure backup of an AKS cluster and to restore from past backup, Backup vault's Managed Identity requires a set of permissions on the AKS cluster and the snapshot resource group where snapshots are created and managed. Currently, the AKS cluster requires a set of permissions on the Snapshot Resource Group. Also, the Backup Extension creates a User Identity and assigns a set of permissions to access the storage account where backups are stored in a blob. You can grant permissions to the Managed Identity using Azure role-based access control (Azure RBAC). Managed Identity is a service principle of a special type that can only be used with Azure resources. Learn more about [Managed Identities](/azure/active-directory/managed-identities-azure-resources/overview.md).
47+
48+
## Restore
49+
50+
You can restore data from any point-in-time for which a recovery point exists. A recovery point is created when a backup instance is in protected state, and can be used to restore data until it's retained by the backup policy.
51+
52+
Azure Backup provides an instant restore experience because the snapshots are stored locally in your subscription. Operational backup gives you the option to restore all the backed-up items or use the granular controls to select specific items from the backup by choosing namespaces and other available filters. Also, you've the ability to perform the restore on the original AKS cluster (that's backed up) or alternate AKS cluster in the same region and subscription.
53+
54+
## Pricing
55+
56+
You won't incur any management charges or instance fee when using AKS backup for Operational Tier in preview. However, you'll incur the charges for:
57+
58+
- Retention of backup data stored in the blob container.
59+
- Disk-based persistent volume snapshots are created by AKS backup are stored in the resource group in your Azure subscription and incur Snapshot Storage charges. Because the snapshots aren't copied to the Backup vault, Backup Storage cost doesn't apply. For more information on the snapshot pricing, see [Managed Disk Pricing](https://azure.microsoft.com/pricing/details/managed-disks/).
60+
61+
AKS backup uses incremental snapshots of the Disk-based persistent volumes. Incremental snapshots are charged *per GiB of the storage occupied by the delta changes* since the last snapshot. For example, if you're using a disk-based persistent volume with a provisioned size of *128 GiB*, with *100 GiB* used, then the first incremental snapshot is charged only for the used size of *100 GiB*. *20 GiB* of data is added on the disk before you create the second snapshot. Now, the second incremental snapshot is charged for only *20 GiB*.
62+
63+
Incremental snapshots are always stored on standard storage, irrespective of the storage type of parent-managed disks and are charged based on the pricing of standard storage. For example, incremental snapshots of a Premium SSD-Managed Disk are stored on standard storage. By default, they're stored on zonal redundant storage (ZRS) in regions that support ZRS. Otherwise, they're stored locally redundant storage (LRS). The per GiB pricing of both the options, LRS and ZRS, is the same.
64+
65+
## Next steps
66+
67+
- [Prerequisites for Azure Kubernetes Service backup (preview)](azure-kubernetes-service-cluster-backup-concept.md)

‎articles/backup/azure-kubernetes-service-backup-troubleshoot.md

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
title: Troubleshoot Azure Kubernetes Service backup
3+
description: Symptoms, causes, and resolutions of Azure Kubernetes Service backup and restore.
4+
ms.topic: troubleshooting
5+
ms.date: 03/14/2023
6+
ms.service: backup
7+
author: jyothisuri
8+
ms.author: jsuri
9+
---
10+
11+
# Troubleshoot Azure Kubernetes Service backup and restore (preview)
12+
13+
This article provides troubleshooting steps that help you resolve Azure Kubernetes Service (AKS) backup, restore, and management errors.
14+
15+
## AKS Backup Extension installation error resolutions
16+
17+
### Scenario 1
18+
19+
**Error message**:
20+
21+
```Erroe
22+
{Helm installation from path [] for release [azure-aks-backup] failed with the following error: err [release azure-aks-backup failed, and has been uninstalled due to atomic being set: failed post-install: timed out waiting for the condition]} occurred while doing the operation: {Installing the extension} on the config"`
23+
```
24+
25+
26+
**Cause**: The extension has been installed successfully, but the pods aren't spawning. This happens because the required compute and memory aren't available for the pods.
27+
28+
**Resolution**: To resolve the issue, increase the number of nodes in the cluster. This allows sufficient compute and memory to be available for the pods to spawn.
29+
To scale node pool on Azure portal, follow these steps:
30+
31+
1. On the Azure portal, open the *AKS cluster*.
32+
1. Go to **Node pools** under **Settings**.
33+
1. Select **Scale node pool**, and then update the *minimum* and *maximum* values on the **Node count range**.
34+
1. Select **Apply**.
35+
36+
### Scenario 2
37+
38+
**Error message**:
39+
40+
```Error
41+
BackupStorageLocation "default" is unavailable: rpc error: code = Unknown desc = azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to https://management.azure.com/subscriptions/e30af180-aa96-4d81-981a-b67570b0d615/resourceGroups/AzureBackupRG_westeurope_1/providers/Microsoft.Storage/storageAccounts/devhayyabackup/listKeys?%24expand=kerb&api-version=2019-06-01: StatusCode=404 -- Original Error: adal: Refresh request failed. Status Code = '404'. Response body: no azure identity found for request clientID 4e95##### REDACTED #####0777`
42+
43+
Endpoint http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&client_id=4e95dcc5-a769-4745-b2d9-
44+
```
45+
46+
**Cause**: When you enable pod-managed identity on your AKS cluster, an *AzurePodIdentityException* named *aks-addon-exception* is added to the *kube-system* namespace. An *AzurePodIdentityException* allows pods with certain labels to access the Azure Instance Metadata Service (IMDS) endpoint without being intercepted by the NMI server.
47+
48+
The extension pods aren't exempt, and require the Azure Active Directory (Azure AD) pod identity to be enabled manually.
49+
50+
**Resolution**: Create *pod-identity* exception in AKS cluster (that works only for *dataprotection-microsoft* namespace and for *not kube-system*). [Learn more](/cli/azure/aks/pod-identity/exception?view=azure-cli-latest&preserve-view=true#az-aks-pod-identity-exception-add).
51+
52+
1. Run the following command:
53+
54+
55+
```azurepowershell-interactive
56+
az aks pod-identity exception add --resource-group shracrg --cluster-name shractestcluster --namespace dataprotection-microsoft --pod-labels app.kubernetes.io/name=dataprotection-microsoft-kubernetes
57+
```
58+
59+
2. To verify *Azurepodidentityexceptions* in cluster, run the following command:
60+
61+
```azurepowershell-interactive
62+
kubectl get Azurepodidentityexceptions --all-namespaces
63+
```
64+
65+
3. To assign the *Storage Account Contributor* role to the extension identity, run the following command:
66+
67+
```azurepowershell-interactive
68+
az role assignment create --assignee-object-id $(az k8s-extension show --name azure-aks-backup --cluster-name aksclustername --resource-group aksclusterresourcegroup --cluster-type managedClusters --query aksAssignedIdentity.principalId --output tsv) --role 'Storage Account Contributor' --scope /subscriptions/subscriptionid/resourceGroups/storageaccountresourcegroup/providers/Microsoft.Storage/storageAccounts/storageaccountname
69+
```
70+
71+
### Scenario 3
72+
73+
**Error message**:
74+
75+
```Error
76+
{"Message":"Error in the getting the Configurations: error {Post \https://centralus.dp.kubernetesconfiguration.azure.com/subscriptions/ subscriptionid /resourceGroups/ aksclusterresourcegroup /provider/managedclusters/clusters/ aksclustername /configurations/getPendingConfigs?api-version=2021-11-01\: dial tcp: lookup centralus.dp.kubernetesconfiguration.azure.com on 10.63.136.10:53: no such host}","LogType":"ConfigAgentTrace","LogLevel":"Error","Environment":"prod","Role":"ClusterConfigAgent","Location":"centralus","ArmId":"/subscriptions/ subscriptionid /resourceGroups/ aksclusterresourcegroup /providers/Microsoft.ContainerService/managedclusters/ aksclustername ","CorrelationId":"","AgentName":"ConfigAgent","AgentVersion":"1.8.14","AgentTimestamp":"2023/01/19 20:24:16"}`
77+
```
78+
**Cause**: Specific FQDN/application rules are required to use cluster extensions in the AKS clusters. [Learn more](/azure/aks/limit-egress-traffic#cluster-extensions).
79+
80+
This error appears due to absence of these FQDN rules because of which configuration information from the Cluster Extensions service wasn't available.
81+
82+
**Resolution**: To resolve the issue, you need to create a *CoreDNS-custom override* for the *DP* endpoint to pass through the public network.
83+
84+
1. To fetch *Existing CoreDNS-custom* YAML in your cluster (save it on your local for reference later), run the following command:
85+
86+
```azurepowershell-interactive
87+
kubectl get configmap coredns-custom -n kube-system -o yaml
88+
```
89+
90+
2. To override mapping for *Central US DP* endpoint to public IP (download the YAML file attached), run the following command:
91+
92+
```azurepowershell-interactive
93+
kubectl apply -f corednsms.yaml
94+
```
95+
96+
3. To force reload `coredns` pods, run the following command:
97+
98+
```azurepowershell-interactive
99+
kubectl delete pod --namespace kube-system -l k8s-app=kube-dns
100+
```
101+
102+
4. To perform `NSlookup` from the *ExtensionAgent* pod to check if *coreDNS-custom* is working, run the following command:
103+
104+
```azurepowershell-interactive
105+
kubectl exec -i -t pod/extension-agent-<pod guid that's there in your cluster> -n kube-system -- nslookup centralus.dp.kubernetesconfiguration.azure.com
106+
```
107+
108+
5. To check logs of the *ExtensionAgent* pod, run the following command:
109+
110+
```azurepowershell-interactive
111+
kubectl logs pod/extension-agent-<pod guid that’s there in your cluster> -n kube-system --tail=200
112+
```
113+
114+
6. Delete and reinstall Backup Extension to initiate backup.
115+
116+
## Next steps
117+
118+
- [About Azure Kubernetes Service (AKS) backup (preview)](azure-kubernetes-service-backup-overview.md)

‎articles/backup/azure-kubernetes-service-cluster-backup-concept.md

Lines changed: 10 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,20 @@
11
---
2-
title: Azure Kubernetes Service (AKS) backup using Azure Backup overview
3-
description: This article explains the concept of Azure Kubernetes Service (AKS) backup using Azure Backup.
2+
title: Azure Kubernetes Service (AKS) backup using Azure Backup prerequisites
3+
description: This article explains the prerequisites for Azure Kubernetes Service (AKS) backup.
44
ms.topic: conceptual
55
ms.service: backup
6-
ms.date: 03/03/2023
6+
ms.date: 03/14/2023
77
author: jyothisuri
88
ms.author: jsuri
99
---
1010

11-
# Overview of Azure Kubernetes Service backup using Azure Backup (preview)
11+
# Prerequisites for Azure Kubernetes Service backup using Azure Backup (preview)
1212

13-
Azure Backup now allows you to back up AKS clusters (cluster resources and persistent volumes attached to the cluster) using a backup extension, which must be installed in the cluster. Backup vault communicates with the cluster via this Backup Extension to perform backup and restore operations.
13+
This article describes the prerequisites for Azure Kubernetes Sercuce (AKS) backup.
1414

15-
## Least privilege security models
15+
Azure Backup now allows you to back up AKS clusters (cluster resources and persistent volumes attached to the cluster) using a backup extension, which must be installed in the cluster. Backup vault communicates with the cluster via this Backup Extension to perform backup and restore operations. Based on the least privileged security model, a Backup vault must have *Trusted Access* enabled to communicate with the AKS cluster.
1616

17-
This section explains the least privilege security models required for a Backup vault (to have Trusted Access enabled) to communicate with the AKS cluster.
18-
19-
### Backup Extension
17+
## Backup Extension
2018

2119
- The extension enables backup and restore capabilities for the containerized workloads and persistent volumes used by the workloads running in AKS clusters.
2220

@@ -28,7 +26,7 @@ This section explains the least privilege security models required for a Backup
2826

2927
Learn [how to manage the operation to install Backup Extension using Azure CLI](azure-kubernetes-service-cluster-manage-backups.md#manage-operations).
3028

31-
### Trusted Access
29+
## Trusted Access
3230

3331
Many Azure services depend on *clusterAdmin kubeconfig* and the *publicly accessible kube-apiserver endpoint* to access AKS clusters. The **AKS Trusted Access** feature enables you to bypass the private endpoint restriction. Without using Microsoft Azure Active Directory (Azure AD) application, this feature enables you to give explicit consent to your system-assigned identity of allowed resources to access your AKS clusters using an Azure resource RoleBinding. The Trusted Access feature allows you to access AKS clusters with different configurations, which aren't limited to private clusters, clusters with local accounts disabled, Azure AD clusters, and authorized IP range clusters.
3432

@@ -38,7 +36,7 @@ For AKS backup, the Backup vault accesses your AKS clusters via Trusted Access t
3836

3937
Learn [how to enable Trusted Access](azure-kubernetes-service-cluster-manage-backups.md#enable-trusted-access).
4038

41-
### AKS Cluster
39+
## AKS Cluster
4240

4341
To enable backup for an AKS cluster, see the following prerequisites: .
4442

@@ -85,6 +83,7 @@ Also, as part of the backup and restore operations, the following roles are assi
8583
8684
## Next steps
8785

86+
- [About Azure Kubernetes Service backup (preview)](azure-kubernetes-service-backup-overview.md)
8887
- [Supported scenarios for Azure Kubernetes Service cluster backup (preview)](azure-kubernetes-service-cluster-backup-support-matrix.md)
8988
- [Back up Azure Kubernetes Service cluster (preview)](azure-kubernetes-service-cluster-backup.md)
9089
- [Restore Azure Kubernetes Service cluster (preview)](azure-kubernetes-service-cluster-restore.md)

‎articles/backup/index.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ landingContent:
7676
- text: Immutable vault (preview)
7777
url: backup-azure-immutable-vault-concept.md
7878
- text: Azure Kubernetes Service backup (preview)
79-
url: azure-kubernetes-service-cluster-backup-concept.md
79+
url: azure-kubernetes-service-backup-overview.md
8080
- text: Azure Blob vaulted backups (preview)
8181
url: blob-backup-overview.md
8282
# Card

‎articles/backup/toc.yml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -129,7 +129,11 @@
129129
- name: Architecture
130130
href: azure-backup-architecture-for-sap-hana-backup.md
131131
- name: Azure Kubernetes Service backup
132-
href: azure-kubernetes-service-cluster-backup-concept.md
132+
items:
133+
- name: Overview
134+
href: azure-kubernetes-service-backup-overview.md
135+
- name: Prerequisites
136+
href: azure-kubernetes-service-cluster-backup-concept.md
133137
- name: Azure Blob backup
134138
href: blob-backup-overview.md
135139
- name: Monitor and Alerts
@@ -693,6 +697,8 @@
693697
href: backup-azure-system-state-troubleshoot.md
694698
- name: Archive tier
695699
href: troubleshoot-archive-tier.md
700+
- name: Azure Kubernetes Service
701+
href: azure-kubernetes-service-backup-troubleshoot.md
696702
- name: Reference
697703
items:
698704
- name: Archived release notes

0 commit comments

Comments
 (0)