feat: add AKS support (#90)

Author: jackfrancis

README.md

@@ -6,7 +6,7 @@ Put simply: "identical", "peer" nodes that exist _in duplicate_ for the purpose
 
 # Status of Project
 
-The kamino set of tools are currently approaching a v1.0 stable release, tested against Kubernetes running on Azure with VMSS-backed node pools, on clusters built with the [AKS Engine](https://github.com/Azure/aks-engine) tool.
+The kamino set of tools are currently approaching a v1.0 stable release, tested against Kubernetes running on Azure with VMSS-backed node pools, on clusters built with the [AKS Engine](https://github.com/Azure/aks-engine) tool. You may also run it as a proof of concept on your AKS cluster, see [here](docs/AKS.md) for more information.
 
 More status [here][status].
 

docs/AKS.md

@@ -0,0 +1,86 @@
+# Kamino on AKS
+
+You can test-drive kamino on your AKS cluster to evaluate the potential value of an optimized OS disk image, with the following, important caveat:
+
+The AKS managed service will eventually overwrite changes that kamino makes to a node pool's underlying VMSS resource:
+
+- kamino-delivered changes to `virtualMachineProfile.storageProfile.imageReference.id` and `virtualMachineProfile.storageProfile.imageReference.resourceGroup` will be reverted the standard AKS OS image maintained by the AKS managed service
+- kamino updates to the `virtualMachineProfile.extensionProfile.extensions` array will be reverted
+
+## How to run kamino on your (non-production!) AKS cluster
+
+1. Get the resource group name that your cluster's VMSS are running in. E.g.:
+
+```sh
+$ az aks show -n aks-kamino -g aks-kamino | jq -r .nodeResourceGroup
+MC_aks-kamino_aks-kamino_westus2
+```
+
+2. Get the managed identity resource for your node VMs. E.g.:
+
+```sh
+$ az identity list -g MC_aks-kamino_aks-kamino_westus2
+[
+  {
+    "clientId": "<clientId value>",
+    "clientSecretUrl": "<clientSecretUrl value>",
+    "id": "<id value>",
+    "location": "westus2",
+    "name": "aks-kamino-agentpool",
+    "principalId": "<principalId value>",
+    "resourceGroup": "MC_aks-kamino_aks-kamino_westus2",
+    "tags": {},
+    "tenantId": "<tenantId value>",
+    "type": "Microsoft.ManagedIdentity/userAssignedIdentities"
+  }
+]
+```
+
+Now you can give your cluster node pool managed identity resource contributor access to the resource group using the actual value of `principalId` from above (substitute `<principalId value>` below with the actual value):
+
+```sh
+$ az role assignment create --assignee <principalId value> --role 'Contributor' --scope /subscriptions/<subscription ID that cluster is in>/resourcegroups/MC_aks-kamino_aks-kamino_westus2
+{
+  "canDelegate": null,
+  "condition": null,
+  "conditionVersion": null,
+  "description": null,
+  "id": "<id value>",
+  "name": "<name value>",
+  "principalId": "<principalId value>",
+  "principalName": "<principalName value>",
+  "principalType": "ServicePrincipal",
+  "resourceGroup": "MC_aks-kamino_aks-kamino_westus2",
+  "roleDefinitionId": "<roleDefinitionId value>",
+  "roleDefinitionName": "Contributor",
+  "scope": "/subscriptions/<subscription ID that cluster is in>/resourceGroups/MC_aks-kamino_aks-kamino_westus2",
+  "type": "Microsoft.Authorization/roleAssignments"
+}
+```
+
+This additional access granted to your node pool managed identity allows the kamino runtime access to create the necessary infra in your cluster resource group.
+
+Now you can target a particular node running on your cluster, make an OS image snapshot from its OS image, and then use that OS image as a Shared Image Gallery image to build new VMSS VMs from. This will replicate any pre-pulled container images onto any newly scaled out nodes, as well as remove the need to run any startup scripts. This can demonstrably improve reliability and responsiveness of new node scale out operations.
+
+```sh
+$ k get nodes
+NAME                                STATUS   ROLES   AGE     VERSION
+aks-nodepool1-68550425-vmss000000   Ready    agent   5h9m    v1.21.7
+aks-nodepool2-35877414-vmss000000   Ready    agent   5h      v1.21.7
+aks-nodepool2-35877414-vmss000002   Ready    agent   4h42m   v1.21.7
+```
+
+From the above set of nodes let's choose `aks-nodepool2-35877414-vmss000000` from nodepool2 to build a new image from, and to use as a base when building any new nodes in nodepool2:
+
+```sh
+$ helm install --repo https://jackfrancis.github.io/kamino/ \
+  update-nodepool2-os-image \
+  vmss-prototype --namespace default \
+  --set kamino.targetNode=aks-nodepool2-35877414-vmss000000
+```
+
+The above command will schedule the kamino runtime as a pod on any schedulable node other than the target node, and do the needful work.
+
+Again, at present this solution is not designed for production AKS clusters, as the managed service will overwrite the changes. But have fun testing!
+
+A more detailed walkthrough of how kamino works is [here](../helm/vmss-prototype/walkthrough.md).

helm/vmss-prototype/Chart.yaml

@@ -1,7 +1,7 @@
 apiVersion: v1
 description: A Helm chart for the Kamino vmss-prototype pattern image generator
 name: vmss-prototype
-version: 0.0.16
+version: 0.0.17
 maintainers:
   - name: Michael Sinz
     email: msinz@microsoft.com

helm/vmss-prototype/templates/vmss-prototype.yaml

@@ -1,6 +1,5 @@
 {{- if hasKey .Values "kamino" -}}
 
-# Pick our job name here
 {{- $jobName := printf "%s-%s" .Values.kamino.name "status" -}}
 {{- if hasKey .Values.kamino "targetVMSS" -}}
 {{- $jobName = printf "%s-%s" .Values.kamino.name "autoupdate" -}}
@@ -9,7 +8,46 @@
 {{- $jobName = printf "%s-%s" .Values.kamino.name (substr 0 (int (sub (len .Values.kamino.targetNode) 6)) .Values.kamino.targetNode) -}}
 {{- end -}}
 {{- end -}}
+{{- $rbacResourceName := printf "kamino-%s" $jobName -}}
 
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: {{ $rbacResourceName }}
+  namespace: {{ .Release.Namespace }}
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: {{ $rbacResourceName }}
+rules:
+- apiGroups: [""]
+  resources: ["pods/eviction"]
+  verbs: ["create"]
+- apiGroups: [""]
+  resources: ["pods"]
+  verbs: ["get", "list", "delete", "create"]
+- apiGroups: [""]
+  resources: ["nodes"]
+  verbs: ["get", "patch"]
+- apiGroups: ["apps"]
+  resources: ["statefulsets", "namespaces", "daemonsets", "replicasets"]
+  verbs: ["get", "list"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: {{ $rbacResourceName }}
+  namespace: {{ .Release.Namespace }}
+subjects:
+- kind: ServiceAccount
+  name: {{ $rbacResourceName }}
+  namespace: {{ .Release.Namespace }}
+roleRef:
+  kind: ClusterRole
+  name: {{ $rbacResourceName }}
+  apiGroup: rbac.authorization.k8s.io
+---
 # If cronjob is enabled and we must have a targetVMSS...
 {{- if .Values.kamino.auto.cronjob.enabled }}
 apiVersion: batch/v1beta1
@@ -63,6 +101,7 @@ spec:
             {{- end }}
             {{- end }}
         spec:
+          serviceAccountName: {{ $rbacResourceName }}
           restartPolicy: Never
 
           {{- if hasKey .Values.kamino.container "pullSecret" }}
@@ -146,10 +185,6 @@ spec:
                 {{- end }}
 
               env:
-                # We use the in-cluster kubeconfig
-                - name: KUBECONFIG
-                  value: /.kubeconfig
-
                 # This gets mapped here since the node has cloud local CA bundles we need
                 - name: REQUESTS_CA_BUNDLE
                   value: /etc/ssl/certs/ca-certificates.crt
@@ -178,9 +213,6 @@ spec:
                 - name: kubectl
                   mountPath: /usr/bin/kubectl
                   readOnly: true
-                - name: kubeconfig
-                  mountPath: /.kubeconfig
-                  readOnly: true
                 - name: host-crt
                   mountPath: /etc/ssl/certs/ca-certificates.crt
                   readOnly: true
@@ -197,11 +229,6 @@ spec:
                 path: /usr/local/bin/kubectl
                 type: File
 
-            - name: kubeconfig
-              hostPath:
-                path: /var/lib/kubelet/kubeconfig
-                type: File
-
             - name: host-crt
               hostPath:
                 path: /etc/ssl/certs/ca-certificates.crt

vmss-prototype/vmss-prototype

@@ -92,7 +92,7 @@ def log_stdout_stderr(stdout, stderr, exit_code, masq=None):
 
 def run(command, timeout=None, env=None, shell=False, check=True, quiet=False, dry_run=False, show_cmd=True, cwd=None,
         echo=False, masq=None, stdin='', comment=None, log_timing=False, log_stdout_on_error=True, log_stderr_on_error=True,
-        retries=1, retry_wait_duration_func=lambda num_retry: 2 ** (num_retry + 2), log_final_error=False, log_retry_errors=False,
+        retries=1, retry_wait_duration_func=lambda num_retry: 2 ** (num_retry + 2), log_final_error=True, log_retry_errors=False,
         retry_func=lambda stdout, stderr, exit_code: True):
     """
     Run the given command in the shell
@@ -116,7 +116,7 @@ def run(command, timeout=None, env=None, shell=False, check=True, quiet=False, d
     :param log_stderr_on_error: If true (default), on error, the stderr of the command is logged
     :param retries: Number of retries before giving up due to errors
     :param retry_wait_duration_func: A function to get the wait in seconds for a particular retry attempt. Default delay is exponential backoff with initial delay of 8 seconds
-    :param log_final_error: Log the final retry error even if check==False
+    :param log_final_error: Log the final retry error even if check==False - defaults to True
     :param log_retry_errors: Log the errors for retries
     :param retry_func: Function to call after an error.   Passed in 3-tuple (stdin, stdout, exit_code), return is if retry should happen (False if no retry)
     :return: 3-tuple (stdout as string, stderr as string and exit code as int)
@@ -1085,7 +1085,7 @@ def vmss_prototype_update(sub_args):
             # provisioning bits on it (prototype) we can remove all extensions
             # But...  We want to keep the AKS Engine-identifying extension for telemetry reasons
             # It is a no-op code wise but gives counts
-            if 'vmss-computeAksLinuxBilling' not in extension['name']:
+            if 'AKSLinuxBilling' not in extension['name']:
                 delete_extensions = True
                 run(az(['vmss', 'extension', 'delete'], subscription) + [
                     '--resource-group', resource_group,