feat: experimental spot node pools (#451)

* feat: Add configuration options for spot node pools

Signed-off-by: Carus Kyle <[email protected]>

* docs: update community vars to include spot node pool feature

Signed-off-by: Carus Kyle <[email protected]>

* fix: update the spot node pool var names

Signed-off-by: Carus Kyle <[email protected]>

* fix: fix to pass community var to real vars

Signed-off-by: Carus Kyle <[email protected]>

* docs: Update docs/community/community_config_vars.md

Co-authored-by: Chris Miller <[email protected]>

* docs: add details on spot node pools and risks

Signed-off-by: Carus Kyle <[email protected]>

* DCO Remediation Commit for Carus Kyle <[email protected]>

I, Carus Kyle <[email protected]>, hereby add my Signed-off-by to this commit: 62600d3

Signed-off-by: Carus Kyle <[email protected]>

* docs: add link to azure documentation on spot node pools

Signed-off-by: Carus Kyle <[email protected]>

* fix: add validation for the new spot variables

Signed-off-by: Carus Kyle <[email protected]>

* chore: correct captialization for spot node variable validations

Signed-off-by: saschjmil <[email protected]>

---------

Signed-off-by: Carus Kyle <[email protected]>
Signed-off-by: saschjmil <[email protected]>
Co-authored-by: Chris Miller <[email protected]>
Co-authored-by: chjmil <[email protected]>

Author: Carus11

docs/community/community_config_vars.md

@@ -12,12 +12,18 @@ Community-contributed configuration variables are listed in the tables below. Th
 <a name="spot_nodes"></a>
 ## Spot Nodes
 
-Here is some information about spot nodes.
+Spot Nodes allow you to run Azure Kubernetes Service (AKS) workloads on low-cost, surplus compute capacity offered by Azure. These Spot Virtual Machines (VMs) can significantly reduce infrastructure costs, especially for workloads that are fault-tolerant or batch-oriented or temporary lab environments. However, Spot VMs can be preempted (evicted) by Azure at any time if the capacity is needed elsewhere, which makes them less suitable for critical or stateful workloads.
 
-Here is a warning about why they might cause issues.
+For further information, see https://learn.microsoft.com/en-us/azure/aks/spot-node-pool
 
-Here is a table with the variables you would use to configure them
+> [!CAUTION] 
+> Spot nodes can be evicted with little notice. They are best used for non-production, non-critical workloads or for scenarios where cost savings outweigh the risk of eviction. This is a configuration not supported by SAS Technical Support. Monitor eviction rates and ensure your workloads can tolerate sudden node loss.
+
+To enable a Spot node pool in your AKS cluster using this module, configure the community-maintained variables listed below. These options customize the behavior of the Spot node pool and its underlying virtual machine scale set.
 
 | Name | Description | Type | Default | Release Added | Notes |
 | :--- | ---: | ---: | ---: | ---: | ---: |
-| contrib_enable_spot_nodes | Enable spot nodes | bool | false | 10.3.0 | |
+| community_priority | (Optional) The Priority for Virtual Machines within the Virtual Machine Scale Set that powers this Node Pool. Possible values are Regular and Spot. Defaults to Regular. Changing this forces a new resource to be created. | string | `Regular` | 10.3.0 | Changing this to Spot enables the Spot node pool functionality |
+| community_eviction_policy | (Optional) The Eviction Policy which should be used for Virtual Machines within the Virtual Machine Scale Set powering this Node Pool. Possible values are Deallocate and Delete. Changing this forces a new resource to be created. | string | `Delete` | 10.3.0 | |
+| community_spot_max_price | (Optional) The maximum price you're willing to pay in USD per Virtual Machine. Valid values are -1 (the current on-demand price for a Virtual Machine) or a positive value with up to five decimal places. Changing this forces a new resource to be created. | string | `-1` | 10.3.0 | |
+

main.tf

@@ -216,6 +216,10 @@ module "node_pools" {
   host_encryption_enabled      = var.aks_cluster_enable_host_encryption
   tags                         = var.tags
   linux_os_config              = each.value.linux_os_config
+  community_priority           = each.value.community_priority 
+  community_eviction_policy    = each.value.community_eviction_policy
+  community_spot_max_price     = each.value.community_spot_max_price
+
 }
 
 # https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/postgresql_flexible_server

modules/aks_node_pool/main.tf

@@ -25,6 +25,9 @@ resource "azurerm_kubernetes_cluster_node_pool" "autoscale_node_pool" {
   node_taints                  = var.node_taints
   orchestrator_version         = var.orchestrator_version
   tags                         = var.tags
+  priority                     = var.community_priority
+  eviction_policy              = var.community_eviction_policy
+  spot_max_price               = var.community_spot_max_price
 
   lifecycle {
     ignore_changes = [node_count]
@@ -58,7 +61,9 @@ resource "azurerm_kubernetes_cluster_node_pool" "static_node_pool" {
   node_taints                  = var.node_taints
   orchestrator_version         = var.orchestrator_version
   tags                         = var.tags
-
+  priority                     = var.community_priority
+  eviction_policy              = var.community_eviction_policy
+  spot_max_price               = var.community_spot_max_price
   linux_os_config {
     sysctl_config {
       vm_max_map_count = try(var.linux_os_config.sysctl_config.vm_max_map_count,null)

modules/aks_node_pool/variables.tf

@@ -116,6 +116,51 @@ variable "proximity_placement_group_id" {
   default     = ""
 }
 
+variable "community_priority" {
+  description = "(Optional) The Priority for Virtual Machines within the Virtual Machine Scale Set that powers this Node Pool. Possible values are Regular and Spot. Defaults to Regular. Changing this forces a new resource to be created."
+  type        = string
+  default     = "Regular"
+  validation {
+    condition     = contains(["Regular", "Spot"], var.community_priority)
+    error_message = "ERROR: community_priority must be either 'Regular' or 'Spot'."
+  }
+}
+
+variable "community_eviction_policy" {
+  description = "(Optional) The Eviction Policy which should be used for Virtual Machines within the Virtual Machine Scale Set powering this Node Pool. Possible values are Deallocate and Delete. Changing this forces a new resource to be created."
+  type        = string
+  default     = null
+  validation {
+    condition     = var.community_eviction_policy == null || (
+                      contains(["Delete", "Deallocate"], coalesce(var.community_eviction_policy, "Delete"))
+                    )
+    error_message = "ERROR: When specified, community_eviction_policy must be either 'Delete' or 'Deallocate'."
+  }
+  validation {
+    condition     = var.community_eviction_policy == null || coalesce(var.community_priority, "Regular") == "Spot"
+    error_message = "ERROR: community_eviction_policy can only be specified when community_priority is set to 'Spot'."
+  }
+}
+
+
+variable "community_spot_max_price" {
+  description = "(Optional) The maximum price you're willing to pay in USD per Virtual Machine. Valid values are -1 (the current on-demand price for a Virtual Machine) or a positive value with up to five decimal places. Changing this forces a new resource to be created."
+  type        = string
+  default     = null
+  validation {
+    condition     = var.community_spot_max_price == null || (
+                      can(tonumber(coalesce(var.community_spot_max_price, "-1"))) && 
+                      (tonumber(coalesce(var.community_spot_max_price, "-1")) == -1 || 
+                       tonumber(coalesce(var.community_spot_max_price, "-1")) > 0)
+                    )
+    error_message = "ERROR: When specified, community_spot_max_price must be either '-1' or a positive number with up to five decimal places."
+  }
+  validation {
+    condition     = var.community_spot_max_price == null || coalesce(var.community_priority, "Regular") == "Spot"
+    error_message = "ERROR: community_spot_max_price can only be specified when community_priority is set to 'Spot'."
+  }
+}
+
 variable "linux_os_config"{
   description = "Specifications of linux os config. Changing this forces a new resource to be created."
   type = object({
@@ -124,4 +169,4 @@ variable "linux_os_config"{
         }))
       })
   default = {}
-}
+}

variables.tf

@@ -568,11 +568,15 @@ variable "node_pools" {
     max_pods     = string
     node_taints  = list(string)
     node_labels  = map(string)
+    community_priority     = optional(string, "Regular")
+    community_eviction_policy = optional(string)
+    community_spot_max_price = optional(string)
     linux_os_config = optional(object({
       sysctl_config = optional(object({
         vm_max_map_count = optional(number)
       }))
     }))
+
   }))
 
   default = {