Terraform module for connecting a AKS cluster to CAST AI

Requirements

Terraform 0.13+

Using the module

A module to create Azure role and a service principal that can be used to connect to CAST AI

Requires castai/castai, hashicorp/azurerm, hashicorp/azuread, hashicorp/helm providers to be configured.

The required parameters can be provided manually or alternatively can be easily acquired from your AKS cluster resource or Azure RM subscription data source.

module "castai-aks-cluster" {
  source = "castai/aks/castai"

  aks_cluster_name    = var.aks_cluster_name
  aks_cluster_region  = var.aks_cluster_region
  node_resource_group = azurerm_kubernetes_cluster.example.node_resource_group
  resource_group      = azurerm_kubernetes_cluster.example.resource_group_name

  delete_nodes_on_disconnect = true

  subscription_id = data.azurerm_subscription.current.subscription_id
  tenant_id       = data.azurerm_subscription.current.tenant_id

  default_node_configuration = module.castai-aks-cluster.castai_node_configurations["default"]

  node_configurations = {
    default = {
      disk_cpu_ratio = 25
      subnets        = [azurerm_subnet.internal.id]
      tags           = {
        "node-config" : "default"
      }
    }
  }
  node_templates = {
    spot_tmpl = {
      configuration_id = module.castai-aks-cluster.castai_node_configurations["default"]

      should_taint = true

      custom_labels = {
        custom-label-key-1 = "custom-label-value-1"
        custom-label-key-2 = "custom-label-value-2"
      }

      custom_taints = [
        {
          key = "custom-taint-key-1"
          value = "custom-taint-value-1"
        },
        {
          key = "custom-taint-key-2"
          value = "custom-taint-value-2"
        }
      ]

      constraints = {
        fallback_restore_rate_seconds = 1800
        spot = true
        use_spot_fallbacks = true
        min_cpu = 4
        max_cpu = 100
        instance_families = {
          exclude = ["standard_DPLSv5"]
        }
        compute_optimized_state = "disabled"
        storage_optimized_state = "disabled"
      }
    }
  }

  autoscaler_settings = {
    enabled                                 = true
    node_templates_partial_matching_enabled = false

    unschedulable_pods = {
      enabled = true
    }

    node_downscaler = {
      enabled = true

      empty_nodes = {
        enabled = true
      }

      evictor = {
        aggressive_mode           = false
        cycle_interval            = "5s10s"
        dry_run                   = false
        enabled                   = true
        node_grace_period_minutes = 10
        scoped_mode               = false
      }
    }

    cluster_limits = {
      enabled = true

      cpu = {
        max_cores = 20
        min_cores = 1
      }
    }
  }

  workload_scaling_policies = {
    default = {
      apply_type        = "IMMEDIATE"
      management_option = "MANAGED"

      cpu = {
        function                 = "QUANTILE"
        args                     = ["0.9"]
        overhead                 = 0.15
        look_back_period_seconds = 172800
        min                      = 0.1
        max                      = 2.0
      }

      memory = {
        function                 = "MAX"
        overhead                 = 0.35
        look_back_period_seconds = 172800

        limit = {
          type = "NO_LIMIT"
        }
      }

      assignment_rules = {
        rules = [
          {
            namespace = {
              names = ["default", "kube-system"]
            }
          },
          {
            workload = {
              gvk = ["Deployment", "StatefulSet"]
              labels_expressions = [
                {
                  key      = "region"
                  operator = "NotIn"
                  values   = ["eu-west-1", "eu-west-2"]
                },
                {
                  key      = "helm.sh/chart"
                  operator = "Exists"
                }
              ]
            }
          }
        ]
      }

      startup = {
        period_seconds = 300
      }

      predictive_scaling = {
        cpu = {
          enabled = true
        }
      }
    }
  }
}

Migrating from 2.x.x to 3.x.x

Version 3.x.x changes:

Removed custom_label attribute in castai_node_template resource. Use custom_labels instead.

Old configuration:

module "castai-aks-cluster" {
  node_templates = {
    spot_tmpl = {
      custom_label = {
        key = "custom-label-key-1"
        value = "custom-label-value-1"
      }
    }
  }
}

New configuration:

module "castai-aks-cluster" {
  node_templates = {
    spot_tmpl = {
      custom_labels = {
        custom-label-key-1 = "custom-label-value-1"
      }
    }
  }
}

Migrating from 3.x.x to 4.x.x

Version 4.x.x changed:

Removed compute_optimized and storage_optimized attributes in castai_node_template resource, constraints object. Use compute_optimized_state and storage_optimized_state instead.

Old configuration:

module "castai-aks-cluster" {
  node_templates = {
    spot_tmpl = {
      constraints = {
        compute_optimized = false
        storage_optimized = true
      }
    }
  }
}

New configuration:

module "castai-aks-cluster" {
  node_templates = {
    spot_tmpl = {
      constraints = {
        compute_optimized_state = "disabled"
        storage_optimized_state = "enabled"
      }
    }
  }
}

Migrating from 5.0.x to 5.2.x

Version 5.2.x changed:

Deprecated autoscaler_policies_json attribute. Use autoscaler_settings instead.

Old configuration:

module "castai-aks-cluster" {
  autoscaler_policies_json = <<-EOT
    {
        "enabled": true,
        "unschedulablePods": {
            "enabled": true
        },
        "nodeDownscaler": {
            "enabled": true,
            "emptyNodes": {
                "enabled": true
            },
            "evictor": {
                "aggressiveMode": false,
                "cycleInterval": "5m10s",
                "dryRun": false,
                "enabled": true,
                "nodeGracePeriodMinutes": 10,
                "scopedMode": false
            }
        },
        "nodeTemplatesPartialMatchingEnabled": false,
        "clusterLimits": {
            "cpu": {
                "maxCores": 20,
                "minCores": 1
            },
            "enabled": true
        }
    }
  EOT
}

New configuration:

module "castai-aks-cluster" {
  autoscaler_settings = {
    enabled                                 = true
    node_templates_partial_matching_enabled = false

    unschedulable_pods = {
      enabled = true
    }

    node_downscaler = {
      enabled = true

      empty_nodes = {
        enabled = true
      }

      evictor = {
        aggressive_mode           = false
        cycle_interval            = "5m10s"
        dry_run                   = false
        enabled                   = true
        node_grace_period_minutes = 10
        scoped_mode               = false
      }
    }

    cluster_limits = {
      enabled = true

      cpu = {
        max_cores = 20
        min_cores = 1
      }
    }
  }
}

Migrating from 9.x.x to 10.x.x

Version 10.x.x removes deprecated autoscaler fields that have been moved to node templates.

Removed:

autoscaler_policies_json - Use autoscaler_settings instead
autoscaler_settings.unschedulable_pods.custom_instances_enabled - No longer needed
autoscaler_settings.unschedulable_pods.headroom - Use low-priority placeholder workloads instead
autoscaler_settings.unschedulable_pods.headroom_spot - Use low-priority placeholder workloads instead
autoscaler_settings.unschedulable_pods.node_constraints - Use node_templates constraints instead
autoscaler_settings.spot_instances - Use node_templates constraints instead

Migration guide:

Old configuration:

module "castai-aks-cluster" {
  source = "castai/aks/castai"

  autoscaler_settings = {
    enabled = true

    unschedulable_pods = {
      enabled                  = true
      custom_instances_enabled = true

      headroom = {
        enabled           = true
        cpu_percentage    = 10
        memory_percentage = 10
      }

      node_constraints = {
        min_cpu_cores = 4
        max_cpu_cores = 32
      }
    }

    spot_instances = {
      enabled = true
      spot_backups = {
        enabled = true
      }
    }
  }
}

New configuration:

module "castai-aks-cluster" {
  source = "castai/aks/castai"

  autoscaler_settings = {
    enabled = true

    unschedulable_pods = {
      enabled = true
    }
  }

  node_templates = {
    default_by_castai = {
      configuration_id = module.castai-aks-cluster.castai_node_configurations["default"]
      is_default       = true

      constraints = {
        min_cpu            = 4
        max_cpu            = 32
        spot               = true
        use_spot_fallbacks = true
      }
    }
  }
}

# For headroom: Deploy low-priority placeholder workloads
# See: https://docs.cast.ai/docs/autoscaler-faq#how-can-i-maintain-cluster-headroom

Key changes:

Node constraints (min_cpu_cores, max_cpu_cores, min_ram_mib, max_ram_mib) are now configured in node_templates.constraints as min_cpu, max_cpu, min_memory, max_memory
Spot instance settings (spot, use_spot_fallbacks) are now configured in node_templates.constraints
The default node template is named default_by_castai
For headroom functionality, deploy low-priority placeholder workloads as described in the CAST AI documentation

Examples

Usage examples are located in terraform provider repo

Requirements

Name	Version
terraform	>= 0.13
azuread	~> 3
azurerm	>= 3.7.0
castai	>= 8.7
helm	>= 3.0.0
null	~> 3

Providers

Name	Version
azuread	~> 3
azurerm	>= 3.7.0
castai	>= 8.7
helm	>= 3.0.0
null	~> 3

Modules

Name	Source	Version
castai_omni_cluster	castai/omni-cluster/castai	~> 2.0

Resources

Name	Type
azuread_application.castai	resource
azuread_application_password.castai	resource
azuread_service_principal.castai	resource
azurerm_federated_identity_credential.this	resource
azurerm_role_assignment.castai_additional_resource_groups	resource
azurerm_role_assignment.castai_node_resource_group	resource
azurerm_role_assignment.castai_resource_group	resource
azurerm_role_definition.castai	resource
azurerm_user_assigned_identity.this	resource
castai_aks_cluster.castai_cluster	resource
castai_autoscaler.castai_autoscaler_policies	resource
castai_node_configuration.this	resource
castai_node_configuration_default.this	resource
castai_node_template.this	resource
castai_workload_scaling_policy.this	resource
helm_release.castai_agent	resource
helm_release.castai_ai_optimizer_proxy	resource
helm_release.castai_ai_optimizer_proxy_self_managed	resource
helm_release.castai_cluster_controller	resource
helm_release.castai_cluster_controller_self_managed	resource
helm_release.castai_evictor	resource
helm_release.castai_evictor_ext	resource
helm_release.castai_evictor_self_managed	resource
helm_release.castai_kvisor	resource
helm_release.castai_kvisor_self_managed	resource
helm_release.castai_pod_mutator	resource
helm_release.castai_pod_mutator_self_managed	resource
helm_release.castai_pod_pinner	resource
helm_release.castai_pod_pinner_self_managed	resource
helm_release.castai_spot_handler	resource
helm_release.castai_workload_autoscaler	resource
helm_release.castai_workload_autoscaler_exporter	resource
helm_release.castai_workload_autoscaler_exporter_self_managed	resource
helm_release.castai_workload_autoscaler_self_managed	resource
null_resource.wait_for_cluster	resource
azuread_client_config.current	data source
azurerm_kubernetes_cluster.aks	data source
azurerm_kubernetes_cluster.castai	data source
castai_impersonation_service_account.this	data source

Inputs

Name	Description	Type	Default	Required
additional_resource_groups	n/a	`list(string)`	`[]`	no
agent_values	List of YAML formatted string values for agent helm chart	`list(string)`	`[]`	no
agent_version	Version of castai-agent helm chart. If not provided, latest version will be used.	`string`	`null`	no
ai_optimizer_values	List of YAML formatted string with ai-optimizer values	`list(string)`	`[]`	no
ai_optimizer_version	Version of castai-ai-optimizer helm chart. Default latest	`string`	`null`	no
aks_cluster_name	Name of the cluster to be connected to CAST AI.	`string`	n/a	yes
aks_cluster_region	Region of the AKS cluster	`string`	n/a	yes
api_url	URL of alternative CAST AI API to be used during development or testing	`string`	`"https://api.cast.ai"`	no
authentication_method	Authentication method for CAST AI. Use 'client_secret' for service principal with password, or 'workload_identity' for federated identity credential	`string`	`"client_secret"`	no
autoscaler_settings	Optional Autoscaler policy definitions to override current autoscaler settings	`any`	`null`	no
azuread_owners	A set of object IDs of principals that will be granted ownership of the Azure AD service principal and application. Defaults to current user.	`list(string)`	`null`	no
castai_api_token	Optional CAST AI API token created in console.cast.ai API Access keys section. Used only when `wait_for_cluster_ready` is set to true	`string`	`""`	no
castai_components_labels	Optional additional Kubernetes labels for CAST AI pods	`map(any)`	`{}`	no
castai_components_sets	Optional additional 'set' configurations for every CAST AI Helm release.	`map(string)`	`{}`	no
cluster_controller_values	List of YAML formatted string values for cluster-controller helm chart	`list(string)`	`[]`	no
cluster_controller_version	Version of castai-cluster-controller helm chart. If not provided, latest version will be used.	`string`	`null`	no
default_node_configuration	ID of the default node configuration	`string`	`""`	no
default_node_configuration_name	Name of the default node configuration	`string`	`""`	no
delete_nodes_on_disconnect	Optionally delete Cast AI created nodes when the cluster is destroyed	`bool`	`false`	no
evictor_ext_values	List of YAML formatted string with evictor-ext values	`list(string)`	`[]`	no
evictor_ext_version	Version of castai-evictor-ext chart. Default latest	`string`	`null`	no
evictor_values	List of YAML formatted string values for evictor helm chart	`list(string)`	`[]`	no
evictor_version	Version of castai-evictor chart. If not provided, latest version will be used.	`string`	`null`	no
grpc_url	gRPC endpoint used by pod-pinner	`string`	`"grpc.cast.ai:443"`	no
http_proxy	Address to use for proxying http requests from CAST AI components running directly on nodes.	`string`	`null`	no
https_proxy	Address to use for proxying https requests from CAST AI components running directly on nodes.	`string`	`null`	no
install_ai_optimizer	Optional flag for installation of AI Optimizer (https://docs.cast.ai/docs/getting-started-ai)	`bool`	`false`	no
install_omni	Optional flag for installing Omni capability	`bool`	`false`	no
install_pod_mutator	Optional flag for installation of pod mutator	`bool`	`false`	no
install_security_agent	Optional flag for installation of security agent (Kvisor - https://docs.cast.ai/docs/kvisor)	`bool`	`false`	no
install_workload_autoscaler	Optional flag for installation of workload autoscaler (https://docs.cast.ai/docs/workload-autoscaling-configuration)	`bool`	`false`	no
install_workload_autoscaler_exporter	Optional flag for installation of workload autoscaler exporter (custom metrics exporter)	`bool`	`false`	no
kvisor_controller_extra_args	⚠️ DEPRECATED: use kvisor_values instead (see example: https://github.com/castai/terraform-provider-castai/tree/master/examples/aks/aks_cluster_with_security/castai.tf ). Extra arguments for the kvisor controller. Optionally enable kvisor to lint Kubernetes YAML manifests, scan workload images and check if workloads pass CIS Kubernetes Benchmarks as well as NSA, WASP and PCI recommendations.	`map(string)`	{ "image-scan-enabled": "true", "kube-bench-enabled": "true", "kube-linter-enabled": "true" }	no
kvisor_grpc_addr	CAST AI Kvisor optimized GRPC API address	`string`	`"kvisor.prod-master.cast.ai:443"`	no
kvisor_values	List of YAML formatted string values for kvisor helm chart, see example: https://github.com/castai/terraform-provider-castai/tree/master/examples/aks/aks_cluster_with_security/castai.tf	`list(string)`	`[]`	no
kvisor_version	Version of kvisor chart. If not provided, latest version will be used.	`string`	`null`	no
kvisor_wait	Wait for kvisor chart to finish release	`bool`	`true`	no
no_proxy	List of addresses to skip proxying requests from CAST AI components running directly on nodes. Used with http_proxy and https_proxy.	`list(string)`	`[]`	no
node_configurations	Map of AKS node configurations to create	`any`	`{}`	no
node_resource_group	n/a	`string`	n/a	yes
node_templates	Map of node templates to create	`any`	`{}`	no
organization_id	DEPRECATED (required only for pod mutator v0.0.25 and older): CAST AI Organization ID	`string`	`""`	no
pod_mutator_values	List of YAML formatted string values for pod-mutator helm chart	`list(string)`	`[]`	no
pod_mutator_version	Version of castai-pod-mutator helm chart. Default latest	`string`	`null`	no
pod_pinner_values	List of YAML formatted string values for agent helm chart	`list(string)`	`[]`	no
pod_pinner_version	Version of pod-pinner helm chart. Default latest	`string`	`null`	no
resource_group	n/a	`string`	n/a	yes
self_managed	Whether CAST AI components' upgrades are managed by a customer; by default upgrades are managed CAST AI central system. WARNING: changing this after the module was created is not supported.	`bool`	`false`	no
spot_handler_values	List of YAML formatted string values for spot-handler helm chart	`list(string)`	`[]`	no
spot_handler_version	Version of castai-spot-handler helm chart. If not provided, latest version will be used.	`string`	`null`	no
subscription_id	Azure subscription ID	`string`	n/a	yes
tenant_id	n/a	`string`	n/a	yes
wait_for_cluster_ready	Wait for cluster to be ready before finishing the module execution, this option requires `castai_api_token` to be set	`bool`	`false`	no
workload_autoscaler_exporter_values	List of YAML formatted string with castai-workload-autoscaler-exporter values	`list(string)`	`[]`	no
workload_autoscaler_exporter_version	Version of castai-workload-autoscaler-exporter helm chart. Default latest	`string`	`null`	no
workload_autoscaler_values	List of YAML formatted string with cluster-workload-autoscaler values	`list(string)`	`[]`	no
workload_autoscaler_version	Version of castai-workload-autoscaler helm chart. Default latest	`string`	`null`	no
workload_scaling_policies	Map of workload scaling policies to create	`any`	`{}`	no

Outputs

Name	Description
castai_node_configurations	Map of node configurations ids by name
castai_node_templates	Map of node template by name
cluster_id	CAST.AI cluster id, which can be used for accessing cluster data using API
organization_id	CAST.AI organization id of the cluster

Name		Name	Last commit message	Last commit date
Latest commit History 136 Commits
.github		.github
examples/basic		examples/basic
tests		tests
.DS_Store		.DS_Store
.gitignore		.gitignore
CODEOWNERS		CODEOWNERS
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
iam.tf		iam.tf
locals.tf		locals.tf
main.tf		main.tf
outputs.tf		outputs.tf
variables.tf		variables.tf
versions.tf		versions.tf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Terraform module for connecting a AKS cluster to CAST AI

Requirements

Using the module

Migrating from 2.x.x to 3.x.x

Migrating from 3.x.x to 4.x.x

Migrating from 5.0.x to 5.2.x

Migrating from 9.x.x to 10.x.x

Examples

Requirements

Providers

Modules

Resources

Inputs

Outputs

About

Uh oh!

Releases 103

Packages

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

Terraform module for connecting a AKS cluster to CAST AI

Requirements

Using the module

Migrating from 2.x.x to 3.x.x

Migrating from 3.x.x to 4.x.x

Migrating from 5.0.x to 5.2.x

Migrating from 9.x.x to 10.x.x

Examples

Requirements

Providers

Modules

Resources

Inputs

Outputs

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 103

Packages 0

Uh oh!

Uh oh!

Contributors

Uh oh!

Languages

Packages