OTC Addon configs from the OTC console in Terraform

[rb090]

Hi dear community 👋,

I'm writing Terraform scripts for a CCE resource, including the node pool and addons.

I have a question about configuring addons, specifically the autoscaler addon, and I would really appreciate your help.

When looking at the autoscaler settings in the OTC console (Auto Scaling Setting in CCE), I see that certain attributes are set:

Am I correct in assuming that these settings need to be configured within the Terraform resource opentelekomcloud_cce_addon_v3 when declaring the autoscaler addon?

So basically like described in section CCE addon management with data sources in the OTC Terraform docs.

When I imported the opentelekomcloud_cce_addon_v3 resource for the existing autoscaler addon configuration in OTC, the values field was not populated in the output.

Additionally, why do we need to specify cceEndpoint and ecsEndpoint in basic? The cluster_id is already specified in a opentelekomcloud_cce_addon_v3 resource. And what should be provided for region_name? Should it be something like eu-de? Or should it contain also the project name (eg. eu-de_my_project)?

Lastly, can you please tell me where exactly do I need to specify the autoscaler rules for a node pool resource in the OTC console?

Thanks in advance for your help! 😊

[rb090]

After doing some research, I implemented the following autoscaler configuration for our CCE setup and added comments based on my understanding. It would be helpful if the OTC documentation please provide a bit more context around the available flags and their usage.

# ------------------------------------------------------------------------------
# data: autoscaler_template
# ------------------------------------------------------------------------------
# Retrieves metadata for the "autoscaler" Add-On template, including image version,
# container image repository address (`swr_addr`) and the system user (`swr_user`) used to pull images from the SWR registry.
# ------------------------------------------------------------------------------
data "opentelekomcloud_cce_addon_template_v3" "autoscaler_template" {
  addon_version = "1.30.51"
  addon_name    = "autoscaler"
}

resource "opentelekomcloud_cce_addon_v3" "autoscaler_addon" {
  cluster_id       = opentelekomcloud_cce_cluster_v3.cce_cluster.id
  template_name    = data.opentelekomcloud_cce_addon_template_v3.autoscaler_template.addon_name
  template_version = data.opentelekomcloud_cce_addon_template_v3.autoscaler_template.addon_version

  values {
    basic = {
      # Endpoint of the CCE API in the target region
      "cceEndpoint" = "https://cce.${var.region}.otc.t-systems.com"
      # Endpoint of the ECS (Elastic Cloud Server) API in the target region
      "ecsEndpoint" = "https://ecs.${var.region}.otc.t-systems.com"
      # Target region (for us yet `eu-de)
      "region" = var.region
      # 💡 For more details regarding those 2 attributes, please check attribute docs in `opentelekomcloud_cce_addon_v3.everest_addon` resource.
      "swr_addr" = data.opentelekomcloud_cce_addon_template_v3.autoscaler_template.swr_addr
      "swr_user" = data.opentelekomcloud_cce_addon_template_v3.autoscaler_template.swr_user
    }
    custom = {
      # ID of the CCE cluster the autoscaler Add-on needs to be installed on,
      # 💡 ideally on our above defined `opentelekomcloud_cce_cluster_v3.cce_cluster`.
      "cluster_id" = opentelekomcloud_cce_cluster_v3.cce_cluster.id
      # Maximum total CPU cores across all nodes managed by the autoscaler
      "coresTotal" = 160
      # How the autoscaler chooses which node pool to scale up.
      # `priority` uses defined node pool priorities to prefer scaling some node pools before others.
      "expander" = "priority"
      # Verbosity level of the autoscaler logs, level 4 is a trace level providing very detailed logs and prints almost everything.
      "logLevel" = 4
      # Max number of nodes that can be deleted in one autoscaler cycle.
      # This value sets the maximum number of empty (idle, unneeded) nodes the autoscaler may remove in one operation.
      # 💡 Prevents large scale disruptions due to aggressive scale-down.
      "maxEmptyBulkDeleteFlag" = 5
      # Timeout in minutes for a node to be provisioned
      "maxNodeProvisionTime" = 15
      # Maximum number of nodes the autoscaler is allowed to manage
      "maxNodesTotal" = 5
      # Total memory in GB available across autoscaled nodes
      "memoryTotal" = 200
      # Delay in minutes after a new node is added before scale-down starts
      "scaleDownDelayAfterAdd" = 10
      # Delay after a node is deleted before the next scale-down check
      "scaleDownDelayAfterDelete" = 11
      # Delay after a failed scale-down before retrying
      "scaleDownDelayAfterFailure" = 3
      # Removes nodes that have been underutilized for a certain period, defined in `scaleDownUnneededTime` property.
      # A node is considered for scale-down if it has low resource utilization (CPU/mem) and no critical pods are scheduled on it.
      "scaleDownEnabled" = true
      # Minimum time a node must be unneeded before being eligible for scale-down
      "scaleDownUnneededTime" = 5
      # CPU utilization threshold below which a node is considered underutilized
      "scaleDownUtilizationThreshold" = 0.3
      # CPU usage threshold to trigger scale-up, 0-1 -> 0 - 100%
      "scaleUpCpuUtilizationThreshold" = 0.9
      # Memory usage threshold to trigger scale-up
      "scaleUpMemUtilizationThreshold" = 0.9
      # Enables scale-up based on pending pods that cannot be scheduled due to lack of capacity
      "scaleUpUnscheduledPodEnabled" = false
      # Enables scale-up based on high resource utilization (CPU, memory)
      "scaleUpUtilizationEnabled" = true
      # The project ID of the OTC project where the CCE the addon is installed for is placed in.
      # It is required by the autoscaler during installation and setup to interact with project-level resources.
      # If not provided, Terraform plan/apply will fail with a 400 Bad Request error.
      "tenant_id" = var.otc_project_id
      # Interval in minutes before checking again whether unremovable nodes can be scaled down
      "unremovableNodeRecheckTimeout" = 5
    }
  }
}

I’m still not sure where to set the autoscaler rules for a node pool in the outside the OTC console in a opentelekomcloud_cce_addon_v3 resource and I would really appreciate a bit of help with this please.

[anton-sidelnikov]

Hi @rb090, you’re right, our documentation doesn’t cover everything in full detail. However, you can find more information here:
https://docs.otc.t-systems.com/cloud-container-engine/api-ref/apis/add-on_instance_parameters/cce_cluster_autoscaler.html#cce-02-0411

I noticed that the some basic parameters are now marked as optional, so we’ll be updating the resource shortly.

As for your second question: you can set autoscaling rules for a node pool by clicking the “Auto Scaling” button.

[rb090]

Thank you for your answer on this @anton-sidelnikov 🙌. For my 2nd question, is there a chance to setup the autoscaling rules also over Terraform within the opentelekomcloud_cce_addon_v3 resource?

[anton-sidelnikov]

Thank you for your answer on this @anton-sidelnikov 🙌. For my 2nd question, is there a chance to setup the autoscaling rules also over Terraform within the opentelekomcloud_cce_addon_v3 resource?

autoscaling rules need to set in opentelekomcloud_cce_node_pool_v3

resource "opentelekomcloud_cce_node_pool_v3" "node_pool" {
  cluster_id         = data.opentelekomcloud_cce_cluster_v3.cluster.id
  name               = "opentelekomcloud-cce-node-pool"
  os                 = "EulerOS 2.9"
  flavor             = "s2.large.2"
  initial_node_count = 1
  availability_zone  = "az"
  key_pair           = "kp"
  runtime            = "containerd"

  scale_enable             = false
  min_node_count           = 1
  max_node_count           = 3
  scale_down_cooldown_time = 6
  priority                 = 1
...
}

[rb090]

Perfect, thank you so much for your help and the clarification @anton-sidelnikov 🙌💗. That really helped a lot to get a better understanding.