Autoscaling doesn't work #1266
-
DescriptionHello, I’ve been struggling all day, I just can’t start autoscaling. Set node group draining-node-pool size from 0 to 0, expected delta 0 Kube.tf filelocals {
hcloud_token = ""
}
module "kube-hetzner" {
providers = {
hcloud = hcloud
}
hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
source = "kube-hetzner/kube-hetzner/hcloud"
cluster_name = "prod"
automatically_upgrade_os = false
automatically_upgrade_k3s = false
ssh_public_key = file("")
ssh_private_key = file("")
network_region = "eu-central"
enable_rancher = true
rancher_hostname = ""
rancher_install_channel = "stable"
rancher_bootstrap_password = ""
initial_k3s_channel = "v1.25"
control_plane_nodepools = [
{
name = "control-plane-hel1",
server_type = "cpx31",
location = "hel1",
labels = [],
taints = [],
count = 1
backups = true
}
]
agent_nodepools = [
{
name = "agent-heavy",
server_type = "cpx31",
location = "hel1",
labels = [],
taints = [],
count = 1
backups = true
}
]
autoscaler_nodepools = [
{
name = "autoscaled-peak-load"
server_type = "cpx21" // Choose a server type with sufficient resources
location = "hel1" // Specify the desired location
min_nodes = 0 // Set a minimum number of nodes
max_nodes = 5 // Set a maximum number to scale up to during peak load
#labels = {
# "node.kubernetes.io/role": "peak-workloads"
#}
#taints = [{
# key: "node.kubernetes.io/role"
# value: "peak-workloads"
# effect: "NoExecute"
#}]
}
]
#load_balancer_type = "lb11"
#load_balancer_location = "hel1"
cluster_autoscaler_image = "registry.k8s.io/autoscaling/cluster-autoscaler"
cluster_autoscaler_version = "v1.29.0"
cluster_autoscaler_log_level = 5
cluster_autoscaler_log_to_stderr = true
cluster_autoscaler_stderr_threshold = "INFO"
#cluster_autoscaler_extra_args = [
# "--ignore-daemonsets-utilization=true",
# "--enforce-node-group-min-size=true",
#]
cni_plugin = "cilium"
cilium_version = "v1.14.0"
cilium_routing_mode = "native"
cilium_egress_gateway_enabled = true
}
provider "hcloud" {
token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}
terraform {
required_version = ">= 1.5.0"
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = ">= 1.43.0"
}
}
}
output "kubeconfig" {
value = module.kube-hetzner.kubeconfig
sensitive = true
}
variable "hcloud_token" {
sensitive = true
default = ""
}ScreenshotsNo response PlatformLinux |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 7 replies
-
|
@homergleason Remove those, we maintain a fork pending fixes related to Hetzner are shipped in a future version. So just delete those lines to use our own fork: cluster_autoscaler_image = "registry.k8s.io/autoscaling/cluster-autoscaler"
cluster_autoscaler_version = "v1.29.0"Then just make sure to give it enough load, with such powerful nodes, you may need to give way bigger ressource requests than in the example below: apiVersion: apps/v1
kind: Deployment
metadata:
name: force-scale-up
spec:
replicas: 1
selector:
matchLabels:
app: force-scale-up
template:
metadata:
labels:
app: force-scale-up
spec:
containers:
- name: busybox
image: busybox
args:
- /bin/sh
- -c
- "while true; do echo 'Forcing scale up...'; sleep 60; done"
resources:
requests:
cpu: 2000m # Requesting a high amount of CPU to force scale up
memory: 4Gi # Requesting a high amount of memory to force scale up |
Beta Was this translation helpful? Give feedback.
-
|
Thank you! It works! 🔥🔥🔥 But how to make the same setting with pods? I load the server with requests, the response time increases up to 10 seconds, and new nodes are not created. |
Beta Was this translation helpful? Give feedback.
-
|
If I were in your position, I would employ a Horizontal Pod Autoscaler (HPA), which monitors the CPU usage of nodes and scales up when a specified threshold is reached. This, combined with the appropriate resource requests, will prompt the Node Autoscaler to provision new nodes, enabling the HPA to scale up and alleviate the load on your requests. Ideally, your system should have the capacity to accommodate one or two additional pods to manage sudden spikes in traffic, while the Node Autoscaler works to create additional capacity. More context below: To complement the node autoscaler setup, you can implement a Horizontal Pod Autoscaler (HPA) to automatically scale the number of pods in your deployment based on observed CPU utilization or other metrics. Here’s how you can set it up:
By setting up the HPA, your application can automatically adjust the number of running pods based on actual demand. Initially, keep an eye on the HPA’s behavior and adjust your thresholds as necessary to ensure optimal performance and resource usage. Remember, for the HPA to make accurate scaling decisions, it's important that your pods have proper resource requests set, as this informs the Kubernetes scheduler of your application's resource needs and influences HPA scaling decisions. |
Beta Was this translation helpful? Give feedback.
-
|
Hi there,
This may be related to this part of the template? |
Beta Was this translation helpful? Give feedback.
-
|
@brain-dev-null how did you fix this? I'm running into the same issue. |
Beta Was this translation helpful? Give feedback.
-
|
The fastest but also most insecure solution I found was to add a line to the packer template which sets the SELinux state to Keep in mind that this was just the way I confirmed my suspiscion and is definitely not recommendable for production environments. |
Beta Was this translation helpful? Give feedback.
-
|
Working on a fix, it was indeed SELinux policy not being applied. Keep you posted ASAP. |
Beta Was this translation helpful? Give feedback.
-
|
Should be fixed in v2.13.3. |
Beta Was this translation helpful? Give feedback.
Should be fixed in v2.13.3.