Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 95 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1005,6 +1005,101 @@ talos_siderolabs_discovery_service_enabled = true
For more details, refer to the [official Talos discovery guide](https://www.talos.dev/latest/talos-guides/discovery/).
</details>

<!-- Talos OOM Handler -->
<details>
<summary><b>Talos OOM Handler Configuration</b></summary>

Talos 1.12+ includes a userspace Out-of-Memory (OOM) handler that is always enabled by default with built-in configuration. This handler provides early detection of memory pressure and helps prevent machine lock-up due to out-of-memory conditions, which is especially important for single-node clusters or when scheduling pods on control plane nodes.

While the Linux kernel's OOM killer only activates when completely out of memory (at which point the system may be unresponsive), the Talos userspace OOM handler proactively monitors memory pressure and can take action earlier to maintain system stability.

#### Default Behavior

The OOM handler uses [Pressure Stall Information (PSI)](https://docs.kernel.org/accounting/psi.html) metrics to detect memory pressure and Common Expression Language (CEL) expressions to determine when and what to kill:

- **Trigger**: Activates when full memory pressure (10s average) exceeds 12% AND is increasing AND at least 500ms have passed since the last OOM event
- **Ranking**: Prioritizes killing pods without memory limits first, specifically BestEffort pods (highest priority), then Burstable pods, while protecting Guaranteed pods and system services

#### Customization

This module allows you to customize the OOM handler behavior if the defaults don't suit your workload requirements:

```hcl
# Custom OOM configuration enabled
talos_custom_oom_enabled = true

# Custom trigger - this expression defines when to trigger OOM action.
talos_custom_oom_trigger_expression = "memory_full_avg10 > 12.0 && d_memory_full_avg10 > 0.0 && time_since_trigger > duration(\"500ms\")"

# Custom ranking - this expression defines how to rank cgroups for OOM handler.
talos_custom_oom_cgroup_ranking_expression = "memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable: 0.5, Guaranteed: 0.0, Podruntime: 0.0, System: 0.0}[class] * double(memory_current.orValue(0u))"

# Custom sample interval - how often should the trigger expression be evaluated.
talos_custom_oom_sample_interval = "100ms"
```

You can override any combination of these settings. The module validates that at least one custom field is provided when `talos_custom_oom_enabled = true`.

For more details, refer to the [Talos OOM Handler documentation](https://docs.siderolabs.com/talos/v1.12/configure-your-talos-cluster/system-configuration/oom) and [OOMConfig reference](https://docs.siderolabs.com/talos/v1.12/reference/configuration/runtime/oomconfig).
</details>

<!-- Talos Directory Volumes -->
<details>
<summary><b>Talos Directory Volumes</b></summary>

Talos 1.12+ supports creating using directories on the EPHEMERAL partition as storage volumes. These volumes are automatically mounted at `/var/mnt/<name>` and provide persistent storage across pod restarts without requiring additional block devices or partitions.

This feature is particularly useful for workloads that need host directories, such as:

- **Local Path Provisioner**: Kubernetes dynamic volume provisioner using local storage
- **Shared cache directories**: Persistent cache storage for applications
- **Build artifacts**: CI/CD pipelines requiring shared build output
- **Logging/metrics collection**: Centralized log or metric storage from workloads

#### Configuration

Directory volumes can be configured separately for each node type (control plane, workers, autoscaler):

```hcl
# Control plane directory volumes
control_plane_directory_volumes = ["local-storage", "cache-data"]

# Worker directory volumes
worker_directory_volumes = ["local-storage", "build-cache"]

# Cluster autoscaler directory volumes (optional)
cluster_autoscaler_directory_volumes = ["temp-storage"]
```

#### Example: Local Path Provisioner

Deploy the Kubernetes Local Path Provisioner to use directory volumes for dynamic PV provisioning:

```hcl
# Define directory volumes for worker nodes
worker_directory_volumes = ["local-storage"]
```

The directory `/var/mnt/local-storage` will be created on all worker nodes and can be used by the Local Path Provisioner for dynamic volume provisioning.

#### Important Considerations

**Data Persistence:**

- ⚠️ **Data loss on node failure**: If a node fails or is replaced, all data in directory volumes on that node is permanently lost
- ⚠️ **Ephemeral partition wipe**: Recreating or resetting a node wipes the EPHEMERAL partition, destroying all directory volume data
- For critical data requiring high availability, use Hetzner Cloud Volumes (via Hcloud CSI) or Longhorn distributed storage instead
- Directory volumes inherit encryption from the EPHEMERAL partition

**Storage Limitations:**

- **Shared capacity**: All directory volumes share the EPHEMERAL partition's total capacity
- **No quotas**: Cannot enforce per-directory storage limits
- **No filesystem isolation**: Directory-type volumes don't provide filesystem-level isolation

For more details, refer to the [Talos UserVolumeConfig documentation](https://docs.siderolabs.com/talos/v1.12/reference/configuration/runtime/uservolumeconfig).
</details>

<!-- Kubernetes RBAC -->
<details>
<summary><b>Kubernetes RBAC</b></summary>
Expand Down
31 changes: 20 additions & 11 deletions oidc.tf
Original file line number Diff line number Diff line change
@@ -1,11 +1,20 @@
locals {
# Kubernetes OIDC configuration
talos_kube_oidc_configuration = var.oidc_enabled ? {
"oidc-issuer-url" = var.oidc_issuer_url
"oidc-client-id" = var.oidc_client_id
"oidc-username-claim" = var.oidc_username_claim
"oidc-groups-claim" = var.oidc_groups_claim
"oidc-groups-prefix" = var.oidc_groups_prefix
} : {}

# Collect all unique k8s cluster roles used across OIDC group mappings
k8s_cluster_roles = var.oidc_enabled ? toset(flatten([
kube_cluster_roles = var.oidc_enabled ? toset(flatten([
for group_mapping in var.oidc_group_mappings : group_mapping.cluster_roles
])) : toset([])

# Collect all unique k8s roles used across OIDC group mappings (grouped by namespace/role)
k8s_roles = var.oidc_enabled ? {
kube_roles = var.oidc_enabled ? {
for role_key, role_info in merge([
for group_mapping in var.oidc_group_mappings : {
for role in group_mapping.roles : "${role.namespace}/${role.name}" => role
Expand All @@ -14,8 +23,8 @@ locals {
} : {}

# Create one ClusterRoleBinding per cluster role with all groups as subjects
cluster_role_binding_manifests = [
for cluster_role in local.k8s_cluster_roles : yamlencode({
kube_cluster_role_binding_manifests = [
for cluster_role in local.kube_cluster_roles : yamlencode({
apiVersion = "rbac.authorization.k8s.io/v1"
kind = "ClusterRoleBinding"
metadata = {
Expand All @@ -42,8 +51,8 @@ locals {
]

# Create one RoleBinding per role with all groups as subjects
role_binding_manifests = [
for role_key, role_info in local.k8s_roles : yamlencode({
kube_role_binding_manifests = [
for role_key, role_info in local.kube_roles : yamlencode({
apiVersion = "rbac.authorization.k8s.io/v1"
kind = "RoleBinding"
metadata = {
Expand Down Expand Up @@ -71,14 +80,14 @@ locals {
]

# Combine all OIDC manifests
oidc_manifests = var.oidc_enabled ? concat(
local.cluster_role_binding_manifests,
local.role_binding_manifests
kube_oidc_manifests = var.oidc_enabled ? concat(
local.kube_cluster_role_binding_manifests,
local.kube_role_binding_manifests
) : []

# Final manifest
oidc_manifest = length(local.oidc_manifests) > 0 ? {
kube_oidc_manifest = length(local.kube_oidc_manifests) > 0 ? {
name = "kube-oidc-rbac"
contents = join("\n---\n", local.oidc_manifests)
contents = trimspace(join("\n---\n", local.kube_oidc_manifests))
} : null
}
9 changes: 6 additions & 3 deletions rbac.tf
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
locals {
# Generate Kubernetes RBAC manifests
rbac_manifests = concat(
kube_rbac_manifests = concat(

# Kubernetes namespaced roles
[for role in var.rbac_roles : yamlencode({
apiVersion = "rbac.authorization.k8s.io/v1"
Expand All @@ -15,6 +16,7 @@ locals {
verbs = rule.verbs
}]
})],

# Kubernetes cluster roles
[for role in var.rbac_cluster_roles : yamlencode({
apiVersion = "rbac.authorization.k8s.io/v1"
Expand All @@ -30,9 +32,10 @@ locals {
})]
)

rbac_manifest = length(local.rbac_manifests) > 0 ? {
# Final manifest
kube_rbac_manifest = length(local.kube_rbac_manifests) > 0 ? {
name = "kube-rbac"
contents = join("\n---\n", local.rbac_manifests)
contents = trimspace(join("\n---\n", local.kube_rbac_manifests))
} : null
}

2 changes: 1 addition & 1 deletion talos.tf
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ resource "talos_machine_bootstrap" "this" {
resource "terraform_data" "synchronize_manifests" {
triggers_replace = [
nonsensitive(sha1(jsonencode(local.talos_inline_manifests))),
nonsensitive(sha1(jsonencode(local.talos_manifests))),
nonsensitive(sha1(jsonencode(local.talos_remote_manifests))),
]

provisioner "local-exec" {
Expand Down
Loading