|
| 1 | +--- |
| 2 | +title: "Hardening Guide - Scheduler Configuration" |
| 3 | +description: > |
| 4 | + Information about how to make the Kubernetes scheduler more secure. |
| 5 | +content_type: concept |
| 6 | +weight: 90 |
| 7 | +--- |
| 8 | + |
| 9 | +<!-- overview --> |
| 10 | +The Kubernetes {{< glossary_tooltip text="scheduler" term_id="kube-scheduler" >}} is |
| 11 | +one of the critical components of the |
| 12 | +{{< glossary_tooltip text="control plane" term_id="control-plane" >}}. |
| 13 | + |
| 14 | +This document covers how to improve the security posture of the Scheduler. |
| 15 | + |
| 16 | +A misconfigured scheduler can have security implications. |
| 17 | +Such a scheduler can target specific nodes and evict the workloads or applications that are sharing the node and its resources. |
| 18 | +This can aid an attacker with a [Yo-Yo attack](https://arxiv.org/abs/2105.00542): an attack on a vulnerable autoscaler. |
| 19 | + |
| 20 | +<!-- body --> |
| 21 | +## kube-scheduler configuration |
| 22 | + |
| 23 | +### Scheduler authentication & authorization command line options |
| 24 | + |
| 25 | +When setting up authentication configuration, it should be made sure that kube-scheduler's authentication remains consistent with kube-api-server's authentication. |
| 26 | +If any request has missing authentication headers, |
| 27 | +the [authentication should happen through the kube-api-server allowing all authentication to be consistent in the cluster](/docs/tasks/extend-kubernetes/configure-aggregation-layer/#original-request-username-and-group). |
| 28 | + |
| 29 | +- `authentication-kubeconfig`: Make sure to provide a proper kubeconfig so that the scheduler can retrieve authentication configuration options from the API Server. This kubeconfig file should be protected with strict file permissions. |
| 30 | +- `authentication-tolerate-lookup-failure`: Set this to `false` to make sure the scheduler _always_ looks up its authentication configuration from the API server. |
| 31 | +- `authentication-skip-lookup`: Set this to `false` to make sure the scheduler _always_ looks up its authentication configuration from the API server. |
| 32 | +- `authorization-always-allow-paths`: These paths should respond with data that is appropriate for anonymous authorization. Defaults to `/healthz,/readyz,/livez`. |
| 33 | +- `profiling`: Set to `false` to disable the profiling endpoints which are provide debugging information but which should not be enabled on production clusters as they present a risk of denial of service or information leakage. The `--profiling` argument is deprecated and can now be provided through the [KubeScheduler DebuggingConfiguration](https://kubernetes.io/docs/reference/config-api/kube-scheduler-config.v1/#DebuggingConfiguration). Profiling can be disabled through the kube-scheduler config by setting `enableProfiling` to `false`. |
| 34 | +- `requestheader-client-ca-file`: Avoid passing this argument. |
| 35 | + |
| 36 | + |
| 37 | +### Scheduler networking command line options |
| 38 | + |
| 39 | +- `bind-address`: In most cases, the kube-scheduler does not need to be externally accessible. Setting the bind address to `localhost` is a secure practice. |
| 40 | +- `permit-address-sharing`: Set this to `false` to disable connection sharing through `SO_REUSEADDR`. `SO_REUSEADDR` can lead to reuse of terminated connections that are in `TIME_WAIT` state. |
| 41 | +- `permit-port-sharing`: Default `false`. Use the default unless you are confident you understand the security implications. |
| 42 | + |
| 43 | + |
| 44 | +### Scheduler TLS command line options |
| 45 | + |
| 46 | +- `tls-cipher-suites`: Always provide a list of preferred cipher suites. This ensures encryption never happens with insecure cipher suites. |
| 47 | + |
| 48 | + |
| 49 | +## Scheduling configurations for custom schedulers |
| 50 | + |
| 51 | +When using custom schedulers based on the Kubernetes scheduling code, cluster administrators need to be careful with |
| 52 | +plugins that use the `queueSort`, `prefilter`, `filter`, or `permit` [extension points](/docs/reference/scheduling/config/#extension-points). |
| 53 | +These extension points control various stages of a scheduling process, and the wrong configuration can impact the kube-scheduler's behavior in your cluster. |
| 54 | + |
| 55 | +### Key considerations |
| 56 | + |
| 57 | +- Exactly one plugin that uses the `queueSort` extension point can be enabled at a time. Any plugins that use `queueSort` should be scrutinized. |
| 58 | +- Plugins that implement the `prefilter` or `filter` extension point can potentially mark all nodes as unschedulable. This can bring scheduling of new pods to a halt. |
| 59 | +- Plugins that implement the `permit` extension point can prevent or delay the binding of a Pod. Such plugins should be thoroughly reviewed by the cluster administrator. |
| 60 | + |
| 61 | +When using a plugin that is not one of the [default plugins](/docs/reference/scheduling/config/#scheduling-plugins), consider disabling the `queueSort`, `filter` and `permit` extension points as follows: |
| 62 | + |
| 63 | +```yaml |
| 64 | +apiVersion: kubescheduler.config.k8s.io/v1 |
| 65 | +kind: KubeSchedulerConfiguration |
| 66 | +profiles: |
| 67 | + - schedulerName: my-scheduler |
| 68 | + plugins: |
| 69 | + # Disable specific plugins for different extension points |
| 70 | + # You can disable all plugins for an extension point using "*" |
| 71 | + queueSort: |
| 72 | + disabled: |
| 73 | + - name: "*" # Disable all queueSort plugins |
| 74 | + # - name: "PrioritySort" # Disable specific queueSort plugin |
| 75 | + filter: |
| 76 | + disabled: |
| 77 | + - name: "*" # Disable all filter plugins |
| 78 | + # - name: "NodeResourcesFit" # Disable specific filter plugin |
| 79 | + permit: |
| 80 | + disabled: |
| 81 | + - name: "*" # Disables all permit plugins |
| 82 | + # - name: "TaintToleration" # Disable specific permit plugin |
| 83 | +``` |
| 84 | +This creates a scheduler profile ` my-custom-scheduler`. |
| 85 | +Whenever the `.spec` of a Pod does not have a value for `.spec.schedulerName`, the kube-scheduler runs for that Pod, |
| 86 | +using its main configuration, and default plugins. |
| 87 | +If you define a Pod with `.spec.schedulerName` set to `my-custom-scheduler`, the kube-scheduler runs but with a custom configuration; in that custom configuration, |
| 88 | +the `queueSort`, `filter` and `permit` extension points are disabled. |
| 89 | +If you use this KubeSchedulerConfiguration, and don't run any custom scheduler, |
| 90 | +and you then define a Pod with `.spec.schedulerName` set to `nonexistent-scheduler` |
| 91 | +(or any other scheduler name that doesn't exist in your cluster), no events would be generated for a pod. |
| 92 | + |
| 93 | +## Disallow labeling nodes |
| 94 | + |
| 95 | +A cluster administrator should ensure that cluster users cannot label the nodes. |
| 96 | +A malicious actor can use `nodeSelector` to schedule workloads on nodes where those workloads should not be present. |
0 commit comments