Add page Block Services with ExternalIPs #52631
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
That uses a VAP to explain how a user may be able to block specific Services with ExternalIPs
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
language/en
adrianmoisey
force-pushed
the
service-external-ips
branch
from
65c39be to
118bb0a
Compare
✅ Pull request preview available for checkingBuilt without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Pull request preview available for checkingBuilt without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify project configuration. |
| This document shares how to control how Services with ExternalIPs are managed within your cluster. | ||
|
|
||
| An ExternalIP is a powerful tool that could be used for [malicious intent](https://www.cvedetails.com/cve/CVE-2020-8554/). | ||
|
|
||
| Any user who can create a Service with ExternalIPs could: | ||
|
|
||
| - intercept other users' outbound traffic to arbitrary IPs. | ||
| - could (non-deterministically) steal other users' inbound traffic to their own ExternalIPs. |
There was a problem hiding this comment.
nit: avoid using UpperCamelCase for externalIPs; in our style guide, we use otherwise unstyled UpperCamelCase for API kinds (eg StatefulSet, Service).
There was a problem hiding this comment.
We need some (style-guide-approved) way of being explicit that we are referring to the externalIPs feature and not just the vague concept of "external IPs", which can easily be misinterpreted. (eg if this point was rewritten as "to their own external IPs", it would be easy to assume it meant cluster-external IPs as in the previous bullet point).
There was a problem hiding this comment.
Use the field capitalization: externalIPs. You can also come up with new jargon if you write it in italics. Example from elsewhere: adding a taint to a Node lets…
However, I couldn't think of any jargon that would work.
For example:
A Kubernetes Service with
externalIPsset lets you expose Pods running in your cluster to the outside network, without using a Gateway, an Ingress, or a load balancer integration. Using a Gateway or a Service oftype: LoadBalanceris usually a better way to expose Pods and their listening ports. In fact, usingexternalIPsis rare in platforms built to use Kubernetes architecture.To use
.spec.externalIPswith a Service, either you as a cluster administrator need a way to configure additional network interfaces on the relevant node(s), or you need to use a controller that automates doing the same thing.The problem with
.spec.externalIPsis that, unfortunately, someone with access to create this kind of Service can abuse…
|
|
||
| <!-- steps --> | ||
|
|
||
| ## Kubernetes Service ExternalIP Policies |
There was a problem hiding this comment.
| ## Kubernetes Service ExternalIP Policies | |
| ## Service external IP address policies for Kubernetes |
|
|
||
| ## Kubernetes Service ExternalIP Policies | ||
|
|
||
| Cluster administrators can implement policies to control the creation and modification of Services with ExternalIPs within the cluster. This allows for centralized management of the allowed ExternalIPs used for Services and helps prevent unintended or conflicting configurations. Kubernetes provides mechanisms like Validating Admission Policies to enforce these rules. |
There was a problem hiding this comment.
| Cluster administrators can implement policies to control the creation and modification of Services with ExternalIPs within the cluster. This allows for centralized management of the allowed ExternalIPs used for Services and helps prevent unintended or conflicting configurations. Kubernetes provides mechanisms like Validating Admission Policies to enforce these rules. | |
| As a cluster administrator, you can implement policies to control the creation and modification of Services with external IP addresses within the cluster. | |
| This allows for centralized management of the allowed external IP addresses that can be used for Services, | |
| and helps prevent unintended or conflicting configurations. | |
| Kubernetes provides mechanisms such as [ValidatingAdmissionPolicies](/docs/reference/access-authn-authz/validating-admission-policy/) that | |
| you can use to enforce these rules. |
nit: I would also move this paragraph to the introduction, since the page is specifically about solving this with ValidatingAdmissionPolicy.
There was a problem hiding this comment.
I would also move this paragraph to the introduction, since the page is specifically about solving this with ValidatingAdmissionPolicy.
(If you add a section mentioning the admission controller (which I'm pretty sure we don't document anywhere else other than the list-of-all-admission-controllers) then that wouldn't apply.)
There was a problem hiding this comment.
In that case, reword it and move the reworded thing.
| resources: ["services"] | ||
| variables: | ||
| - name: allowed | ||
| expression: "['192.0.2.0/24', '2001:db8::/64']" |
There was a problem hiding this comment.
Needs a comment: change this to your actual allowed IP address range.
| apiVersion: admissionregistration.k8s.io/v1 | ||
| kind: ValidatingAdmissionPolicy | ||
| metadata: | ||
| name: "allow-specific-users-to-manage-externalips" |
There was a problem hiding this comment.
Consider adding a kubernetes.io/description annotation; I see that as a good practice.
| apiVersion: admissionregistration.k8s.io/v1 | ||
| kind: ValidatingAdmissionPolicy | ||
| metadata: | ||
| name: "allow-specific-externalips" |
There was a problem hiding this comment.
Consider adding a kubernetes.io/description annotation; I see that as a good practice.
|
|
||
| Cluster administrators can implement policies to control the creation and modification of Services with ExternalIPs within the cluster. This allows for centralized management of the allowed ExternalIPs used for Services and helps prevent unintended or conflicting configurations. Kubernetes provides mechanisms like Validating Admission Policies to enforce these rules. | ||
|
|
||
| ### Allowing only specific ExternalIPs within a certain IP range to be created |
There was a problem hiding this comment.
| ### Allowing only specific ExternalIPs within a certain IP range to be created | |
| ## Restrict Service external IP addresses to permitted address ranges |
| validationActions: [Deny, Audit] | ||
| ``` | ||
|
|
||
| ### Restricting which users/groups may create/update Services with ExternalIPs |
There was a problem hiding this comment.
| ### Restricting which users/groups may create/update Services with ExternalIPs | |
| ## Restrict which users or groups may specify external IP addresses for Services |
|
|
||
| This document shares how to control how Services with ExternalIPs are managed within your cluster. | ||
|
|
||
| An ExternalIP is a powerful tool that could be used for [malicious intent](https://www.cvedetails.com/cve/CVE-2020-8554/). |
There was a problem hiding this comment.
| An ExternalIP is a powerful tool that could be used for [malicious intent](https://www.cvedetails.com/cve/CVE-2020-8554/). | |
| The ability to [set an external IP address for a Service](/docs/concepts/services-networking/service/#external-ips) could be misused as a way for | |
| an otherwise unprivileged user to intercept traffic associated with that IP address. | |
| See | |
| [CVE-2020-8554](https://www.cvedetails.com/cve/CVE-2020-8554/) for more details. |
?
nit: also update https://k8s.io/docs/concepts/services-networking/service/#external-ips to link to this new task page.
Aside: if we fix #46623 (which will create a page for each known Kubernetes vulnerability), we can link to that page instead.
|
|
||
| <!-- overview --> | ||
|
|
||
| This document shares how to control how Services with ExternalIPs are managed within your cluster. |
There was a problem hiding this comment.
| This document shares how to control how Services with ExternalIPs are managed within your cluster. | |
| This document explains a way to control how {{< glossary_tooltip text="Services" term_id="service" >}} with external IP address(es) are managed within your cluster. |
k8s-ci-robot
added
the
lgtm
|
LGTM label has been added. Git tree hash: cec916167c8f71b098f93b303c3be48590db9fdc
|
|
/sig network |
k8s-ci-robot
added
the
sig/network
|
|
||
| Any user who can create a Service with ExternalIPs could: | ||
|
|
||
| - intercept other users' outbound traffic to arbitrary IPs. |
There was a problem hiding this comment.
| - intercept other users' outbound traffic to arbitrary IPs. | |
| - intercept other users' outbound traffic to arbitrary cluster-external IPs. |
| Any user who can create a Service with ExternalIPs could: | ||
|
|
||
| - intercept other users' outbound traffic to arbitrary IPs. | ||
| - could (non-deterministically) steal other users' inbound traffic to their own ExternalIPs. |
There was a problem hiding this comment.
| - could (non-deterministically) steal other users' inbound traffic to their own ExternalIPs. | |
| - (non-deterministically) steal other users' inbound traffic to their own ExternalIPs. |
(You already have a "could" introducing the list.)
|
|
||
| This document shares how to control how Services with ExternalIPs are managed within your cluster. | ||
|
|
||
| An ExternalIP is a powerful tool that could be used for [malicious intent](https://www.cvedetails.com/cve/CVE-2020-8554/). |
There was a problem hiding this comment.
The problem is that it's not supposed to be a powerful tool. The power was accidental, and that's the problem.
Some background to incorporate:
- The feature was originally intended to be an alternative to
LoadBalancerServices for non-cloud platforms. - It was added in the very early days of Kubernetes when there was no real concept of untrusted users, and the API is designed in a way that makes it difficult to lock down when there are untrusted users.
- Ordinary (non-admin) users generally don't have the ability to use the feature in the intended way (since this usually requires adding the externalIP to an interface on the node) but they do have the ability to use it to exploit the CVE!
- The feature is enabled and unrestricted by default for backward-compatibility reasons, but we recommend people disable or restrict it in clusters that don't need it.
| This document shares how to control how Services with ExternalIPs are managed within your cluster. | ||
|
|
||
| An ExternalIP is a powerful tool that could be used for [malicious intent](https://www.cvedetails.com/cve/CVE-2020-8554/). | ||
|
|
||
| Any user who can create a Service with ExternalIPs could: | ||
|
|
||
| - intercept other users' outbound traffic to arbitrary IPs. | ||
| - could (non-deterministically) steal other users' inbound traffic to their own ExternalIPs. |
There was a problem hiding this comment.
We need some (style-guide-approved) way of being explicit that we are referring to the externalIPs feature and not just the vague concept of "external IPs", which can easily be misinterpreted. (eg if this point was rewritten as "to their own external IPs", it would be easy to assume it meant cluster-external IPs as in the previous bullet point).
| {{< version-check >}} | ||
|
|
||
| <!-- steps --> | ||
|
|
There was a problem hiding this comment.
somewhere in this doc you should also meantion that you can disable the feature entirely by enabling the DenyServiceExternalIPs admission controller via kube-apiserver's --enable-admission-plugins flag.
|
|
||
| ## Kubernetes Service ExternalIP Policies | ||
|
|
||
| Cluster administrators can implement policies to control the creation and modification of Services with ExternalIPs within the cluster. This allows for centralized management of the allowed ExternalIPs used for Services and helps prevent unintended or conflicting configurations. Kubernetes provides mechanisms like Validating Admission Policies to enforce these rules. |
There was a problem hiding this comment.
I would also move this paragraph to the introduction, since the page is specifically about solving this with ValidatingAdmissionPolicy.
(If you add a section mentioning the admission controller (which I'm pretty sure we don't document anywhere else other than the list-of-all-admission-controllers) then that wouldn't apply.)
| spec: | ||
| policyName: "allow-specific-users-to-manage-externalips" | ||
| validationActions: [Deny, Audit] | ||
| ``` |
There was a problem hiding this comment.
should get someone in sig-auth to review your VAPs and make sure they're correct too
There was a problem hiding this comment.
SIG Security are also fine folks to ask.
| spec: | ||
| policyName: "allow-specific-users-to-manage-externalips" | ||
| validationActions: [Deny, Audit] | ||
| ``` |
There was a problem hiding this comment.
You need to be clear about what these policies do and don't protect against.
The first policy protects against "users can intercept traffic to arbitrary cluster-external IPs", but it doesn't prevent "users can intercept each others' ExternalIPs".
The second policy doesn't protect against either problem; it just assumes that you're restricting the feature to users who you trust to not (intentionally or accidentally) exploit the CVE.
Does CEL let you use maps in variables? It would be cool to have a policy that says "service account X is allowed to use these specific externalIPs, and service account Y is allowed to use these specific externalIPs (which are different from X's)". That would block both problems.
There was a problem hiding this comment.
You can use parameters in bindings, which gives you a way to do mappings I think (not tried it).
Description
Add a page called "Block Services with ExternalIPs" , that uses a VAP to explain how a user may be able to block specific Services with ExternalIPs
Issue
Closes: #51689
/cc @danwinship