Skip to content

Commit 6f44250

Browse files
deads2kdeads2k
committed
add an option for skipping tls verifiation on logs requests
1 parent 2ac1dcd commit 6f44250

File tree

1 file changed

+222
-0
lines changed

1 file changed

+222
-0
lines changed

‎keps/sig-api-machinery/20190927-insecure-backend-proxy.md

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
---
2+
title: Insecure Backend Proxy
3+
authors:
4+
- "@deads2k"
5+
owning-sig: sig-xxx
6+
participating-sigs:
7+
- sig-apimachinery
8+
- sig-auth
9+
- sig-cli
10+
reviewers:
11+
- "@sttts"
12+
- "@cheftako"
13+
- "@liggitt"
14+
- "@soltysh"
15+
approvers:
16+
- "@lavalamp"
17+
- "@mikedanese"
18+
editor: TBD
19+
creation-date: 2019-09-27
20+
last-updated: 2019-09-27
21+
status: implementable
22+
see-also:
23+
replaces:
24+
superseded-by:
25+
---
26+
27+
# Insecure Backend Proxy
28+
29+
When trying to get logs for a pod, it is possible for a kubelet to have an expired serving certificate.
30+
If a client chooses, it should be possible to bypass the default behavior of the kube-apiserver and allow the kube-apiserver
31+
to skip TLS verification of the kubelet to allow gathering logs. This is especially important for debugging
32+
misbehaving self-hosted clusters.
33+
34+
## Table of Contents
35+
36+
<!-- toc -->
37+
- [Release Signoff Checklist](#release-signoff-checklist)
38+
- [Summary](#summary)
39+
- [Motivation](#motivation)
40+
- [Goals](#goals)
41+
- [Non-Goals](#non-goals)
42+
- [Proposal](#proposal)
43+
- [User Stories [optional]](#user-stories-optional)
44+
- [Story 1](#story-1)
45+
- [Story 2](#story-2)
46+
- [Implementation Details/Notes/Constraints [optional]](#implementation-detailsnotesconstraints-optional)
47+
- [Risks and Mitigations](#risks-and-mitigations)
48+
- [Design Details](#design-details)
49+
- [Test Plan](#test-plan)
50+
- [Graduation Criteria](#graduation-criteria)
51+
- [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
52+
- [Version Skew Strategy](#version-skew-strategy)
53+
- [Implementation History](#implementation-history)
54+
- [Drawbacks [optional]](#drawbacks-optional)
55+
- [Alternatives [optional]](#alternatives-optional)
56+
- [Infrastructure Needed [optional]](#infrastructure-needed-optional)
57+
<!-- /toc -->
58+
59+
## Release Signoff Checklist
60+
61+
**ACTION REQUIRED:** In order to merge code into a release, there must be an issue in [kubernetes/enhancements] referencing this KEP and targeting a release milestone **before [Enhancement Freeze](https://github.com/kubernetes/sig-release/tree/master/releases)
62+
of the targeted release**.
63+
64+
For enhancements that make changes to code or processes/procedures in core Kubernetes i.e., [kubernetes/kubernetes], we require the following Release Signoff checklist to be completed.
65+
66+
Check these off as they are completed for the Release Team to track. These checklist items _must_ be updated for the enhancement to be released.
67+
68+
- [x] kubernetes/enhancements issue in release milestone, which links to KEP https://github.com/kubernetes/enhancements/issues/1295
69+
- [x] KEP approvers have set the KEP status to `implementable`
70+
- [x] Design details are appropriately documented
71+
- [x] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
72+
- [x] Graduation criteria is in place
73+
- [ ] "Implementation History" section is up-to-date for milestone
74+
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
75+
- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
76+
77+
**Note:** Any PRs to move a KEP to `implementable` or significant changes once it is marked `implementable` should be approved by each of the KEP approvers. If any of those approvers is no longer appropriate than changes to that list should be approved by the remaining approvers and/or the owning SIG (or SIG-arch for cross cutting KEPs).
78+
79+
**Note:** This checklist is iterative and should be reviewed and updated every time this enhancement is being considered for a milestone.
80+
81+
[kubernetes.io]: https://kubernetes.io/
82+
[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues
83+
[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes
84+
[kubernetes/website]: https://github.com/kubernetes/website
85+
86+
## Summary
87+
88+
When trying to get logs for a pod, it is possible for a kubelet to have an expired serving certificate.
89+
If a client chooses, it should be possible to bypass the default behavior of the kube-apiserver and allow the kube-apiserver
90+
to skip TLS verification of the kubelet to allow gathering logs. This is safe because the kube-apiserver's credentials
91+
are always client certificates which cannot be replayed by an evil-kubelet and risk is contained to an evil-kubelet
92+
returning false log data. If the user has chosen to accept this risk, we should allow it for the same reason we
93+
have an option for `--insecure-skip-tls-verify`.
94+
95+
## Motivation
96+
97+
On self-hosted clusters it is possible to end up in a state where a kubelet's serving certificate has expired so a kube-apiserver
98+
cannot verify the kubelet identity, *but* the kube-apiserver's client certificate is still valid so the kubelet can still
99+
verify the kube-apiserver. In this condition, a cluster-admin may need to get pod logs to debug his cluster.
100+
101+
### Goals
102+
103+
1. Allow cluster-admins to get pod logs from kubelets with expired serving certificates. This will include an API change
104+
and an addition argument to `kubectl log`
105+
106+
### Non-Goals
107+
108+
1. Allow any bidirectional traffic proxied to kubelets. This may be a future objective, but is not in scope for the current KEP.
109+
110+
## Proposal
111+
112+
In [PodLogOptions](https://github.com/kubernetes/api/blob/d58b53da08f5430bb0f4e1154a73314e82b5b3aa/core/v1/types.go),
113+
add a `InsecureSkipTLSVerifyBackend bool`
114+
```go
115+
// PodLogOptions is the query options for a Pod's logs REST call.
116+
type PodLogOptions struct {
117+
// ... existing fields snipped
118+
119+
// insecureSkipTLSVerifyBackend indicates that the apiserver should not confirm the validity of the
120+
// serving certificate of the backend it is connecting to. This will make the HTTPS connection between the apiserver
121+
// and the backend insecure. This means the apiserver cannot verify the log data it is receiving came from the real
122+
// kubelet. If the kubelet is configured to verify the apiserver's TLS credentials, it does not mean the
123+
// connection to the real kubelet is vulnerable to a man in the middle attack (e.g. an attacker could not intercept
124+
// the actual log data coming from the real kubelet).
125+
// +optional
126+
InsecureSkipTLSVerifyBackend bool `json:"insecureSkipTLSVerifyBackend,omitempty" protobuf:"varint,9,opt,name=insecureSkipTLSVerifyBackend"`
127+
}
128+
```
129+
The streamer for logs already prevents redirects (see https://github.com/kubernetes/kubernetes/blob/4ee9f007cbc88cca5fa3e8576ff951a52a248e3c/pkg/registry/core/pod/rest/log.go#L83) , so an evil kubelet intercepting this traffic cannot redirect the kube-apiserver
130+
to use its high powered credentials for a nefarious purpose.
131+
The `LocationStreamer` can take an additional argument and the `LogLocation`'s `NodeTransport` can actually produce a purpose
132+
built transport for insecure connections.
133+
134+
To make this easier to use, we can add an `--insecure-skip-tls-verify-backend` flag to `kubectl log` which plumbs the option.
135+
This part of the KEP is a nice to have, since the kube-apiserver owns the backing capability which has general utility.
136+
137+
### User Stories [optional]
138+
139+
#### Story 1
140+
141+
#### Story 2
142+
143+
### Implementation Details/Notes/Constraints [optional]
144+
145+
This design is safe based on the following conditions:
146+
147+
1. kube-apiservers only authenticate to kubelets using certificates. See `--kubelet-client-certificate`. Certificate based
148+
authentication does not send replayable credentials to backends.
149+
2. Clients must opt-in to the functionality and the documentation must include the impact in plain english.
150+
3. Evil kubelets cannot trick kube-apiservers into using their credentials for any other purpose.
151+
In order to use the kube-apiserver creds, the target of any proxy
152+
must terminate the connection. Since the URL is chosen by the kube-apiserver the evil kubelet cannot rewrite that destination URL. The destination is the URL for
153+
getting logs for one particular pod.
154+
155+
156+
### Risks and Mitigations
157+
158+
A super user with write permissions to Node and Pod API objects and read permissions to Pod logs to make the API
159+
server exfiltrate data at `https://<nodeName>:<nodePort>/containerLogs/<podNamespace>/<podName>/<containerName>`, where
160+
all the bracketed parameters were under their control.
161+
This is ability is already present for someone with full API control via an APIService configured to point to a service
162+
with `insecureSkipTLSVerify:true`, with a similar restriction on the path of requests sent to it (must have a leading
163+
`/apis/<group>/<version>` path prefix).
164+
This could affect
165+
the only scenarios I can really think of are:
166+
1. unsecured kubelets from another cluster (or kubelets from another cluster using the same CA)
167+
2. a non-kubelet endpoint that ignored the specified path and served something confidential (and was unsecured or
168+
honored the CA that signed the apiserver's cert)
169+
Both scenarios are pretty unlikely.
170+
171+
Trying to restrict the insecureSkipTLSVerify option to ignoring only the expiry date requires reimplementing the TLS
172+
handshake with a custom method to
173+
1. get the serving cert
174+
2. pull the notAfter to find a time when the cert was valid
175+
3. construct a set of custom verify options at that time to verify the signature and hostname
176+
The risk in doing that is greater than the additional benefit.
177+
178+
179+
## Design Details
180+
181+
### Test Plan
182+
183+
1. Positive and negative tests for this are fairly easy to write and the changes are narrow in scope.
184+
185+
### Graduation Criteria
186+
187+
The problem and solution are well understood and congruent to our past solutions.
188+
This new API field will start at beta level.
189+
190+
### Upgrade / Downgrade Strategy
191+
192+
Because the change is isolated to non-persisted API contracts with the kube-apiserver, there are no skew or upgrade/downgrade considerations.
193+
194+
### Version Skew Strategy
195+
196+
Because the change is isolated to non-persisted API contracts with the kube-apiserver, there are no skew or upgrade/downgrade considerations.
197+
198+
## Implementation History
199+
200+
Major milestones in the life cycle of a KEP should be tracked in `Implementation History`.
201+
Major milestones might include
202+
203+
- the `Summary` and `Motivation` sections being merged signaling SIG acceptance
204+
- the `Proposal` section being merged signaling agreement on a proposed design
205+
- the date implementation started
206+
- the first Kubernetes release where an initial version of the KEP was available
207+
- the version of Kubernetes where the KEP graduated to general availability
208+
- when the KEP was retired or superseded
209+
210+
## Drawbacks [optional]
211+
212+
Why should this KEP _not_ be implemented.
213+
214+
## Alternatives [optional]
215+
216+
Similar to the `Drawbacks` section the `Alternatives` section is used to highlight and record other possible approaches to delivering the value proposed by a KEP.
217+
218+
## Infrastructure Needed [optional]
219+
220+
Use this section if you need things from the project/SIG.
221+
Examples include a new subproject, repos requested, github details.
222+
Listing these here allows a SIG to get the process for these resources started right away.

0 commit comments

Comments
 (0)