Skip to content

Commit 0d2e19b

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request kubernetes#1406 from nckturner/credential-provider-extraction
Credential provider extraction
2 parents a89676d + 6358cda commit 0d2e19b

File tree

1 file changed

+261
-0
lines changed

1 file changed

+261
-0
lines changed

‎keps/sig-cloud-provider/20191004-out-of-tree-credential-providers.md

Lines changed: 261 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,261 @@
1+
---
2+
title: Out-of-Tree Credential Providers
3+
authors:
4+
- "@mcrute"
5+
- "@nckturner"
6+
owning-sig: sig-cloud-provider
7+
participating-sigs:
8+
- sig-node
9+
- sig-auth
10+
reviewers:
11+
- "@andrewsykim"
12+
- "@cheftako"
13+
- "@tallclair"
14+
- "@mikedanese"
15+
approvers:
16+
- "@andrewsykim"
17+
- "@cheftako"
18+
- "@tallclair"
19+
- "@mikedanese"
20+
editor: TBD
21+
creation-date: 2019-10-04
22+
last-updated: 2019-12-10
23+
status: implementable
24+
---
25+
26+
# Out-of-Tree Credential Providers
27+
28+
## Table of Contents
29+
30+
<!-- toc -->
31+
- [Release Signoff Checklist](#release-signoff-checklist)
32+
- [Summary](#summary)
33+
- [Motivation](#motivation)
34+
- [Goals](#goals)
35+
- [Non-Goals](#non-goals)
36+
- [Proposal](#proposal)
37+
- [External Credential Provider](#external-credential-provider)
38+
- [Example](#example)
39+
- [Alternatives Considered](#alternatives-considered)
40+
- [API Server Proxy](#api-server-proxy)
41+
- [Sidecar Credential Daemon](#sidecar-credential-daemon)
42+
- [Bound Service Account Token Flow](#bound-service-account-token-flow)
43+
- [Pushing Credential Management into the CRI](#pushing-credential-management-into-the-cri)
44+
- [Risks and Mitigations](#risks-and-mitigations)
45+
- [Design Details](#design-details)
46+
- [Test Plan](#test-plan)
47+
- [Graduation Criteria](#graduation-criteria)
48+
- [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
49+
- [Version Skew Strategy](#version-skew-strategy)
50+
- [Implementation History](#implementation-history)
51+
- [Infrastructure Needed](#infrastructure-needed)
52+
<!-- /toc -->
53+
54+
## Release Signoff Checklist
55+
56+
- [x] kubernetes/enhancements issue in release milestone, which links to KEP (this should be a link to the KEP location in kubernetes/enhancements, not the initial KEP PR)
57+
- [x] KEP approvers have set the KEP status to `implementable`
58+
- [x] Design details are appropriately documented
59+
- [x] Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
60+
- [x] Graduation criteria is in place
61+
- [ ] "Implementation History" section is up-to-date for milestone
62+
- [ ] User-facing documentation has been created in [kubernetes/website], for publication to [kubernetes.io]
63+
- [ ] Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
64+
65+
## Summary
66+
67+
This KEP replaces the existing in-tree container image registry credential providers with an external and pluggable credential provider mechanism and removes the in-tree credential providers.
68+
69+
## Motivation
70+
71+
Kubelet uses cloud provider specific SDKs to obtain credentials when pulling container images from cloud provider specific registries. The use of cloud provider specific SDKs from within the main Kubernetes tree is deprecated by [KEP-0002](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/20180530-cloud-controller-manager.md) and all existing uses need to be migrated out-of-tree. This KEP supports that migration process by removing this SDK usage.
72+
73+
### Goals
74+
75+
* Develop/test/release an interface for kubelet to obtain registry credentials from a cloud provider specific binary
76+
* Update/test/release the credential acquisition logic within kubelet
77+
* Build user documentation for out-of-tree credential providers
78+
* Support migration from existing in-tree credential providers to the new credential provider interface, along with dynamic roll back.
79+
* Remove in-tree credential provider code from Kubernetes core
80+
81+
### Non-Goals
82+
83+
* Broad removal of cloud SDK usage falls under the [KEP for removing in-tree providers](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/2019-01-25-removing-in-tree-providers.md).
84+
* Continuing to support projects that import the credential provider package.
85+
86+
## Proposal
87+
88+
### External Credential Provider
89+
90+
An executable capable of providing container registry credentials will be pre-installed on each node so that it exists when kubelet starts running. This binary will be executed by the kubelet to obtain container registry credentials in a format compatible with container runtimes. Credential responses may be cached within the kubelet.
91+
92+
This architecture is similar to the approach taken by the exec based credential plugin architecture already present in client-go and CNI, and is a well understood pattern. The API types are modeled after the ExecConfig and ExecCredential in client-go which define exec based credential retrieval for similar use cases.
93+
94+
A `RegistryCredentialConfig` and `RegistryCredentialProvider` configuration API type (similar to [clientauthentication](https://github.com/kubernetes/kubernetes/tree/0273d43ae9486e9d0be292c01de2dd4143522b86/staging/src/k8s.io/client-go/pkg/apis/clientauthentication/v1beta1)) will be added to Kubernetes:
95+
96+
```go
97+
type RegistryCredentialConfig struct {
98+
metav1.TypeMeta `json:",inline"`
99+
100+
Providers []RegistryCredentialProvider `json:"providers"`
101+
}
102+
103+
// RegistryCredentialProvider is used by the kubelet container runtime to match the
104+
// image property string (from the container spec) with exec-based credential provider
105+
// plugins that provide container registry credentials.
106+
type RegistryCredentialProvider struct {
107+
metav1.TypeMeta `json:",inline"`
108+
109+
// ImageMatchers is a list of strings used to match against the image property
110+
// (sometimes called "registry path") to determine which images to provide
111+
// credentials for. If one of the strings matches the image property, then the
112+
// RegistryCredentialProvider will be used by kubelet to provide credentials
113+
// for the image pull.
114+
115+
// The image property of a container supports the same syntax as the docker
116+
// command does, including private registries and tags. A registry path is
117+
// similar to a URL, but does not contain a protocol specifier (https://).
118+
//
119+
// Each ImageMatcher string is a pattern which can optionally contain
120+
// a port and a path, similar to the image spec. Globs can be used in the
121+
// hostname (but not the port or the path).
122+
//
123+
// Globs are supported as subdomains (*.k8s.io) or (k8s.*.io), and
124+
// top-level-domains (k8s.*). Matching partial subdomains is also supported
125+
// (app*.k8s.io). Each glob can only match a single subdomain segment, so
126+
// *.io does not match *.k8s.io.
127+
//
128+
// The image property matches when it has the same number of parts as the
129+
// ImageMatcher string, and each part matches. Additionally the path of
130+
// ImageMatcher must be a prefix of the target URL. If the ImageMatcher
131+
// contains a port, then the port must match as well.
132+
ImageMatchers []string `json:"imageMatchers"`
133+
134+
// Exec specifies a custom exec-based plugin. This type is defined in
135+
// https://github.com/kubernetes/client-go/blob/62f256057db7571c5ed1aba47eea291f72dd557a/tools/clientcmd/api/types.go#L184
136+
Exec clientcmd.ExecConfig
137+
}
138+
```
139+
140+
The RegistryCredentialConfig will be encoded in YAML and located in a file on disk. The exact path of the credential provider configuration file will be passed to kubelet via a new configuration option `RegistryCredentialConfigPath`.
141+
142+
We will create new types `RegistryCredentialPluginRequest` and `RegistryCredentialPluginResponse` which will define the interface between the plugin and the kubelet runtime. After the kubelet matches the image property string to a RegistryCredentialProvider, the kubelet will exec the plugin binary, and pass the JSON encoded request to the plugin via stdin. This includes the image that is to be pulled. The plugin will report back the response, which includes the credentials that kubelet needs to pull the image.
143+
144+
In the in-tree implementation, the docker keyring, which has N credential providers, returns an `[]AuthConfig` on a Lookup(image string) call. This struct will be populated by the plugin rather than the in-tree provider.
145+
146+
```go
147+
// RegistryCredentialPluginRequest is passed to the plugin via stdin, and includes the image that will be pulled by kubelet.
148+
type RegistryCredentialPluginRequest struct {
149+
// Image is used when passed to registry credential providers as part of an
150+
// image pull
151+
Image string `json:"image"`
152+
}
153+
154+
// RegistryCredentialPluginResponse holds credentials for the kubelet runtime
155+
// to use for image pulls. It is returned from the plugin as stdout.
156+
type RegistryCredentialPluginResponse struct {
157+
metav1.TypeMeta `json:",inline"`
158+
159+
// +optional
160+
ExpirationTimestamp *metav1.Time `json:"expirationTimestamp,omitempty"`
161+
162+
// +optional
163+
Username *string `json:"username,omitempty"`
164+
// +optional
165+
Password *string `json:"password,omitempty"`
166+
167+
// IdentityToken is used to authenticate the user and get
168+
// an access token for the registry.
169+
// +optional
170+
IdentityToken *string `json:"identitytoken,omitempty"`
171+
172+
// RegistryToken is a bearer token to be sent to a registry
173+
// +optional
174+
RegistryToken *string `json:"registrytoken,omitempty"`
175+
}
176+
177+
```
178+
179+
### Example
180+
181+
A registry credential provider configuration for Amazon ECR could look like the following:
182+
183+
```yaml
184+
kind: credentialprovider
185+
apiVersion: v1alpha1
186+
providers:
187+
-
188+
imageMatchers:
189+
- *.dkr.ecr.*.amazonaws.com
190+
- *.dkr.ecr.*.amazonaws.com.cn
191+
exec:
192+
command: ecr-creds
193+
args: token
194+
apiVersion: v1alpha1
195+
```
196+
197+
Where ecr-creds is a binary that vends ecr credentials. This would execute the binary `ecr-creds` with the argument `token` for the image `012345678910.dkr.ecr.us-east-1.amazonaws.com/my-image`.
198+
199+
### Alternatives Considered
200+
201+
#### API Server Proxy
202+
203+
The API server would act as a proxy to an external container registry credential provider that may support multiple cloud providers. The credential provider service will return container runtime compatible responses of the type currently used by the credential provider infrastructure within the kubelet along with credential expiration information to allow the API server to cache credential responses for a period of time.
204+
205+
This limits the cloud-specific privileges required for each node for the purpose of fetching credentials. Centralized caching helps to avoid cloud-specific rate limits for credential acquisition by consolidating that credential acquisition within the API server.
206+
207+
We chose not to follow this approach because although less privileges on each node and centralized caching are good, we have not seen enough evidence that these features are commonly requested by users. Also, it is outside the stated goals of this KEP. Lastly, taking the time to design such a system would probably take long enough to push back the date that we could extract the in-tree cloud providers completely from Kubernetes.
208+
209+
#### Sidecar Credential Daemon
210+
211+
Each node would run a sidecar credential daemon that can obtain cloud-specific container registry credentials and may support multiple cloud providers. This service will be available to the kubelet on the local host and will return container runtime responses compatible with those currently used by the credential provider infrastructure within kubelet. Each daemon will perform its own caching of credentials for the node on which it runs.
212+
213+
The added complexity of running a daemon over executing a binary made this option less desirable to us. If a daemon implementation is necessary for a cloud provider, the binary can talk to one to retrieve credentials upon each execution.
214+
215+
#### Bound Service Account Token Flow
216+
217+
Suggested in https://github.com/kubernetes/kubernetes/issues/68810, an image pull flow built on bound service account tokens would provide kubelet with credentials to pull images for pods running as a specific service account.
218+
219+
This approach might be better suited as a future enhancement to either the credential provider or ImagePullSecrets, but is out of scope for extracting the cloud provider specific code.
220+
221+
#### Pushing Credential Management into the CRI
222+
223+
Another possibility is moving the credential management logic into the CRI, so that Kubelet doesn't provide any credentials for image pulls. Similarly, this approach is also out of scope for extracting cloud provider code because it would be a more significant redesign but should be considered for a future enhancement.
224+
225+
### Risks and Mitigations
226+
227+
This is a critical feature of kubelet and pods cannot start if it does not work correctly. This functionality will be labeled alpha and hidden behind a feature gate in v1.18. It will use DynamicKubeletConfig so that it can be disabled during runtime if any problems occur.
228+
229+
## Design Details
230+
231+
### Test Plan
232+
233+
* Unit tests for image matching logic.
234+
* E2E tests for image pulls from cloud providers.
235+
236+
### Graduation Criteria
237+
238+
Successful Alpha Criteria
239+
* Multiple plugin implentations created.
240+
* One E2E test implemented.
241+
242+
### Upgrade / Downgrade Strategy
243+
244+
Upgrading
245+
* Add any cloud provider plugin binaries for image repositories that you use to your worker nodes.
246+
* Enable this feature in kubelet with a feature flag.
247+
248+
Downgrading
249+
* Disable this feature in kubelet with a feature flag.
250+
251+
### Version Skew Strategy
252+
253+
TODO
254+
255+
## Implementation History
256+
257+
TODO
258+
259+
## Infrastructure Needed
260+
261+
* New GitHub repos for existing credential providers (AWS, Azure, GCP)

0 commit comments

Comments
 (0)