Skip to content

Commit 6d5ee25

Browse files
aramasearamase
authored
Add blog post for KMSv2 beta (#40194)
* Add blog post for KMSv2 beta Signed-off-by: Anish Ramasekar <[email protected]> * add svg and reference in blog Signed-off-by: Anish Ramasekar <[email protected]> * add performance improvements Signed-off-by: Anish Ramasekar <[email protected]> * address review feedback Signed-off-by: Anish Ramasekar <[email protected]> * use 2023-05-01 for date/path and address review feedback Signed-off-by: Anish Ramasekar <[email protected]> * wrap long lines Signed-off-by: Anish Ramasekar <[email protected]> * change blog date to 2023-05-16 Signed-off-by: Anish Ramasekar <[email protected]> --------- Signed-off-by: Anish Ramasekar <[email protected]>
1 parent 6bf15f5 commit 6d5ee25

File tree

5 files changed

+239
-0
lines changed

5 files changed

+239
-0
lines changed

‎content/en/blog/_posts/2023-05-16-kmsv2-beta.md

Lines changed: 235 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,235 @@
1+
---
2+
layout: blog
3+
title: "Kubernetes 1.27: KMS V2 Moves to Beta"
4+
date: 2023-05-16
5+
slug: kms-v2-moves-to-beta
6+
---
7+
8+
**Authors:** Anish Ramasekar, Mo Khan, and Rita Zhang (Microsoft)
9+
10+
With Kubernetes 1.27, we (SIG Auth) are moving Key Management Service (KMS) v2 API to beta.
11+
12+
## What is KMS?
13+
One of the first things to consider when securing a Kubernetes cluster is encrypting etcd data at
14+
rest. KMS provides an interface for a provider to utilize a key stored in an external key service to
15+
perform this encryption.
16+
17+
KMS v1 has been a feature of Kubernetes since version 1.10, and is currently in beta as of version
18+
v1.12. KMS v2 was introduced as alpha in v1.25.
19+
20+
{{% alert title="Note" color="primary" %}}
21+
The KMS v2 API and implementation changed in incompatible
22+
ways in-between the alpha release in v1.25 and the beta release in v1.27. The design of KMS v2 has
23+
changed since [the previous blog post](https://kubernetes.io/blog/2022/09/09/kms-v2-improvements/)
24+
was written and it is not compatible with the design in this blog post. Attempting to upgrade from
25+
old versions with the alpha feature enabled will result in data loss.
26+
{{% /alert %}}
27+
28+
## What’s new in `v2beta1`?
29+
The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd. The data is
30+
encrypted using a data encryption key (DEK). The DEKs are encrypted with a key encryption key (KEK)
31+
that is stored and managed in a remote KMS. With KMS v1, a new DEK is generated for each encryption.
32+
With KMS v2, a new DEK is only generated on server startup and when the KMS plugin informs the API
33+
server that a KEK rotation has occurred.
34+
35+
{{% alert title="Caution" color="warning" %}}
36+
If you are running virtual machine (VM) based nodes
37+
that leverage VM state store with this feature, you must not use KMS v2.
38+
39+
With KMS v2, the API server uses AES-GCM with a 12 byte nonce (8 byte atomic counter and 4 bytes
40+
random data) for encryption. The following issues could occur if the VM is saved and restored:
41+
42+
1. The counter value may be lost or corrupted if the VM is saved in an inconsistent state or
43+
restored improperly. This can lead to a situation where the same counter value is used twice,
44+
resulting in the same nonce being used for two different messages.
45+
2. If the VM is restored to a previous state, the counter value may be set back to its previous
46+
value, resulting in the same nonce being used again.
47+
48+
Although both of these cases are partially mitigated by the 4 byte random nonce, this can compromise
49+
the security of the encryption.
50+
{{% /alert %}}
51+
52+
### Sequence Diagram
53+
54+
#### Encrypt Request
55+
56+
<!-- source
57+
```mermaid
58+
%%{init:{"theme":"neutral", "sequence": {"mirrorActors":true},
59+
"themeVariables": {
60+
"actorBkg":"royalblue",
61+
"actorTextColor":"white"
62+
}}}%%
63+
64+
sequenceDiagram
65+
participant user
66+
participant kube_api_server
67+
participant kms_plugin
68+
participant external_kms
69+
alt Generate DEK at startup
70+
Note over kube_api_server,external_kms: Refer to Generate Data Encryption Key (DEK) diagram for details
71+
end
72+
user->>kube_api_server: create/update resource that's to be encrypted
73+
kube_api_server->>kube_api_server: encrypt resource with DEK
74+
kube_api_server->>etcd: store encrypted object
75+
```
76+
-->
77+
78+
{{< figure src="/images/blog/2023-05-16-kubernetes-1.27-kmsv2-beta/kubernetes-1.27-encryption.svg"
79+
alt="Sequence diagram for KMSv2 beta Encrypt" class="diagram-large" >}}
80+
81+
#### Decrypt Request
82+
83+
<!-- source
84+
```mermaid
85+
%%{init:{"theme":"neutral", "sequence": {"mirrorActors":true},
86+
"themeVariables": {
87+
"actorBkg":"royalblue",
88+
"actorTextColor":"white"
89+
}}}%%
90+
91+
sequenceDiagram
92+
participant user
93+
participant kube_api_server
94+
participant kms_plugin
95+
participant external_kms
96+
participant etcd
97+
user->>kube_api_server: get/list resource that's encrypted
98+
kube_api_server->>etcd: get encrypted resource
99+
etcd->>kube_api_server: encrypted resource
100+
alt Encrypted DEK not in cache
101+
kube_api_server->>kms_plugin: decrypt request
102+
kms_plugin->>external_kms: decrypt DEK with remote KEK
103+
external_kms->>kms_plugin: decrypted DEK
104+
kms_plugin->>kube_api_server: return decrypted DEK
105+
kube_api_server->>kube_api_server: cache decrypted DEK
106+
end
107+
kube_api_server->>kube_api_server: decrypt resource with DEK
108+
kube_api_server->>user: return decrypted resource
109+
```
110+
-->
111+
112+
{{< figure src="/images/blog/2023-05-16-kubernetes-1.27-kmsv2-beta/kubernetes-1.27-decryption.svg"
113+
alt="Sequence diagram for KMSv2 beta Decrypt" class="diagram-large" >}}
114+
115+
#### Status Request
116+
117+
<!-- source
118+
```mermaid
119+
%%{init:{"theme":"neutral", "sequence": {"mirrorActors":true},
120+
"themeVariables": {
121+
"actorBkg":"royalblue",
122+
"actorTextColor":"white"
123+
}}}%%
124+
125+
sequenceDiagram
126+
participant kube_api_server
127+
participant kms_plugin
128+
participant external_kms
129+
alt Generate DEK at startup
130+
Note over kube_api_server,external_kms: Refer to Generate Data Encryption Key (DEK) diagram for details
131+
end
132+
loop every minute (or every 10s if error or unhealthy)
133+
kube_api_server->>kms_plugin: status request
134+
kms_plugin->>external_kms: validate remote KEK
135+
external_kms->>kms_plugin: KEK status
136+
kms_plugin->>kube_api_server: return status response <br/> {"healthz": "ok", key_id: "<remote KEK ID>", "version": "v2beta1"}
137+
alt KEK rotation detected (key_id changed), rotate DEK
138+
Note over kube_api_server,external_kms: Refer to Generate Data Encryption Key (DEK) diagram for details
139+
end
140+
end
141+
```
142+
-->
143+
144+
{{< figure src="/images/blog/2023-05-16-kubernetes-1.27-kmsv2-beta/kubernetes-1.27-status.svg"
145+
alt="Sequence diagram for KMSv2 beta Status" class="diagram-large" >}}
146+
147+
#### Generate Data Encryption Key (DEK)
148+
149+
<!-- source
150+
```mermaid
151+
%%{init:{"theme":"neutral", "sequence": {"mirrorActors":true},
152+
"themeVariables": {
153+
"actorBkg":"royalblue",
154+
"actorTextColor":"white"
155+
}}}%%
156+
157+
sequenceDiagram
158+
participant kube_api_server
159+
participant kms_plugin
160+
participant external_kms
161+
kube_api_server->>kube_api_server: generate DEK
162+
kube_api_server->>kms_plugin: encrypt request
163+
kms_plugin->>external_kms: encrypt DEK with remote KEK
164+
external_kms->>kms_plugin: encrypted DEK
165+
kms_plugin->>kube_api_server: return encrypt response <br/> {"ciphertext": "<encrypted DEK>", key_id: "<remote KEK ID>", "annotations": {}}
166+
```
167+
-->
168+
169+
{{< figure src="/images/blog/2023-05-16-kubernetes-1.27-kmsv2-beta/kubernetes-1.27-generate-dek.svg"
170+
alt="Sequence diagram for KMSv2 beta Generate DEK" class="diagram-large" >}}
171+
172+
### Performance Improvements
173+
With KMS v2, we have made significant improvements to the performance of the KMS encryption
174+
provider. In case of KMS v1, a new DEK is generated for every encryption. This means that for every
175+
write request, the API server makes a call to the KMS plugin to encrypt the DEK using the remote
176+
KEK. The API server also has to cache the DEKs to avoid making a call to the KMS plugin for every
177+
read request. When the API server restarts, it has to populate the cache by making a call to the KMS
178+
plugin for every DEK in the etcd store based on the cache size. This is a significant overhead for
179+
the API server. With KMS v2, the API server generates a DEK at startup and caches it. The API server
180+
also makes a call to the KMS plugin to encrypt the DEK using the remote KEK. This is a one-time call
181+
at startup and on KEK rotation. The API server then uses the cached DEK to encrypt the resources.
182+
This reduces the number of calls to the KMS plugin and improves the overall latency of the API
183+
server requests.
184+
185+
We conducted a test that created 12k secrets and measured the time taken for the API server to
186+
encrypt the resources. The metric used was
187+
[`apiserver_storage_transformation_duration_seconds`](https://kubernetes.io/docs/reference/instrumentation/metrics/).
188+
For KMS v1, the test was run on a managed Kubernetes v1.25 cluster with 2 nodes. There was no
189+
additional load on the cluster during the test. For KMS v2, the test was run in the Kubernetes CI
190+
environment with the following [cluster
191+
configuration](https://github.com/kubernetes/kubernetes/blob/release-1.27/test/e2e/testing-manifests/auth/encrypt/kind.yaml).
192+
193+
| KMS Provider | Time taken by 95 percentile |
194+
| ------------ | --------------------------- |
195+
| KMS v1 | 160ms |
196+
| KMS v2 | 80μs |
197+
198+
The results show that the KMS v2 encryption provider is three orders of magnitude faster than the
199+
KMS v1 encryption provider.
200+
201+
## What's next?
202+
For Kubernetes v1.28, we expect the feature to stay in beta. In the coming releases we want to
203+
investigate:
204+
- Cryptographic changes to remove the limitation on VM state store.
205+
- Kubernetes REST API changes to enable a more robust story around key rotation.
206+
- Handling undecryptable resources. Refer to the
207+
[KEP](https://github.com/kubernetes/enhancements/pull/3927) for details.
208+
209+
You can learn more about KMS v2 by reading [Using a KMS provider for data
210+
encryption](/docs/tasks/administer-cluster/kms-provider/). You can also follow along on the
211+
[KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-auth/3299-kms-v2-improvements/#readme)
212+
to track progress across the coming Kubernetes releases.
213+
214+
## Call to action
215+
216+
In this blog post, we have covered the improvements made to the KMS encryption provider in
217+
Kubernetes v1.27. We have also discussed the new KMS v2 API and how it works. We would love to hear
218+
your feedback on this feature. In particular, we would like feedback from Kubernetes KMS plugin
219+
implementors as they go through the process of building their integrations with this new API. Please
220+
reach out to us on the [#sig-auth-kms-dev](https://kubernetes.slack.com/archives/C03035EH4VB)
221+
channel on Kubernetes Slack.
222+
223+
## How to get involved
224+
If you are interested in getting involved in the development of this feature, share feedback, or
225+
participate in any other ongoing SIG Auth projects, please reach out on the
226+
[#sig-auth](https://kubernetes.slack.com/archives/C0EN96KUY) channel on Kubernetes Slack.
227+
228+
You are also welcome to join the bi-weekly [SIG Auth
229+
meetings](https://github.com/kubernetes/community/blob/master/sig-auth/README.md#meetings), held
230+
every-other Wednesday.
231+
232+
## Acknowledgements
233+
This feature has been an effort driven by contributors from several different companies. We would
234+
like to extend a huge thank you to everyone that contributed their time and effort to help make this
235+
possible.

‎static/images/blog/2023-05-16-kubernetes-1.27-kmsv2-beta/kubernetes-1.27-decryption.svg

Lines changed: 1 addition & 0 deletions
Loading

0 commit comments

Comments
 (0)