Add AppArmor GA KEP

Signed-off-by: Sascha Grunert <[email protected]>

Author: saschagrunert

keps/sig-node/20200110-apparmor-ga.md

@@ -0,0 +1,387 @@
+---
+title: AppArmor to GA
+authors:
+  - "@saschagrunert"
+  - "@tallclair"
+owning-sig: sig-node
+participating-sigs:
+  - sig-api-machinery
+  - sig-auth
+reviewers:
+  - TBD
+approvers:
+  - TBD
+editor: TBD
+creation-date: 2020-01-10
+status: provisional
+---
+
+# AppArmor to GA
+
+## Table of Contents
+
+<!-- toc -->
+
+- [Release Signoff Checklist](#release-signoff-checklist)
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [API](#api)
+    - [Pod API](#pod-api)
+    - [PodSecurityPolicy API](#podsecuritypolicy-api)
+- [Design Details](#design-details)
+  - [Version Skew Strategy](#version-skew-strategy)
+    - [Pod Creation](#pod-creation)
+    - [Pod Update](#pod-update)
+    - [PodSecurityPolicy Creation](#podsecuritypolicy-creation)
+    - [PodSecurityPolicy Update](#podsecuritypolicy-update)
+    - [PodSecurityPolicy Enforcement](#podsecuritypolicy-enforcement)
+    - [PodTemplates](#podtemplates)
+    - [Upgrade / Downgrade](#upgrade--downgrade)
+  - [Test Plan](#test-plan)
+  - [Graduation Criteria](#graduation-criteria)
+  - [Upgrade / Downgrade Strategy](#upgrade--downgrade-strategy)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+  <!-- /toc -->
+
+## Release Signoff Checklist
+
+**ACTION REQUIRED:** In order to merge code into a release, there must be an
+issue in [kubernetes/enhancements] referencing this KEP and targeting a release
+milestone **before [Enhancement
+Freeze](https://github.com/kubernetes/sig-release/tree/master/releases) of the
+targeted release**.
+
+For enhancements that make changes to code or processes/procedures in core
+Kubernetes i.e., [kubernetes/kubernetes], we require the following Release
+Signoff checklist to be completed.
+
+Check these off as they are completed for the Release Team to track. These
+checklist items _must_ be updated for the enhancement to be released.
+
+- [ ] kubernetes/enhancements issue in release milestone, which links to KEP
+      (this should be a link to the KEP location in kubernetes/enhancements, not the
+      initial KEP PR)
+- [ ] KEP approvers have set the KEP status to `implementable`
+- [ ] Design details are appropriately documented
+- [ ] Test plan is in place, giving consideration to SIG Architecture and SIG
+      Testing input
+- [ ] Graduation criteria is in place
+- [ ] "Implementation History" section is up-to-date for milestone
+- [ ] User-facing documentation has been created in [kubernetes/website], for
+      publication to [kubernetes.io]
+- [ ] Supporting documentation e.g., additional design documents, links to
+      mailing list discussions/SIG meetings, relevant PRs/issues, release notes
+
+**Note:** Any PRs to move a KEP to `implementable` or significant changes once
+it is marked `implementable` should be approved by each of the KEP approvers. If
+any of those approvers is no longer appropriate than changes to that list should
+be approved by the remaining approvers and/or the owning SIG (or SIG-arch for
+cross cutting KEPs).
+
+**Note:** This checklist is iterative and should be reviewed and updated every
+time this enhancement is being considered for a milestone.
+
+[kubernetes.io]: https://kubernetes.io/
+[kubernetes/enhancements]: https://github.com/kubernetes/enhancements/issues
+[kubernetes/kubernetes]: https://github.com/kubernetes/kubernetes
+[kubernetes/website]: https://github.com/kubernetes/website
+
+## Summary
+
+This is a proposal to upgrade the AppArmor annotation on pods & pod security
+policies (PSP) to a dedicated field, and mark the feature as GA. This proposal
+aims to do the _bare minimum_ to clean up the feature, without blocking future
+enhancements.
+
+## Motivation
+
+AppArmor support has been added with Kubernetes v1.14 and is already in beta.
+Profiles have to be available on each node whereas the container runtime ensures
+that the profile is loaded when specified at pod or PSP level. Profiles
+can be specified per-container via the pod’s metadata annotation:
+
+```
+container.apparmor.security.beta.kubernetes.io/<container_name>: {unconfined,runtime/default,localhost/<profile>}
+```
+
+The feature has been more or less unchanged ever since. Also note that the
+addition predates feature gates or our modern concept of feature lifecycle. So,
+even though the annotations include `beta` in the key, this is entirely useable
+on any production GA cluster.
+
+The main motivation behind this KEP is to promote the AppArmor feature to GA.
+
+_NOTE: Seccomp is in a very similar state, but with some subtle differences.
+Promoting Seccomp to GA will be covered by a [separate
+KEP][https://github.com/kubernetes/enhancements/pull/1148]._
+
+### Goals
+
+- Declare AppArmor as GA
+- Fully document and formally spec the feature support
+- Add equivalent API fields to replace AppArmor annotations
+- Deprecate the AppArmor annotations
+
+### Non-Goals
+
+This KEP proposes the absolute minimum to get AppArmor to GA, therefore all
+functional enhancements are out of scope, including:
+
+- Defining any standard "Kubernetes branded" AppArmor profiles
+- Formally specifying the AppArmor profile format in Kubernetes
+- Providing mechanisms for loading profiles from outside the of the node
+- Changing the semantics around AppArmor support
+- Windows support
+
+## Proposal
+
+AppArmor is not available on every Linux distribution. Beside this, container
+runtimes have AppArmor as compile-time feature which may be disabled as well.
+With the GA API we do not change the error handling and behave exactly the same
+as the current error propagation paths.
+
+### API
+
+The AppArmor API will be functionally equivalent to the current beta API. This
+includes the Pod API, which specifies what profile the containers run with, and
+the `PodSecurityPolicy` API which specifies allowed profiles & a default
+profile.
+
+#### Pod API
+
+The Pod AppArmor API is generally immutable, except in `PodTemplates`.
+
+```go
+type PodSecurityContext struct {
+    ...
+    // The AppArmor options to use by the containers in this pod.
+    // +optional
+    AppArmor  *AppArmorProfile
+    ...
+}
+
+type SecurityContext struct {
+    ...
+    // The AppArmor options to use by this container. If AppArmor options are
+    // provided at both the pod & container level, the container options
+    // override the pod options.
+    // +optional
+    AppArmor  *AppArmorProfile
+    ...
+}
+
+type AppArmorProfileType string
+
+const (
+    AppArmorProfileUnconfined AppArmorProfileType = "Unconfined"
+    AppArmorProfileDefault    AppArmorProfileType = "Default"
+    AppArmorProfileLocalhost  AppArmorProfileType = "Localhost"
+)
+
+// Only one profile source may be set.
+// +union
+type AppArmorProfile struct {
+    // +unionDescriminator
+    Type AppArmorProfileType
+
+    // Load a profile defined on the node.
+    // The profile must be available on the node to work.
+    // The length of the profile is limited to 253 characters.
+    // The `Type` in this struct has to be set to `AppArmorProfileLocalhost`.
+    // +optional
+    LocalhostProfile *string
+}
+```
+
+This API makes the options more explicit and leaves room for new profile sources
+to be added in the future (e.g. Kubernetes predefined profiles or ConfigMap
+profiles). The AppArmor options structure leaves room for future extensions,
+such as defining the behavior when a profile cannot be set.
+
+#### PodSecurityPolicy API
+
+```go
+type PodSecurityPolicySpec struct {
+    ...
+    // AppArmor is the strategy that will dictate allowable and default AppArmor
+    // profiles for the container.
+    // +optional
+    AppArmor *ApparmorStrategyOptions
+    ...
+}
+
+type AppArmorStrategyOptions struct {
+    // The default profile to set on the pod, if none is specified.
+    // The default MUST be allowed by the allowedProfiles.
+    // +optional
+    DefaultProfile *v1.AppArmorProfile
+
+    // The set of profiles that may be set on the pod or containers.
+    // If unspecified, AppArmor profiles are unrestricted by this policy.
+    // +optional
+    AllowedProfiles *AppArmorProfileSet
+}
+
+// A set of AppArmor profiles. This struct should be a plural of
+// `v1.AppArmorProfile`.
+// All values are optional, and an unspecified field excludes all profiles of
+// that type from the set.
+type AppArmorProfileSet struct {
+    // The allowed AppArmor profile types.
+    // +optional
+    Types []AppArmorProfileType
+
+    // The allowed runtimeProfiles. A value of '*' allows all runtimeProfiles.
+    // +optional
+    RuntimeProfiles []string
+
+    // The allowed localhostProfiles. Values may end in '*' to include all
+    // localhostProfiles with a prefix.
+    // +optional
+    LocalhostProfiles []string
+}
+```
+
+## Design Details
+
+### Version Skew Strategy
+
+All API skew is resolved in the API server. New Kubelets will only use the
+AppArmor values specified in the fields and ignore the annotations therefore.
+
+#### Pod Creation
+
+If no AppArmor annotations or fields are specified, no action is necessary.
+
+If _only_ AppArmor fields are specified, add the corresponding annotations. This
+ensures that the fields are enforced even if the node version trails the API
+version. Since [we
+support](https://kubernetes.io/docs/setup/release/version-skew-policy) up to 2
+minor releases of version skew between the master and node, this behavior will
+be removed in version N+4 (where N is the release with the upgraded AppArmor
+support).
+
+If _only_ AppArmor annotations are specified, copy the values into the
+corresponding fields. This ensures that existing applications continue to
+enforce AppArmor, and prevents the kubelet from needing to resolve annotations &
+fields.
+
+If both AppArmor annotations _and_ fields are specified, the values MUST match.
+This will be enforced in API validation.
+
+To raise awareness of annotation usage (in case of old automation), an
+additional warning annotation will be added when a pod is created with an
+AppArmor annotation:
+
+```
+warning.kubernetes.io/apparmor: "AppArmor set through annotations. Support will be dropped in v1.22"
+```
+
+#### Pod Update
+
+The AppArmor fields on a pod are immutable.
+
+The behavior on annotation update is currently ill-defined: the annotation
+update is allowed, but the new value will not be used until the container is
+restarted. There is no way to tell (from the API) what value a container is
+using.
+
+Therefore, AppArmor annotation updates will be ignored. This maintains backwards
+API compatibility (no tightening validation), and makes a small stabilizing
+change to behavior (new Kubelets will ignore the update).
+
+#### PodSecurityPolicy Creation
+
+Unlike with pods, PodSecurityPolicy AppArmor annotations and fields are _not_
+synced.
+
+If only AppArmor annotations or fields are specified, no action is necessary.
+The set value is used when applying the PodSecurityPolicy.
+
+If both AppArmor annotations _and_ fields are specified, the values MUST match.
+This will be enforced in API validation.
+
+#### PodSecurityPolicy Update
+
+PodSecurityPolicy AppArmor fields are mutable. On an update, the same rules are
+applied as for creation, ignoring the old values.
+
+If only AppArmor annotations or fields are specified in the updated PSP, no
+action is necessary, and the specified values are used.
+
+If both AppArmor annotations _and_ fields are specified in the updated PSP, the
+values MUST match.
+
+#### PodSecurityPolicy Enforcement
+
+The PodSecurityPolicy admission controller must continue to check the PSP object
+for annotations, as well as for fields.
+
+When setting default profiles, PSP only needs to set the field. The API
+machinery will handle setting the annotation as necessary.
+
+When enforcing allowed profiles, the PSP should check BOTH the annotations &
+fields. In most cases, they should be consistent. On pod update, the AppArmor
+annotations may differ from the fields. In that case, the PSP enforcement should
+check both values as the effective value depends on the node version running the
+pod.
+
+#### PodTemplates
+
+PodTemplates (e.g. ReplaceSets, Deployments, StatefulSets, etc.) will be
+ignored. The field/annotation resolution will happen on template instantiation.
+
+However, to raise awareness of existing controllers using the AppArmor
+annotations that need to be migrated, the same warning annotation will be added
+to the controller as for pods:
+
+```
+warning.kubernetes.io/apparmor: "AppArmor set through annotations. Support will be dropped in v1.22"
+```
+
+#### Upgrade / Downgrade
+
+Nodes do not currently support in-place upgrades, so pods will be recreated on
+node upgrade and downgrade. No special handling or consideration is needed to
+support this.
+
+On the API server side, we've already taken version skew in HA clusters into
+account. The same precautions make upgrade & downgrade handling a non-issue.
+
+### Test Plan
+
+AppArmor already has [e2e tests][https://github.com/kubernetes/kubernetes/blob/6596a14/test/e2e_node/apparmor_test.go],
+but the tests are guarded by the `[Feature:AppArmor]` tag and not run in the
+standard test suites.
+
+Prior to being marked GA, the feature tag will be removed from the AppArmor
+tests, and the tests will be migrated to the new fields API. Tests will be
+tagged as `[LinuxOnly]`.
+
+New tests will be added covering the annotation/field conflict cases described
+under [Version Skew Strategy](#version-skew-strategy).
+
+Test coverage for localhost profiles will be added as well.
+
+### Graduation Criteria
+
+_This section is excluded, as it is the subject of the entire proposal._
+
+### Upgrade / Downgrade Strategy
+
+See [Version Skew Strategy](#version-skew-strategy).
+
+## Implementation History
+
+- 2020-01-10: Initial KEP
+
+## Drawbacks
+
+Promoting AppArmor as-is to GA may be seen as "blessing" the current
+functionality, and make it harder to make some of the enhancements listed under
+[Non-Goals](#non-goals). Since the current behavior is unguarded, I think we
+already need to treat the behavior as GA.