Update KEP based on the recent feedback

Signed-off-by: Marko Mudrinić <[email protected]>

Author: xmudrii

keps/sig-release/1731-publishing-packages/README.md

@@ -19,7 +19,12 @@
     - [Package Sources](#package-sources)
     - [Package Specs](#package-specs)
     - [Integrating OBS with our current release pipeline](#integrating-obs-with-our-current-release-pipeline)
+    - [Recovering from a failed build](#recovering-from-a-failed-build)
     - [Authentication to OBS and User Management](#authentication-to-obs-and-user-management)
+    - [Ownership of OBS infrastructure and commitments](#ownership-of-obs-infrastructure-and-commitments)
+    - [GPG key ownership as part of the OBS infrastructure](#gpg-key-ownership-as-part-of-the-obs-infrastructure)
+    - [Clarification on the community-owned infrastructure](#clarification-on-the-community-owned-infrastructure)
+    - [Risks of vendor-lock in](#risks-of-vendor-lock-in)
     - [How are packages used?](#how-are-packages-used)
   - [Test Plan](#test-plan)
   - [Graduation Criteria](#graduation-criteria)
@@ -185,15 +190,15 @@ Scenario: [...]
 
 ### Risks and Mitigations
 
-- _Risk_: The OBS installation provided by openSUSE is unable to serve the load generated by the Kubernetes project
-  _Mitigation_: We can host our own mirrors and take some load from openSUSE (e.g. on Equinix Metal)
+- _Risk_: The OBS installation provided by SUSE is unable to serve the load generated by the Kubernetes project
+  _Mitigation_: We can host our own mirrors and take some load from SUSE (e.g. on Equinix Metal)
 - _Risk_: Building all the packages for all the distributions and their version takes too long to be done nightly or via cutting the release  
   _Mitigation_: We do not deliver nightly packages or wait for packages to be published in the release pipeline.
 
 ## Design Details
 
-Packages will be built and published using [Open Build Service (OBS)][obs]. openSUSE will sponsor the Kubernetes
-project by giving us access to the [OBS instance hosted by openSUSE][obs-build].
+Packages will be built and published using [Open Build Service (OBS)][obs]. SUSE will sponsor the Kubernetes
+project by giving us access to the [OBS instance hosted by SUSE][obs-build].
 
 [obs]: https://openbuildservice.org/
 [obs-build]: https://build.opensuse.org/
@@ -206,7 +211,7 @@ The reasons for using Open Build Service (OBS) instead of building and hosting p
   - Managing GPG keys ourselves represents a security risk. For example, if a Release Manager with access to the GPG key
     steps down, we might need to rotate the key. This is a process that affects End Users, therefore
     we want to avoid it
-  - In this case, GPG keys are securely managed by the OBS platform hosted by openSUSE. No one from the Kubernetes
+  - In this case, GPG keys are securely managed by the OBS platform hosted by SUSE. No one from the Kubernetes
     project will have direct access to the key, mitigating one of the main risks of this proposal
 - We want to avoid managing the infrastructure ourselves, including buckets, mirrors/CDNs...
 - We want to provide 'as a service' access to the packages infrastructure to Release Managers and eventually other
@@ -242,17 +247,20 @@ architectures:
 * `ppc64le`
 * `s390x`
 
-We'll **build** packages for those architectures on the following Operating Systems:
+It's important to note that we should build packages in a way to ensure the best compatibility with both newer and older versions of operating systems and package managers. That can be done by building packages on OS versions that has the appropriate build toolchain. For example, the latest OS version might have the latest build toolchain version which doesn't produce packages compatible with older operating systems and package managers. We refer to OS-es used for building packages as **builders**.
+
+Builders we'll be changed over time as operating systems reach End-of-Life (EOL) and new versions are becoming widely adopted. At the time of writing this KEP, we're looking at using following builders:
 
 - Ubuntu 20.04 (`aarch64`, `armv7l`, `ppc64le`, `s390x`, `x86_64`)
 - CentOS Stream 8 (`aarch64`, `ppc64le`, `x86_64`)
-  - CentOS Stream 8 doesn't support `armv7l` and `s390x` on OBS, hence, we use OpenSUSE Factory which does support
-    those architectures
-- OpenSUSE Factory (`armv7l`, `s390x`)
+  - CentOS Stream 8 doesn't support `armv7l` and `s390x`, hence, we need to use another builder for those architectures
+- SUSE Linux Enterprise (SLE) 12 SP5 (`armv7l`, `s390x`)
+
+[The following discussion][kep-1731-builders] has more details on reasoning behind looking at those builders.
 
-Those operating systems are used as **builders**. The resulting packages are supposed to work on any Debian-based
-and RPM-based distribution, respectively. Those distributions were chosen based on recommendations from the OBS team
-to provide the best compatibility with all all Debian-based and RPM-based distributions.
+**Note: this is a non-binding list of builders that can change at any time without prior notification. This list should be used only as a guidance for the initial implementation.**
+
+[kep-1731-builders]: https://github.com/kubernetes/enhancements/pull/3750#discussion_r1083146253
 
 The following packages will be published for all operating system and architectures listed earlier. For simplicity,
 we'll refer to those as packages as the **core packages**:
@@ -284,10 +292,7 @@ Speaking of OBS, **Packages** are located in **Projects**. We'll use two differe
 - Building/Staging project - we'll build packages in a project of this type
 - Publishing/Maintenance project - we'll publish packages from a project of this type
 
-The reason for using two different types of projects is that OBS doesn't keep previously published packages.
-For example, if we publish v1.26.0, and then v1.26.1, after v1.26.1 is published, the v1.26.0 package is removed.
-To be able to keep all packages/versions (e.g. v1.26.0 and v1.26.1), we need to publish packages from the
-maintenance project.
+The reason for using two different types of projects is that in the default OBS projects, which are of build-type, only one set of binaries is kept at any point in time and used to provide build dependencies for all the artifacts to build. By releasing built binaries into  a maintenance-type project, they can be snapshot-ed and stored together. The published repository includes all snapshotted binaries.
 
 The repository layout mentioned earlier replicates in OBS as:
 
@@ -386,13 +391,13 @@ As described above, currently, it's up to the Release Manager to create the subp
 before releasing the first alpha release. This should be the only manual steps required by Release Managers (besides
 the user management, described below). We'll consider automating those steps in the future if possible.
 
-The workflow for publishing packages is:
-
+`krel` should be extended with the following workflow for publishing packages:
+ 
 - Authenticate to OBS via `osc` and pull existing packages from the appropriate **building subproject**
 - Update specs and generate the sources tarball using `kubepkg`
 - Commit changes and push them to the **building subproject**
 - Wait for packages to be built
-  - There's [RabbitMQ][obs-rabbitmq] available that we can use to listen for events
+  - There's [RabbitMQ][obs-rabbitmq] hosted by SUSE that we can use to listen for events
   - This might not be feasible for all architectures, for example, building for `s390x` can take quite a while
 - Once packages are built, release those packages by running `osc release`
   - Releasing (or publishing) packages means taking them from the **building subproject** and publishing them
@@ -432,6 +437,10 @@ we'll use `osc` directly (by exec-ing), which also requires adding `osc` to our
 
 [obs-rabbitmq]: https://rabbit.opensuse.org/
 
+#### Recovering from a failed build
+
+In case a build fails for any reason, the Release Manager handling the release needs to take manual steps to recover that build. That might include logging in to the OBS platform and retriggering the build. If it's not possible to recover the build by restarting the build job, the Release Manager should bring it up to the appropriate channel for discussion on how should we proceed with that build. This is very similar to how we handle Kubernetes releases in general.
+
 #### Authentication to OBS and User Management
 
 The concept of API tokens in OBS is very limited and provides access only to a very few endpoints. In other words,
@@ -442,6 +451,51 @@ Users are managed manually via the OBS web interface. SIG Release Leads must hav
 our OBS project. Release Managers should be given read/write access, so they can maintain and create projects and
 packages.
 
+We'll consider automating the user management in the future by integrating with the `osc` tool or by building a Go library/tool for that ourselves.
+
+#### Ownership of OBS infrastructure and commitments
+
+Packages will be hosted, published, and served from the [OpenBuildService instance hosted and maintained by SUSE][obs-build]. The instance is under the full ownership and control of SUSE. SUSE provide us:
+
+- access to the instance to build, publish, and maintain packages form their instance
+- [best-effort based support][obs-support] (commercial/business support is not included and might be negotiated if needed)
+
+OBS consists of multiple parts and [its architecture is described in their docs][obs-arch]. If we want to take ownership of the infrastructure we have two options:
+
+- Partial ownership: Host the Download Infrastructure ourselves. The Download Infrastructure is used to serve packages to the end users. Depending on load that the Kubernetes project is generating, we might need to do it in order to take some load from the SUSE OBS instance (this is documented as a risk)
+- Full ownership: OBS platform is open source under the GPL license and we can completely run it on our own servers, however, we want to avoid this if possible
+  - We want to avoid this because of increased costs, problems with managing the GPG key, and efforts needed to maintain the infrastructure ourselves
+
+As per [comments from the OBS team][obs-installation], OBS is serving 25 TB over several hundred thousand request per day at the time of writing this KEP. The SUSE OBS Download Infrastructure is geo-replicated in various places on the planet, and can be scaled up if needed. As stated in the Risks section, we don't know how much load is generated by Kubernetes packages currently and that's why we might need to take some of load generated by the Kubernetes project by hosting a part of Download Infrastructure ourselves.
+
+[obs-arch]: https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.architecture.html#_overview_graph
+[obs-support]: https://github.com/kubernetes/enhancements/pull/3750#discussion_r1106006818
+[obs-installation]: https://github.com/kubernetes/enhancements/pull/3750#discussion_r1105994163
+
+#### GPG key ownership as part of the OBS infrastructure
+
+One of the main reasons we want to use OBS is to handover managing the GPG key to the platform instead of doing it ourselves. Reasons for that are described at the beginning of this KEP.
+
+OBS handles GPG keys in the following way:
+
+- The signing keys are generated on the signing server automatically
+- The signing keys are backed-up in the encrypted form on the OBS backend servers
+
+This means that there's no way to get the decrypted private key outside of the isolated signing server.
+
+#### Clarification on the community-owned infrastructure
+
+The initial version of this KEP insisted on the completely community owned infrastructure. However, due to all the challenges described above and time constraints, we decided to change this and integrate with a platform that can take most of those tasks from us. With OBS, we get much more control then we had before, because we can trigger build ourselves, we completely control spec files ourselves, what packages we want to publish, who should have access to the platform, and more.
+
+#### Risks of vendor-lock in
+
+It's important to clarify that there are no risks of vendor lock-in by using the OBS platform. Under the hood, the OBS platform is using upstream/standard tooling to build and publish packages (`dpkg-buildpackage` and `rpmbuild`). This also means that the package spec files can be used outside of OBS out-of-box.
+
+We'll take additional steps to ensure that we don't depend on the platform as an implementation detail:
+
+- We'll use a vanity domain pointing to the SUSE OBS platform, so in case we change the platform, users don't need to change the URL
+- Redistribution of the new GPG key can be done from the current platform, so that users don't need to take manual steps to update their GPG key
+
 #### How are packages used?
 
 The End Users can configure their systems’ package managers to use
@@ -554,6 +608,11 @@ It's up to the user what package repository (OBS or Google) they want to use.
 In case OBS doesn't work for them, they can reconfigure their systems to use
 the Google package repository.
 
+Once we graduate this KEP to beta, we'll strongly recommend using OBS
+instead of Google repos in order to migrate the load from Google to OBS as soon
+as possible. We also plan to deprecate and stop publishing to Google repos
+once the OBS implementation graduates to stable.
+
 ###### How can this feature be enabled / disabled in a live cluster?
 
 N/A. This is configured on the operating system (i.e. package manager) level. 
@@ -613,7 +672,7 @@ previous answers based on experience in the field.
 
 ###### How can an operator determine if the feature is in use by workloads?
 
-We'll ask openSUSE to provide us with metrics on the repository usage. We don't
+We'll ask SUSE to provide us with metrics on the repository usage. We don't
 have any metrics for the Google repository and there's no way that we can
 get those metrics.