Skip to content

Commit 54efd9b

Browse files
xrstfxrstf
committed
add documentation
1 parent c47a6fe commit 54efd9b

File tree

4 files changed

+875
-0
lines changed

4 files changed

+875
-0
lines changed

‎docs/README.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# The Servlet
2+
3+
The Servlet is a Kubernetes agent responsible for integrating external Kubernetes clusters.
4+
It runs on a Kubernetes cluster, is configured with credentials to a kcp instance and will then
5+
synchronize data out of kcp (i.e. out of kcp workspaces) onto the local cluster, and vice versa.
6+
7+
The name Servlet is an obvious reference to the "kubelet" in a regular Kubernetes cluster.
8+
9+
## High-level Overview
10+
11+
The intended usecase follows roughly these steps:
12+
13+
1. A user in KDP with sufficient permissions creates an `APIExport` object and provides appropriate
14+
credentials for the Servlet (e.g. by creating a Kubernetes Secret with a preconfigured kubeconfig
15+
in it).
16+
3. A service owner will now take these credentials and the configured API group (the `APIExport`'s
17+
name) and use them to setup the Servlet. It is assumed that the service owner (i.e. the
18+
cluster-admin in a service cluster) wants to make some resources (usually CRDs) available to use
19+
inside of kcp.
20+
4. The service owner uses the Servlet Helm chart (or similar deployment technique) to install the
21+
Servlet in their cluster.
22+
5. To actually make resources available in the platform, the service owner now has to create a
23+
set of `PublishedResource` objects. The configuration happens from their point of view, meaning
24+
they define how to publish a CRD to the platform, defining renaming rules and other projection
25+
settings.
26+
6. Once a `PublishedResource` is created in the service cluster, the Servlet will pick it up,
27+
find the referenced CRD, convert/project this CRD into an `APIResourceSchema` (ARS) for kcp and
28+
then create the ARS in org workspace.
29+
7. Finally the Servlet will take all `PublishedResources` and bundle them into the pre-existing
30+
`APIExport` in the org workspace. This APIExport can then be bound in the org workspace itself
31+
(or later any workspaces (depending on permissions)) and be used there.
32+
8. kcp automatically provides a virtual workspace for the `APIExport` and this is what the Servlet
33+
then uses to watch all objects for the relevant resources in the platform (i.e. in all workspaces).
34+
9. The Servlet will now begin to synchronize objects back and forth between the service cluster
35+
and KDP.
36+
37+
## Details
38+
39+
### Data Flow Direction
40+
41+
It might be a bit confusing at first: The `PublishedResource` CRD describes the world from the
42+
standpoint of a service owner, i.e. a person or team that owns a Kubernetes cluster and is tasked
43+
with making their CRDs available in kcp (i.e. "publish" them).
44+
45+
However the actual data flow later will work in the opposite direction: users creating objects inside
46+
their kcp workspaces serve as the source of truth. From there they are synced down to the service
47+
cluster, which is doing the projection of the `PublishedResource` _in reverse_.
48+
49+
Of course additional, auxiliary (related) objects could originate on the service cluster. For example
50+
if you create a Certificate object in a kcp workspace and it's synced down, cert-manager will then
51+
acquire the certificate and create a Kubernetes `Secret`, which will have to be synced back up (into
52+
a kcp workspace, where the certificate originated from). So the source of truth can also be, for
53+
auxiliary resources, on the service cluster.
54+
55+
### Servlet Naming
56+
57+
Each Servlet must have a name, like "nora" or "oskar". The FQ name for a Servlet is
58+
`<servletname>.<apigroup>`, so if the user in KDP had created a new `APIExport` named
59+
`databases.examplecorp`, the name of the Servlet that serves this Service (sic) could be
60+
`nora.databases.examplecorp`.
61+
62+
### Uniqueness
63+
64+
A single `APIExport` in kcp must only be processed by exactly 1 Servlet. There is currently no
65+
mechanism planned to subdivide an `APIExport` into shards, where multiple service clusters (and
66+
therefore multiple Servlets) could process each shard.
67+
68+
Later the Servlet might be extended with Label Selectors, alternatively they might also "claim" any
69+
object by annotating it in the kcp workspace. These things are not yet worked out, so for now we have
70+
this 1:1 restriction.
71+
72+
Servlets make use of leader election, so it's perfectly fine to have multiple Servlet replicas, as
73+
long as only one them is leader and actually doing work.
74+
75+
### kcp-awareness
76+
77+
controller-runtime can be used in a "kcp-aware" mode, where the cache, clients, mappers etc. are
78+
aware of the workspace information. This however is neither well tested upstream and the code would
79+
require shard-admin permissions to behave like this work regular kcp workspaces. The controller-runtime
80+
fork's kcp-awareness is really more geared towards working in virtual workspaces.
81+
82+
Because of this the Servlet needs to get a kubeconfig to kcp that already points to the `APIExport`'s
83+
workspace (i.e. the `server` URL already contains a `/clusters/root:myorg` path). The basic
84+
controllers in the Servlet then treat this as a plain ol', regular Kubernetes cluster
85+
(no kcp-awareness).
86+
87+
To this end, the Servlet will, upon startup, try to access the `cluster` object in the target
88+
workspace. This is to resolve the cluster name (e.g. `root:myorg`) into a logicalcluster name (e.g.
89+
`gibd3r1sh`). The Servlet has to know which logicalcluster the target workspace represents in order
90+
to query resources properly.
91+
92+
Only the controllers that are later responsible for interacting with the virtual workspace are
93+
kcp-aware. They have to be in order to know what workspace a resource is living in.
94+
95+
### PublishedResources
96+
97+
A `PublishedResource` describes which CRD should be made available inside kcp. The CRD name can be
98+
projected (i.e. renamed), so a `kubermatic.k8c.io/v1 Cluster` can become a
99+
`cloud.examplecorp/v1 KubernetesCluster`.
100+
101+
In addition to projecting (mapping) the GVK, the `PublishedResource` also contains optional naming
102+
rules, which influence how the local objects that the Servlet is creating are named.
103+
104+
As a single Servlet serves a single service, the API group used in kcp is the same for all
105+
`PublishedResources`. It's the API group configured in the `APIExport` inside the platform (created
106+
in step 1 in the overview above).
107+
108+
To prevent chaos, `PublishedResources` are immutable: handling the case that a PR first wants to
109+
publish `kubermatic.k8c.io/v1 Cluster` and then suddenly `kubermatic.k8c.io/v1 User` resources would
110+
mean to re-sync and cleanup everything in all affected kcp workspaces. The Servlet would need to be
111+
able to delete and recreate objects to follow this GVK change, which is a level of complexity we
112+
simply do not want to deal with at this point in time. Also, `APIResourceSchemas` are immutable
113+
themselves.
114+
115+
More information is available in the [Publishing Resources][publish-resources.md] guide.
116+
117+
### APIExports
118+
119+
An `APIExport` in kcp combines multiple `APIResourceSchemas` (ARS). Each ARS is created based on a
120+
`PublishedResource` in the service cluster.
121+
122+
To prevent data loss, ARS are never removed from an `APIExport`. We simply do not have enough
123+
experience to really know what happens when an ARS would suddenly become unavailable. To prevent
124+
damage and confusion, the Servlet will only ever add new ARS to the one `APIExport` it manages.
125+
126+
## Controllers
127+
128+
The Servlet consists of a number of independent controllers.
129+
130+
### apiexport
131+
132+
This controller aggregates the `PublishedResources` and manages a single `APIExport` in KDP.
133+
134+
### apiresourceschema
135+
136+
This controller takes `PublishedResources`, projects and converts them and creates `APIResourceSchemas`
137+
in KDP.
138+
139+
### syncmanager
140+
141+
This controller watches the `APIExport` and waits for the virtual workspace to become available. It
142+
also watches all `PublishedResources` (PRs) and reconciles when any of them is changed (they are
143+
immutable, but the controller is still reacting to any events on them).
144+
145+
The controller will then setup a controller-runtime `Cluster` abstraction for the virtual workspace
146+
and then start many `sync` controllers (one for each `PublishedResource`). Whenever PRs change, the
147+
syncmanager will make sure that the correct set of `sync` controller is running.
148+
149+
### sync
150+
151+
This is where the meat and potatoes happen. The sync controller is started for a single
152+
`PublishedResource` and is responsible for synchronizing all objects for that resource between the
153+
local service cluster and kcp.
154+
155+
The `sync` controller was written to handle a single `PublishedResource` so that it does not have to
156+
deal with dynamically registering/stopping watches on its own. Instead the sync controller can be
157+
written as more or less "normal" controller-runtime controller.

‎docs/consuming-services.md‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Consuming Services
2+
3+
This document describes how to use (consume) services offered by a Servlet.
4+
5+
## Background
6+
7+
A "service" defines a unique Kubernetes API Group and offers a number of resources (types) to
8+
use. A service could offer certificate management, databases, cloud infrastructure or any other set
9+
of Kubernetes resources.
10+
11+
Services are provided by service owners, who run their own Kubernetes clusters and take care of the
12+
maintenance and scaling tasks for the workload provisioned by all users of the service(s) they
13+
offer.
14+
15+
A Service provided by a Servlet should not be confused with a Kubernetes Service. Internally, a
16+
"Servlet Service" is ultimately translated into a kcp `APIExport` with a number of
17+
`APIResourceSchemas` (which are more or less equivalent to CRDs).
18+
19+
## Consuming a Service
20+
21+
To consume a service (or to make use of an `APIExport`) you have to create an `APIBinding` object
22+
in the kcp workspace where the servie should be used. This section assumes that you are familiar
23+
with kcp on the command line and have the kcp kubectl plugin installed.
24+
25+
First you need to get the kubeconfig for accessing your kcp workspaces. Once you have set your
26+
kubeconfig up, make sure you're in the correct namespace by using
27+
`kubectl ws <path to your workspace>`. Use `kubectl ws .` if you're unsure where you're at.
28+
29+
To enable a Service, use `kcp bind apiexport` and specify the path to and name of the `APIExport`.
30+
31+
```bash
32+
# kubectl kcp bind apiexport <path to KDP Service>:<API Group of the Service>
33+
kubectl kcp bind apiexport :root:my-org:my.fancy.api
34+
```
35+
36+
Without the plugin, you can create an `APIBinding` manually, simply `kubectl apply` this:
37+
38+
```yaml
39+
apiVersion: apis.kcp.io/v1alpha1
40+
kind: APIBinding
41+
metadata:
42+
name: my.fancy.api
43+
spec:
44+
reference:
45+
export:
46+
name: my.fancy.api
47+
path: root:my-org
48+
```
49+
50+
Shortly after, the new API will be available in the workspace. Check via `kubectl api-resources`.
51+
You can now create objects for types in that API group to your liking and they will be synced and
52+
processed behind the scenes.
53+
54+
Note that a Service often has related resources, often Secrets and ConfigMaps. You must explicitly
55+
allow the Service to access these in your workspace and this means editing/patching the `APIBinding`
56+
object (the kcp kubectl plugin currently has no support for managing permission claims). For each of
57+
the claimed resources, you have to accept or reject them:
58+
59+
```yaml
60+
spec:
61+
permissionClaims:
62+
# Nearly all Servlets require access to namespaces, rejecting this will
63+
# most likely break the Service, even more than rejecting any other claim.
64+
- all: true
65+
resources: namespaces
66+
state: Accepted
67+
- all: true
68+
resources: secrets
69+
state: Accepted # or Rejected
70+
```
71+
72+
Rejecting a claim will severely impact a Service, if not even break it. Consult with the Service's
73+
documentation or the service owner if rejecting a claim is supported.
74+
75+
When you _change into_ (`kubctl ws …`) a different workspace, kubectl will inform you if there are
76+
outstanding permission claims that you need to accept or reject.

‎docs/getting-started.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# Getting Started with the Servlet
2+
3+
All that is necessary to run the Servlet is a running Kubernetes cluster (for testing you can use
4+
[kind][kind]) [kcp][kcp] installation.
5+
6+
## Prerequisites
7+
8+
- A running Kubernetes cluster to run the Servlet in.
9+
- A running kcp installation as the source of truth.
10+
- A kubeconfig with admin or comparable permissions in a specific kcp workspace.
11+
12+
## APIExport Setup
13+
14+
Before installing the Servlet it is necessary to create an `APIExport` on kcp. The `APIExport` should
15+
be empty, because it is updated later by the Servlet, but it defines the new API group we're
16+
introducing. An example file could look like this:
17+
18+
```yaml
19+
apiVersion: apis.kcp.io/v1alpha1
20+
kind: APIExport
21+
metadata:
22+
name: test.example.com
23+
spec: {}
24+
```
25+
26+
Create a file with a similar content (you most likely want to change the name, as that is the API
27+
group under which your published resources will be made available) and create it in a kcp workspace
28+
of your choice:
29+
30+
```sh
31+
# use the kcp kubeconfig
32+
$ export KUBECONFIG=/path/to/kcp.kubeconfig
33+
34+
# nativagate to the workspace wher the APIExport should exist
35+
$ kubectl ws :workspace:you:want:to:create:it
36+
37+
# create it
38+
$ kubectl create --filename apiexport.yaml
39+
apiexport/test.example.com created
40+
```
41+
42+
## Servlet Installation
43+
44+
The Servlet can be installed into any namespace, but in our example we are going with `k8c-system`.
45+
It doesn't necessarily have to live in the same Kubernetes cluster where it is synchronizing data
46+
to, but that is the common setup. Ultimately the Servlet synchronizes data between two kube
47+
endpoints.
48+
49+
Now that the `APIExport` is created, switch to the Kubernetes cluster from which you wish to
50+
[publish resources](publish-resources.md). You will need to ensure that a kubeconfig with access to
51+
the kcp workspace that the `APIExport` has been created in is stored as a `Secret` on this cluster.
52+
Make sure that the kubeconfig points to the right workspace (not necessarily the `root` workspace).
53+
54+
This can be done via a command like this:
55+
56+
```sh
57+
$ kubectl create secret generic kcp-kubeconfig \
58+
--namespace k8c-system \
59+
--from-file "kubeconfig=admin.kubeconfig"
60+
```
61+
62+
The Servlet is shipped as a Helm chart and to install it, the next step is preparing a `values.yaml`
63+
file for the Servlet Helm chart. We need to pass the target `APIExport`, a name for the Servlet
64+
itself and a reference to the kubeconfig secret we just created.
65+
66+
```yaml
67+
servlet:
68+
# Required: the name of the APIExport in kcp that this Servlet is supposed to serve.
69+
apiExportName: test.example.com
70+
71+
# Required: this Servlet's public name, will be shown in kcp, purely for informational purposes.
72+
servletName: unique-test
73+
74+
# Required: Name of the Kubernetes Secret that contains a "kubeconfig" key, with the kubeconfig
75+
# provided by kcp to access it.
76+
platformKubeconfig: kcp-kubeconfig
77+
78+
# Create additional RBAC on the service cluster. These rules depend somewhat on the Servlet
79+
# configuration, but the following two rules are very common. If you configure the Servlet to
80+
# only work with cluster-scoped objects, you do not need to grant it permissions to create
81+
# namespaces, for example.
82+
rbac:
83+
createClusterRole: true
84+
rules:
85+
# in order to create APIResourceSchemas
86+
- apiGroups:
87+
- apiextensions.k8s.io
88+
resources:
89+
- customresourcedefinitions
90+
verbs:
91+
- get
92+
- list
93+
- watch
94+
# so copies of remote objects can be placed in their target namespaces
95+
- apiGroups:
96+
- ""
97+
resources:
98+
- namespaces
99+
verbs:
100+
- get
101+
- list
102+
- watch
103+
- create
104+
```
105+
106+
In addition, it is important to create RBAC rules for the resources you want to publish. If you want
107+
to publish the `Certificate` resource as created by cert-manager, you will need to append the
108+
following ruleset:
109+
110+
```yaml
111+
# so we can manage certificates
112+
- apiGroups:
113+
- cert-manager.io
114+
resources:
115+
- certificates
116+
verbs:
117+
- '*'
118+
```
119+
120+
Once this `values.yaml` file is prepared, install a recent development build of the Servlet:
121+
122+
```sh
123+
helm install servlet oci://quay.io/kubermatic/helm-charts/kdp-servlet --version 9.9.9-9fc9a430d95f95f4b2210f91ef67b3ec153b5cab -f values.yaml -n k8c-system
124+
```
125+
126+
Two `servlet` Pods should start in the `k8c-system` namespace. If they crash you will need to
127+
identify the reason from container logs. A possible issue is that the provided kubeconfig does not
128+
have permissions against the target kcp workspace.
129+
130+
## Publish Resources
131+
132+
Once the Servlet Pods are up and running, you should be able to follow the
133+
[Publishing Resources](publish-resources.md) guide.
134+
135+
## Consume Service
136+
137+
Once resources have been published through the Servlet, they can be consumed on the kcp side (i.e.
138+
objects on kcp will be synced back and forth with the service cluster). Follow the
139+
guide to [consuming services](consuming-services.md).
140+
141+
[kind]: https://github.com/kubernetes-sigs/kind
142+
[kcp]: https://kcp.io

0 commit comments

Comments
 (0)