Skip to content

Commit 93c4948

Browse files
authored
Merge pull request #87 from xrstf/shard-indexing
Deploy internal front-proxy
2 parents 18dadc4 + 16531c4 commit 93c4948

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2638
-377
lines changed

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,3 +27,6 @@ go.work.sum
2727
# Downloaded and built binaries
2828
/_build/
2929
/_tools/
30+
31+
# e2e test logs and artifacts
32+
/.e2e-*

.golangci.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,6 @@ linters:
66
enable:
77
- copyloopvar
88
- errcheck
9-
- ginkgolinter
109
- goconst
1110
- gocyclo
1211
- govet

Makefile

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -65,9 +65,14 @@ test: fmt vet ## Run tests.
6565
go test $$(go list ./... | grep -v /e2e) -coverprofile cover.out
6666

6767
# Utilize Kind or modify the e2e tests to load the image locally, enabling compatibility with other vendors.
68-
.PHONY: test-e2e # Run the e2e tests against a Kind k8s instance that is spun up.
68+
.PHONY: test-e2e # Run the e2e tests against a kind k8s instance that is already spun up.
6969
test-e2e:
70-
go test ./test/e2e/ -v -ginkgo.v
70+
go test ./test/e2e/ -v
71+
72+
# Creates a kind cluster and runs the e2e tests in them. The kind cluster is destroyed after the tests.
73+
.PHONY: test-e2e-with-kind # Run the e2e tests against a temporary kind cluster.
74+
test-e2e-with-kind: helm
75+
@./hack/run-e2e-tests.sh
7176

7277
.PHONY: lint
7378
lint: golangci-lint ## Run golangci-lint linter.

config/crd/bases/operator.kcp.io_frontproxies.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -355,7 +355,7 @@ spec:
355355
type: object
356356
description: |-
357357
CertificateTemplates allows to customize the properties on the generated
358-
certificates for this root shard.
358+
certificates for this front-proxy.
359359
type: object
360360
deploymentTemplate:
361361
description: 'Optional: DeploymentTemplate configures the Kubernetes

config/crd/bases/operator.kcp.io_rootshards.yaml

Lines changed: 1443 additions & 1 deletion
Large diffs are not rendered by default.

config/crd/bases/operator.kcp.io_shards.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -425,7 +425,7 @@ spec:
425425
type: object
426426
description: |-
427427
CertificateTemplates allows to customize the properties on the generated
428-
certificates for this root shard.
428+
certificates for this shard.
429429
type: object
430430
clusterDomain:
431431
type: string

docs/content/.pages

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
nav:
22
- Home: README.md
33
- Setup: setup
4+
- Architecture: architecture
45
- Reference: reference
56
- Contributing: contributing

docs/content/architecture/.pages

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
nav:
2+
- index.md
3+
- basics.md
4+
- front-proxy.md
5+
- Certificate Management: pki.md
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
---
2+
description: >
3+
A general overview over the resources provided by the kcp-operator.
4+
---
5+
6+
# Basics
7+
8+
kcp-operator ships with a number of resources that, together, can be used to create a kcp installation.
9+
10+
## Resource Relationships
11+
12+
```mermaid
13+
flowchart TD
14+
FrontProxy -- ObjRef (n:1)--> RootShard
15+
Shard --ObjRef (n:1)--> RootShard
16+
RootShard --ObjRef (1:1, optional)--> CacheServer
17+
Shard --ObjRef (n:1, optional)--> CacheServer
18+
Kubeconfig --ObjRef (n:1)-->Shard
19+
Kubeconfig --ObjRef (n:1)-->RootShard
20+
Kubeconfig --ObjRef (n:1)-->FrontProxy
21+
```
22+
23+
### `RootShard` & `Shard`
24+
25+
Each kcp installation consists at the minimum of one root shard and one [front-proxy](front-proxy.md), but you can add additional "regular" shards to scale kcp horizontally.
26+
27+
Creating a new `RootShard` object means creating a new kcp installation, as you cannot have more than one root shard in one kcp installation, however multiple installations can run in the same Kubernetes namespace (though this is not necessarily recommended). The `RootShard` object's name is not really important inside kcp itself.
28+
29+
Each additional shard is created by creating a `Shard` object, which will reference the root shard it belongs to. Shard names are relevant in kcp, as each shard will register itself on its root shard, using the name of the `Shard` object.
30+
31+
### `FrontProxy`
32+
33+
The kcp front-proxy can be used to provide access to either a whole or a subset of a kcp installation. Its main purpose is to act as a gateway, since it builds up a runtime map of all existing workspaces across all shards that it targets, so it knows where to route a request to `/clusters/root:my-team` to the shard where the logicalcluster for that workspace resides.
34+
35+
A kcp installation can contain multiple front-proxies with different settings; one might perform additional authentication while another might pass unauthenticated requests to the shards (which will then perform their own authentication).
36+
37+
For developing controllers against kcp, it is often necessary to access the shards directly, so front-proxies are not the only way to access kcp.
38+
39+
### `Kubeconfig`
40+
41+
Kubeconfigs allow the easy creation of credentials to access kcp. As a sharded system, kcp relies on client certificate authentication and the kcp-operator will ensure the correct certificates are generated and then neatly wrapped up in ready-to-use kubeconfig Secrets.
42+
43+
Kubeconfigs can be configured to point to a specific shard or to a front-proxy instance, which affects which client CA is used to generate the certificates.
44+
45+
## Cross-Namespace/Cluster References
46+
47+
Due to the potential "global" nature of a kcp setup it might be necessary to run kcp-operator on multiple clusters while attempting to form one single kcp setup with multiple shards and front proxies.
48+
49+
To make this possible, resources with object references (see above) could have a secondary way of reading necessary configuration (instead of a `corev1.LocalObjectReference`). This could be a reference to a `ConfigMap` or a `Secret` (to be determined) which are automatically generated for various resource types. A sync process (outside of the kcp-operator) could then sync the `ConfigMap` (or the `Secret`, or a custom resource type) across namespaces or even clusters, where e.g. a `Shard` object references a `Secret` which was generated for a `RootShard` on another cluster.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
description: >
3+
Explains how the kcp front-proxy can be used to ingest traffic.
4+
---
5+
6+
# kcp front-proxy
7+
8+
The kcp front-proxy can be used to provide access to either a whole or a subset of a kcp installation. Its main purpose is to act as a gateway, since it builds up a runtime map of all existing workspaces across all shards that it targets, so it knows where to route a request to `/clusters/root:my-team` to the shard where the logicalcluster for that workspace resides.
9+
10+
!!! warning
11+
Currently it is required to have at least one front-proxy in a kcp installation, which must be reachable at the hostname also configured as the external name for the root shard. Without a front-proxy, new workspaces will not be able to be scheduled.
12+
13+
The front-proxy must exist even if there is only one shard (the RootShard). This is a requirement of kcp, not the operator itself.
14+
15+
## Usage
16+
17+
As mentioned above, your first front-proxy should have the same external hostname as is configured on the root shard. Every additional front-proxy can be served at arbitrary hostnames. Since the CA used to generate the serving certs for front-proxies is embedded in kubeconfigs generated by the kcp-operator, there is no need to use well-known CAs or ACME providers like Let's Encrypt, unless you intend to also access kcp from systems that do not use kubeconfig files and have no way to configure additional CAs.
18+
19+
## Mode of Operation
20+
21+
The kcp front-proxy will build up a runtime map of all shards, workspaces and logicalclusters in a kcp installation. For this it connects dynamically to each shard individually and begins to watch the necessary resources. This runtime index is then used to route incoming requests to their target shards.
22+
23+
The proxy can optionally perform authentication, for example using OIDC, and pass authentication information (like username and groups) on to the shard (via HTTP headers). This can be configured for each front-proxy individually. If no authentication is performed, the requests will be passed unauthenticated to their target shards, where then authentication happens.
24+
25+
## RootShard Proxy
26+
27+
The kcp-operator will deploy an internal front-proxy for every `RootShard` (i.e. one for each kcp installation). This internal proxy is solely used by the operator itself to allow it to resolve workspace paths to logicalclusters and provision resources inside those workspaces.
28+
29+
The reason a standalone proxy is deployed is that the admin-configured front-proxies (based on `FrontProxy` objects) can be configured to drop sensitive groups, like `system:masters`, from incoming requests. This is a security measure, ensuring no super admin access is possible to kcp from the outside.

0 commit comments

Comments
 (0)