Skip to content

Commit 82f4036

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request kubernetes#17818 from justinsb/discovery
Simple discovery server
2 parents 55074dc + 29cf465 commit 82f4036

File tree

44 files changed

+2735
-64
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+2735
-64
lines changed

‎cmd/kops/create_cluster.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,14 @@ func NewCmdCreateCluster(f *util.Factory, out io.Writer) *cobra.Command {
223223
return nil, cobra.ShellCompDirectiveNoFileComp
224224
})
225225

226+
if featureflag.DiscoveryService.Enabled() {
227+
cmd.Flags().StringVar(&options.PublicDiscoveryServiceURL, "discovery-service", options.PublicDiscoveryServiceURL, "A URL to a server implementing public OIDC discovery. Enables IRSA in AWS.")
228+
cmd.RegisterFlagCompletionFunc("discovery-service", func(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
229+
// TODO complete vfs paths
230+
return nil, cobra.ShellCompDirectiveNoFileComp
231+
})
232+
}
233+
226234
var validClouds []string
227235
{
228236
allClouds := clouds.SupportedClouds()

‎discovery/GEMINI.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Discovery Service Project
2+
3+
## Overview
4+
This project implements a public discovery service designed for decentralized, secure peer discovery. The core innovation is the use of **Custom Certificate Authorities (CAs)** to define isolated "Universes". Clients register and discover peers within their own Universe, identified and secured purely by mTLS.
5+
6+
The service emulates a Kubernetes API, allowing interaction via `kubectl`, including support for **Server-Side Apply**.
7+
8+
## Key Concepts
9+
10+
### 1. The "Universe"
11+
- A **Universe** is an isolated scope for peer discovery.
12+
- It is cryptographically defined by the **SHA256 hash of the Root CA's Public Key**.
13+
- Any client possessing a valid certificate signed by a specific CA belongs to that CA's Universe.
14+
- Different CAs = Different Universes. There is no crossover.
15+
16+
### 2. Authentication & Authorization
17+
- **Mechanism**: Mutual TLS (mTLS).
18+
- **Client Identity**: Derived from the **Common Name (CN)** of the leaf certificate.
19+
- **Universe Context**: Derived from the **Root CA** presented in the TLS handshake.
20+
- **Requirement**: Clients **MUST** present the full certificate chain (Leaf + Root CA) during the handshake. The server does not maintain a pre-configured trust store for these custom CAs; it uses the presented chain to determine the scope.
21+
22+
### 3. API Resources
23+
- **DiscoveryEndpoint** (`discovery.kops.k8s.io/v1alpha1`): Represents a peer in the discovery network. Can optionally hold OIDC configuration (Issuer URL, JWKS).
24+
- **Validation**: A client with CN `client1` can only Create/Update a `DiscoveryEndpoint` named `client1`.
25+
- **Apply Support**: The server supports `PATCH` requests to facilitate `kubectl apply --server-side`.
26+
27+
### 4. OIDC Discovery
28+
The server acts as an OIDC Discovery Provider for the Universe.
29+
- **Public Endpoints**:
30+
- `/.well-known/openid-configuration`: Returns the OIDC discovery document.
31+
- `/openid/v1/jwks`: Returns the JSON Web Key Set (JWKS).
32+
- **Data Source**: These endpoints serve data uploaded by clients via the `DiscoveryEndpoint` resource.
33+
34+
## Architecture
35+
36+
### Project Structure
37+
- `cmd/discovery-server/`: Main entry point. Wires up the HTTP server with TLS configuration.
38+
- `pkg/discovery/`:
39+
- `auth.go`: logic for inspecting TLS `PeerCertificates` to extract the Universe ID (CA hash) and Client ID.
40+
- `store.go`: In-memory thread-safe storage (`MemoryStore`) mapping Universe IDs to lists of `DiscoveryEndpoint` objects.
41+
- `server.go`: HTTP handlers implementing the K8s API emulation for `/apis/discovery.kops.k8s.io/v1alpha1`.
42+
- `k8s_types.go`: Definitions of `DiscoveryEndpoint`, `DiscoveryEndpointList`, `TypeMeta`, `ObjectMeta` etc.
43+
44+
### Data Model
45+
- **DiscoveryEndpoint**: The core resource. Contains `Spec.Addresses` and metadata.
46+
- **Universe**: Contains a map of `DiscoveryEndpoint` objects (keyed by name).
47+
- **Unified Types**: The API type `DiscoveryEndpoint` is used directly for in-memory storage, ensuring zero conversion overhead.
48+
49+
## Security Model
50+
- **Trust Delegation**: The server delegates trust to the CA. If you hold the CA key, you control the Universe.
51+
- **Isolation**: The server ensures that a client presenting a cert chain for `CA_A` cannot read or write data to the Universe defined by `CA_B`.
52+
- **Ephemeral**: The current implementation uses in-memory storage. Data is lost on restart.
53+
54+
## Building and Running
55+
56+
### Build
57+
```bash
58+
go build ./cmd/discovery-server
59+
```
60+
61+
### Run
62+
63+
See docs/walkthrough.md for instructions on testing functionality.

‎discovery/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# Discovery Service
2+
3+
A public discovery service using mTLS for authentication and "Universe" isolation, emulating a Kubernetes API.
4+
5+
## Concept
6+
7+
- **Universe**: Defined by the SHA256 Fingerprint of a Custom CA Certificate.
8+
- **Client**: Identified by a Client Certificate signed by that Custom CA.
9+
- **DiscoveryEndpoint**: The resource type representing a registered client.
10+
- **Isolation**: Clients can only see `DiscoveryEndpoint` objects signed by the same Custom CA.
11+
12+
## Usage
13+
14+
### Run Server
15+
16+
```bash
17+
go run ./cmd/discovery-server --tls-cert server.crt --tls-key server.key --listen :8443
18+
```
19+
20+
(You can generate a self-signed server certificate for testing, see the [walkthrough](docs/walkthrough.md) ).
21+
22+
### Client Requirement
23+
24+
Clients must authenticate using mTLS.
25+
**Important**: The client MUST provide the full certificate chain, including the Root CA, because the server does not have pre-configured trust stores for these custom universes.
26+
The server identifies the Universe from the SHA256 hash of the Root CA certificate found in the TLS chain.
27+
28+
### Quick start
29+
30+
See `docs/walkthrough.md` for detailed instructions.
31+
32+
33+
## OIDC Discovery
34+
35+
The discovery server also serves OIDC discovery information publicly, allowing external systems (like AWS IAM) to discover the cluster's identity provider configuration.
36+
37+
- `GET /<universe-id>/.well-known/openid-configuration`: Returns the OIDC discovery document.
38+
- `GET /<universe-id>/openid/v1/jwks`: Returns the JWKS.
39+
40+
This information is populated by clients uploading `DiscoveryEndpoint` resources containing the `oidc` spec.
41+
42+
## Building and Running

‎discovery/apis/discovery.kops.k8s.io/v1alpha1/types.go‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package v1alpha1
18+
19+
import (
20+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
21+
"k8s.io/apimachinery/pkg/runtime/schema"
22+
)
23+
24+
var DiscoveryEndpointGVR = schema.GroupVersionResource{
25+
Group: "discovery.kops.k8s.io",
26+
Version: "v1alpha1",
27+
Resource: "discoveryendpoints",
28+
}
29+
30+
var DiscoveryEndpointGVK = schema.GroupVersionKind{
31+
Group: "discovery.kops.k8s.io",
32+
Version: "v1alpha1",
33+
Kind: "DiscoveryEndpoint",
34+
}
35+
36+
// DiscoveryEndpoint represents a registered client in the discovery service.
37+
type DiscoveryEndpoint struct {
38+
metav1.TypeMeta `json:",inline"`
39+
metav1.ObjectMeta `json:"metadata,omitempty"`
40+
41+
Spec DiscoveryEndpointSpec `json:"spec,omitempty"`
42+
}
43+
44+
// DiscoveryEndpointSpec corresponds to our internal Node data.
45+
type DiscoveryEndpointSpec struct {
46+
Addresses []string `json:"addresses,omitempty"`
47+
LastSeen string `json:"lastSeen,omitempty"`
48+
OIDC *OIDCSpec `json:"oidc,omitempty"`
49+
}
50+
51+
type OIDCSpec struct {
52+
// IssuerURL string `json:"issuerURL,omitempty"`
53+
Keys []JSONWebKey `json:"keys,omitempty"`
54+
}
55+
56+
type JSONWebKey struct {
57+
Use string `json:"use,omitempty"`
58+
KeyType string `json:"kty,omitempty"`
59+
KeyID string `json:"kid,omitempty"`
60+
Algorithm string `json:"alg,omitempty"`
61+
N string `json:"n,omitempty"`
62+
E string `json:"e,omitempty"`
63+
// Crv string `json:"crv,omitempty"`
64+
// X string `json:"x,omitempty"`
65+
// Y string `json:"y,omitempty"`
66+
}
67+
68+
// DiscoveryEndpointList is a list of DiscoveryEndpoint objects.
69+
type DiscoveryEndpointList struct {
70+
metav1.TypeMeta `json:",inline"`
71+
// Standard list metadata.
72+
// We implement a minimal subset.
73+
Metadata metav1.ListMeta `json:"metadata,omitempty"`
74+
Items []DiscoveryEndpoint `json:"items"`
75+
}

‎discovery/apis/discovery.kops.k8s.io/v1alpha1/universeid.go‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package v1alpha1
18+
19+
import (
20+
"crypto/sha256"
21+
"crypto/x509"
22+
"encoding/hex"
23+
)
24+
25+
func ComputeUniverseIDFromCertificate(cert *x509.Certificate) string {
26+
hash := sha256.Sum256(cert.RawSubjectPublicKeyInfo)
27+
universeID := hex.EncodeToString(hash[:])
28+
return universeID
29+
}

‎discovery/cmd/discovery-server/main.go‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package main
18+
19+
import (
20+
"crypto/tls"
21+
"flag"
22+
"fmt"
23+
"log"
24+
"net/http"
25+
"os"
26+
27+
"k8s.io/kops/discovery/pkg/discovery"
28+
)
29+
30+
func main() {
31+
certFile := os.Getenv("TLS_CERT")
32+
flag.StringVar(&certFile, "tls-cert", certFile, "Path to server TLS certificate")
33+
34+
keyFile := os.Getenv("TLS_KEY")
35+
flag.StringVar(&keyFile, "tls-key", keyFile, "Path to server TLS key")
36+
37+
addr := flag.String("listen", ":8443", "Address to listen on")
38+
storageType := flag.String("storage", "memory", "Storage backend (memory, gcs)")
39+
flag.Parse()
40+
41+
if certFile == "" || keyFile == "" {
42+
fmt.Fprintf(os.Stderr, "Error: --tls-cert and --tls-key are required\n")
43+
flag.Usage()
44+
os.Exit(1)
45+
}
46+
47+
var store discovery.Store
48+
49+
switch *storageType {
50+
case "memory":
51+
store = discovery.NewMemoryStore()
52+
default:
53+
log.Fatalf("Unknown storage type: %s", *storageType)
54+
}
55+
56+
handler := discovery.NewServer(store)
57+
58+
tlsConfig := &tls.Config{
59+
ClientAuth: tls.RequestClientCert,
60+
// We do not set ClientCAs because we accept any CA and use it to define the universe.
61+
MinVersion: tls.VersionTLS12,
62+
}
63+
64+
server := &http.Server{
65+
Addr: *addr,
66+
Handler: handler,
67+
TLSConfig: tlsConfig,
68+
}
69+
70+
log.Printf("Discovery server listening on %s using %s storage", *addr, *storageType)
71+
if err := server.ListenAndServeTLS(certFile, keyFile); err != nil {
72+
log.Fatalf("Server failed: %v", err)
73+
}
74+
}

‎discovery/dev/tasks/deploy-to-k8s‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2025 The Kubernetes Authors.
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
set -o errexit
18+
set -o nounset
19+
set -o pipefail
20+
21+
REPO_ROOT="$(git rev-parse --show-toplevel)"
22+
cd "${REPO_ROOT}/discovery"
23+
24+
if [[ -z "${IMAGE_PREFIX:-}" ]]; then
25+
IMAGE_PREFIX="${USER}/"
26+
fi
27+
28+
IMAGE_TAG=$(date +%Y%m%d%H%M%S)
29+
30+
# Build the discovery-server image
31+
VERSION=${IMAGE_TAG} GITSHA=$(git describe --always) KO_DOCKER_REPO="${IMAGE_PREFIX}discovery-server" go run github.com/google/[email protected] \
32+
build --tags "${IMAGE_TAG}" --platform=linux/amd64,linux/arm64 --bare ./cmd/discovery-server/
33+
34+
echo "Can install cert-manager with the following command:"
35+
echo "kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml"
36+
37+
kubectl create namespace discovery-k8s-io --dry-run=client -o yaml | kubectl apply -f -
38+
39+
40+
cat k8s/manifest.yaml | \
41+
sed "s|discovery-server:latest|${IMAGE_PREFIX}discovery-server:${IMAGE_TAG}|g" | \
42+
KUBECTL_APPLYSET=true kubectl apply -n discovery-k8s-io --prune --applyset=discovery-server -f -

0 commit comments

Comments
 (0)