Skip to content

[WIP]:sparkles: Enhance ClusterProxy API. #401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
xuezhaojun wants to merge 1 commit into from
Closed

[WIP]:sparkles: Enhance ClusterProxy API. #401

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions feature/feature.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,26 @@ const (
// When enabled, the work controller will automatically clean up completed manifest works based on the configured
// time-to-live duration to prevent accumulation of old completed resources.
CleanUpCompletedManifestWork featuregate.Feature = "CleanUpCompletedManifestWork"

// ClusterProxy integrates cluster-proxy functionality directly into the klusterlet-agent, enabling
// HTTP-based proxying to managed cluster API servers through gRPC tunnels.
//
// When enabled on the hub (via ClusterManager), it starts a gRPC server that provides an externally accessible
// HTTP endpoint to proxy requests to managed cluster API servers. The API path format is:
// https://<server-address>:<port>/<cluster-name>
//
// When enabled on the spoke (via Klusterlet), the agent establishes a gRPC connection to the hub and proxies
// requests to its local API server. The agent sends a CSR with signer name "open-cluster-management.io/klusterlet-proxy"
// to obtain the gRPC configuration, which is stored in the hub-kubeconfig-secret as "proxy-grpc.yaml".
//
// This feature requires gRPC configuration in ClusterManager.spec.grpcConfiguration with the ClusterProxy
// feature gate enabled. Users can authenticate using either userToken or impersonation methods.
//
// Use cases include: fetching pod logs, accessing VM consoles (kubevirt), multicluster job submission (MultiKueue),
// and multicluster apiserver access (Kubernetes MCP).
//
// When disabled, the legacy cluster-proxy addon should be used instead for proxy functionality.
ClusterProxy featuregate.Feature = "ClusterProxy"
)

// DefaultSpokeRegistrationFeatureGates consists of all known ocm-registration
Expand Down Expand Up @@ -137,3 +157,7 @@ var DefaultSpokeWorkFeatureGates = map[featuregate.Feature]featuregate.FeatureSp
ExecutorValidatingCaches: {Default: false, PreRelease: featuregate.Alpha},
RawFeedbackJsonString: {Default: false, PreRelease: featuregate.Alpha},
}

var DefaultServerConfigFeatureGates = map[featuregate.Feature]featuregate.FeatureSpec{
ClusterProxy: {Default: false, PreRelease: featuregate.Alpha},
}
106 changes: 106 additions & 0 deletions operator/v1/0000_00_operator.open-cluster-management.io_klusterlets.crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,112 @@ spec:
deployed klusterlet agent. It will be ignored when the PriorityClass/v1 API
is not available on the managed cluster.
type: string
proxyConfig:
description: |-
ProxyConfig holds the configuration for enabling klusterlet-proxy functionality,
which allows the hub cluster to access the managed cluster's API server through
a gRPC-based proxy tunnel established by the klusterlet agent.

When configured, the klusterlet agent establishes a gRPC connection to the hub's
proxy server and proxies incoming HTTP requests to the local managed cluster API server.
This enables hub-to-spoke API access even when the managed cluster is not directly
accessible from the hub (e.g., behind a firewall or NAT).

This feature requires the ClusterProxy feature gate to be enabled and corresponding
GRPCConfiguration to be set in the ClusterManager on the hub side.
properties:
authentications:
default:
- userToken
description: |-
Authentications defines how the agent authenticates with the cluster.
If not specified, defaults to ["userToken"].
items:
enum:
- userToken
- impersonation
type: string
type: array
grpcEndpoint:
description: GRPCEndpoint represents the gRPC endpoint configuration
for the proxy connection.
properties:
grpc:
description: grpc represents the configuration for grpc endpoint.
properties:
hostname:
description: hostname points to a fixed hostname for serving
agents' handshakes.
properties:
caBundle:
description: caBundle of the endpoint.
format: byte
type: string
host:
description: host is the host name of the endpoint.
type: string
required:
- host
type: object
type:
default: hostname
description: |-
type specifies how the endpoint is exposed.
You may need to apply an object to expose the endpoint, for example: a route.
enum:
- hostname
type: string
required:
- type
type: object
https:
description: https represents the configuration for https
endpoint.
properties:
hostname:
description: hostname points to a fixed hostname for serving
agents' handshakes.
properties:
caBundle:
description: caBundle of the endpoint.
format: byte
type: string
host:
description: host is the host name of the endpoint.
type: string
required:
- host
type: object
type:
default: hostname
description: |-
type specifies how the endpoint is exposed.
You may need to apply an object to expose the endpoint, for example: a route.
enum:
- hostname
type: string
required:
- type
type: object
protocol:
default: grpc
description: protocol is the protocol used for the endpoint,
could be https or grpc.
enum:
- grpc
- https
type: string
usage:
description: |-
usage defines the usage of the endpoint. It could be "agentToHub" indicating the endpoint is used
for communication between agent and hub, or "consumer" indicating the endpoint is used for external consumer.
type: string
required:
- protocol
type: object
required:
- grpcEndpoint
type: object
Comment on lines +184 to +289
Copy link

@coderabbitai coderabbitai bot Oct 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

🧩 Analysis chain

Harden schema: enforce grpc/https exclusivity, add list uniqueness, and validate usage.

Without cross-field rules, invalid configs (e.g., protocol=grpc but only https set, or both set) will pass CRD validation and fail at runtime. Also, authentications can duplicate entries, and usage is free-form.

Suggested CRD edits:

  • Enforce exactly-one of grpc/https matching protocol using CEL.
  • Make authentications a set and bound its size.
  • Constrain usage to known values and set a default.
  • Tighten hostname.host to FQDN pattern and length.
   proxyConfig:
     properties:
       authentications:
         items:
           enum:
           - userToken
           - impersonation
           type: string
         type: array
+        x-kubernetes-list-type: set
+        minItems: 1
+        maxItems: 2

       grpcEndpoint:
         description: GRPCEndpoint represents the gRPC endpoint configuration
           for the proxy connection.
         properties:
           grpc:
             properties:
               hostname:
                 properties:
                   caBundle:
                     format: byte
                     type: string
                   host:
                     description: host is the host name of the endpoint.
-                    type: string
+                    type: string
+                    maxLength: 253
+                    pattern: ^(([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$
...
           https:
             properties:
               hostname:
                 properties:
                   caBundle:
                     format: byte
                     type: string
                   host:
-                    type: string
+                    type: string
+                    maxLength: 253
+                    pattern: ^(([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$
...
           protocol:
             default: grpc
             enum:
             - grpc
             - https
             type: string
           usage:
-            description: |-
+            description: |-
               usage defines the usage of the endpoint. It could be "agentToHub" indicating the endpoint is used
               for communication between agent and hub, or "consumer" indicating the endpoint is used for external consumer.
-            type: string
+            type: string
+            enum:
+            - agentToHub
+            - consumer
+            default: agentToHub
         required:
         - protocol
         type: object
+        x-kubernetes-validations:
+        - rule: "self.protocol == 'grpc' ? has(self.grpc) : has(self.https)"
+          message: "Provide grpc when protocol=grpc, and https when protocol=https."
+        - rule: "!(has(self.grpc) && has(self.https))"
+          message: "Specify only one of grpc or https."

Run to confirm validations materialize after regeneration:

🏁 Script executed:

#!/usr/bin/env bash
# After applying Go marker changes + regen, check CRD for new validations.
fd 'klusterlets.crd.yaml' -a | xargs -I{} rg -n 'x-kubernetes-validations|x-kubernetes-list-type|enum:\s+- agentToHub' {}

Length of output: 55

I need to verify the current state of the schema before finalizing the rewritten review comment. Let me check the actual content of the CRD file.


---

🏁 Script executed:

```shell
#!/bin/bash
# Check the actual current state of authentications, usage, and grpcEndpoint fields
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} cat {} | grep -A 50 "authentications:" | head -60

Length of output: 2412

🏁 Script executed:

#!/bin/bash
# Check for usage field and its current definition
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} rg -A 5 -B 2 '^\s+usage:' {}

Length of output: 464

🏁 Script executed:

#!/bin/bash
# Check grpcEndpoint structure for grpc/https fields
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} rg -n 'grpc:|https:' {} | head -20

Length of output: 550

🏁 Script executed:

#!/bin/bash
# Check for any x-kubernetes-validations already present in the file
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} rg -B 3 -A 3 'x-kubernetes-validations' {}

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Check hostname.host field for patterns/constraints
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} rg -A 3 'host is the host name' {}

Length of output: 406

🏁 Script executed:

#!/bin/bash
# Count occurrences of grpc and https at the same level (both present?)
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} rg -n 'properties:' {} | grep -i grpc

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Look at grpcEndpoint properties structure
fd 'klusterlets.crd.yaml' -a | head -1 | xargs -I{} sed -n '184,289p' {}

Length of output: 5270

CRD is missing critical validation rules: enforce protocol-endpoint matching, add authentications list constraints, validate usage values, and constrain hostname format.

The schema currently lacks cross-field validation and type constraints that allow invalid configurations:

  • grpcEndpoint: Both grpc and https are optional; no rule ensures exactly one matches protocol (e.g., protocol=grpc but only https defined will pass CRD validation and fail at runtime).
  • authentications: Unconstrained array—duplicates are allowed, and no size bounds.
  • usage: Free-form string with no enum; accepts any value despite documentation indicating only "agentToHub" or "consumer" are valid.
  • hostname.host: No FQDN pattern or length validation.

Apply the following to harden the schema:

   proxyConfig:
     properties:
       authentications:
         items:
           enum:
           - userToken
           - impersonation
           type: string
         type: array
+        x-kubernetes-list-type: set
+        minItems: 1
+        maxItems: 2

       grpcEndpoint:
         description: GRPCEndpoint represents the gRPC endpoint configuration
           for the proxy connection.
         properties:
           grpc:
             properties:
               hostname:
                 properties:
                   caBundle:
                     format: byte
                     type: string
                   host:
                     description: host is the host name of the endpoint.
                     type: string
+                    maxLength: 253
+                    pattern: ^(([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$
           https:
             properties:
               hostname:
                 properties:
                   caBundle:
                     format: byte
                     type: string
                   host:
                     description: host is the host name of the endpoint.
                     type: string
+                    maxLength: 253
+                    pattern: ^(([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$
           protocol:
             default: grpc
             enum:
             - grpc
             - https
             type: string
           usage:
             description: |-
               usage defines the usage of the endpoint. It could be "agentToHub" indicating the endpoint is used
               for communication between agent and hub, or "consumer" indicating the endpoint is used for external consumer.
             type: string
+            enum:
+            - agentToHub
+            - consumer
+            default: agentToHub
         required:
         - protocol
         type: object
+        x-kubernetes-validations:
+        - rule: "self.protocol == 'grpc' ? has(self.grpc) : has(self.https)"
+          message: "Provide grpc when protocol=grpc, and https when protocol=https."
+        - rule: "!(has(self.grpc) && has(self.https))"
+          message: "Specify only one of grpc or https."
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
proxyConfig:
description: |-
ProxyConfig holds the configuration for enabling klusterlet-proxy functionality,
which allows the hub cluster to access the managed cluster's API server through
a gRPC-based proxy tunnel established by the klusterlet agent.
When configured, the klusterlet agent establishes a gRPC connection to the hub's
proxy server and proxies incoming HTTP requests to the local managed cluster API server.
This enables hub-to-spoke API access even when the managed cluster is not directly
accessible from the hub (e.g., behind a firewall or NAT).
This feature requires the ClusterProxy feature gate to be enabled and corresponding
GRPCConfiguration to be set in the ClusterManager on the hub side.
properties:
authentications:
default:
- userToken
description: |-
Authentications defines how the agent authenticates with the cluster.
If not specified, defaults to ["userToken"].
items:
enum:
- userToken
- impersonation
type: string
type: array
grpcEndpoint:
description: GRPCEndpoint represents the gRPC endpoint configuration
for the proxy connection.
properties:
grpc:
description: grpc represents the configuration for grpc endpoint.
properties:
hostname:
description: hostname points to a fixed hostname for serving
agents' handshakes.
properties:
caBundle:
description: caBundle of the endpoint.
format: byte
type: string
host:
description: host is the host name of the endpoint.
type: string
required:
- host
type: object
type:
default: hostname
description: |-
type specifies how the endpoint is exposed.
You may need to apply an object to expose the endpoint, for example: a route.
enum:
- hostname
type: string
required:
- type
type: object
https:
description: https represents the configuration for https
endpoint.
properties:
hostname:
description: hostname points to a fixed hostname for serving
agents' handshakes.
properties:
caBundle:
description: caBundle of the endpoint.
format: byte
type: string
host:
description: host is the host name of the endpoint.
type: string
required:
- host
type: object
type:
default: hostname
description: |-
type specifies how the endpoint is exposed.
You may need to apply an object to expose the endpoint, for example: a route.
enum:
- hostname
type: string
required:
- type
type: object
protocol:
default: grpc
description: protocol is the protocol used for the endpoint,
could be https or grpc.
enum:
- grpc
- https
type: string
usage:
description: |-
usage defines the usage of the endpoint. It could be "agentToHub" indicating the endpoint is used
for communication between agent and hub, or "consumer" indicating the endpoint is used for external consumer.
type: string
required:
- protocol
type: object
required:
- grpcEndpoint
type: object
proxyConfig:
description: |-
ProxyConfig holds the configuration for enabling klusterlet-proxy functionality,
which allows the hub cluster to access the managed cluster's API server through
a gRPC-based proxy tunnel established by the klusterlet agent.
When configured, the klusterlet agent establishes a gRPC connection to the hub's
proxy server and proxies incoming HTTP requests to the local managed cluster API server.
This enables hub-to-spoke API access even when the managed cluster is not directly
accessible from the hub (e.g., behind a firewall or NAT).
This feature requires the ClusterProxy feature gate to be enabled and corresponding
GRPCConfiguration to be set in the ClusterManager on the hub side.
properties:
authentications:
default:
- userToken
description: |-
Authentications defines how the agent authenticates with the cluster.
If not specified, defaults to ["userToken"].
items:
enum:
- userToken
- impersonation
type: string
type: array
x-kubernetes-list-type: set
minItems: 1
maxItems: 2
grpcEndpoint:
description: GRPCEndpoint represents the gRPC endpoint configuration
for the proxy connection.
properties:
grpc:
description: grpc represents the configuration for grpc endpoint.
properties:
hostname:
description: hostname points to a fixed hostname for serving
agents' handshakes.
properties:
caBundle:
description: caBundle of the endpoint.
format: byte
type: string
host:
description: host is the host name of the endpoint.
type: string
maxLength: 253
pattern: ^(([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$
required:
- host
type: object
type:
default: hostname
description: |-
type specifies how the endpoint is exposed.
You may need to apply an object to expose the endpoint, for example: a route.
enum:
- hostname
type: string
required:
- type
type: object
https:
description: https represents the configuration for https
endpoint.
properties:
hostname:
description: hostname points to a fixed hostname for serving
agents' handshakes.
properties:
caBundle:
description: caBundle of the endpoint.
format: byte
type: string
host:
description: host is the host name of the endpoint.
type: string
maxLength: 253
pattern: ^(([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\-]*[A-Za-z0-9])$
required:
- host
type: object
type:
default: hostname
description: |-
type specifies how the endpoint is exposed.
You may need to apply an object to expose the endpoint, for example: a route.
enum:
- hostname
type: string
required:
- type
type: object
protocol:
default: grpc
description: protocol is the protocol used for the endpoint,
could be https or grpc.
enum:
- grpc
- https
type: string
usage:
description: |-
usage defines the usage of the endpoint. It could be "agentToHub" indicating the endpoint is used
for communication between agent and hub, or "consumer" indicating the endpoint is used for external consumer.
type: string
enum:
- agentToHub
- consumer
default: agentToHub
required:
- protocol
type: object
x-kubernetes-validations:
- rule: "self.protocol == 'grpc' ? has(self.grpc) : has(self.https)"
message: "Provide grpc when protocol=grpc, and https when protocol=https."
- rule: "!(has(self.grpc) && has(self.https))"
message: "Specify only one of grpc or https."
required:
- grpcEndpoint
type: object

registrationConfiguration:
description: RegistrationConfiguration contains the configuration
of registration
Expand Down
36 changes: 36 additions & 0 deletions operator/v1/types_klusterlet.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,20 @@ type KlusterletSpec struct {
// is not available on the managed cluster.
// +optional
PriorityClassName string `json:"priorityClassName,omitempty"`

// ProxyConfig holds the configuration for enabling klusterlet-proxy functionality,
// which allows the hub cluster to access the managed cluster's API server through
// a gRPC-based proxy tunnel established by the klusterlet agent.
//
// When configured, the klusterlet agent establishes a gRPC connection to the hub's
// proxy server and proxies incoming HTTP requests to the local managed cluster API server.
// This enables hub-to-spoke API access even when the managed cluster is not directly
// accessible from the hub (e.g., behind a firewall or NAT).
//
// This feature requires the ClusterProxy feature gate to be enabled and corresponding
// GRPCConfiguration to be set in the ClusterManager on the hub side.
// +optional
ProxyConfig *ProxyConfig `json:"proxyConfig,omitempty"`
}

// ServerURL represents the apiserver url and ca bundle that is accessible externally
Expand Down Expand Up @@ -331,6 +345,28 @@ const (
ClusterAnnotationsKeyPrefix = "agent.open-cluster-management.io"
)

// ProxyConfig holds the configuration of klusterlet-proxy.
type ProxyConfig struct {
// GRPCEndpoint represents the gRPC endpoint configuration for the proxy connection.
// +required
// +kubebuilder:validation:Required
GRPCEndpoint *EndpointExposure `json:"grpcEndpoint"`

// Authentications defines how the agent authenticates with the cluster.
// If not specified, defaults to ["userToken"].
// +optional
// +kubebuilder:default={userToken}
Authentications []ProxyAuthenticationType `json:"authentications,omitempty"`
}

// +kubebuilder:validation:Enum=userToken;impersonation
type ProxyAuthenticationType string

const (
ProxyAuthenticationUserToken ProxyAuthenticationType = "userToken"
ProxyAuthenticationImpersonation ProxyAuthenticationType = "impersonation"
)

// KlusterletDeployOption describes the deployment options for klusterlet
type KlusterletDeployOption struct {
// Mode can be Default, Hosted, Singleton or SingletonHosted. It is Default mode if not specified
Expand Down
31 changes: 31 additions & 0 deletions operator/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.