feat: multi cluster support #240

vertex451 · 2025-05-30T15:37:51Z

Resolves #149

Changes

Gateway

Refactored gateway/manager and took out its parts into separate packages:
At some point I realized that we have map for GraphQL schemas and map for resolvers/runtime-clients and they are coupled due to relation to a single cluster.
And I migrated to a domain driven design - introduced targetcluster package that represents a single k8s cluster. So now, instead of having several maps, we have one map of clusters, where each single cluster holds its related schema, resolvers and runtime clients.

Listener

Moved to reconciler-driven code design. Now we have 3 reconcilers - KCP, ClusterAccess and Standard.
Now it easy to follow the flow of the listnener app.
Reworked package structure - packages that are used by one reconciler I moved to that reconciler, shared packages moved to listener/pkg dir.
Used golang-commons lifecycle package for Reconcilers.

Testing

Tested 01.07 by checking all three paths working:

local-setup
multicluster
single cluster

Tested local setup after removed singlenode mode

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

…into feat/mutli-cluster

gateway/manager/targetcluster/connection.go

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

aaronschweig

Left a few comments in the PR.

In general I think the PR is already in quite a good state. What I can see though is a diversion from the usual pattern in which we write operators.

A few things to note:

to adhere to best practices the usage of our lifecycle package in the golang-commons library is encouraged
Artem: Done
Keep the Reconcilers straight. I am not sold completely on the introduction of the strategies pattern. It feels a bit too much, for something that is determined once on startup. I could understand the introduction of such a pattern a bit more, if we would have on reconciler potentially working on multiple different patterns/inputs. But since we have a KCP reconciler, which is dedicated to performing specific KCP tasks, a ClusterAccess resource reconciler which is able to reconcile ClusterAccesses and non of them are managed by a single reconciler (which would not be ideal anyway) the strategies feel a bit like an overengineered solution for a simple configuration switch in the startup period of the operator
Artem: Replaced strategy patter with conditional approach
I do think it is a good idea, to abstract a cluster layer, that handles a bit the logic of getting a client and the necessary information to gateway to a cluster
I didn't look into the tests yet, since you mentioned there were still WIP
A small wish from mine: Don't code it like Java. Whilst I sometimes see the benefits of introducing design patterns, I want to avoid to have too many indirections or abstractions where not absolutely necessary. Within the framework of controller-runtime and the combination with the golang-commons library there already are a lot of useful abstractions that help us build mantainable operators, so maybe looking into them, also the pattern of subroutines (term used to encapsulate a specific portion of reconcilation logic) could help when trying to find the right way to abstract/implement the logic needed.
Artem: Appreciate your feedback, will keep in mind the idea of following the idiomatic Golang way

aaronschweig · 2025-06-12T11:37:13Z

gateway/manager/targetcluster/connection.go

+	tmpFile, err := os.CreateTemp("", "kubeconfig-*.yaml")
+	if err != nil {
+		return nil, fmt.Errorf("failed to create temporary kubeconfig file: %w", err)
+	}
+	defer os.Remove(tmpFile.Name())
+
+	if _, err := tmpFile.Write(kubeconfigData); err != nil {
+		tmpFile.Close()
+		return nil, fmt.Errorf("failed to write kubeconfig to temporary file: %w", err)
+	}
+	tmpFile.Close()
+
+	// Build config from kubeconfig
+	config, err := clientcmd.BuildConfigFromFlags("", tmpFile.Name())
+	if err != nil {
+		return nil, fmt.Errorf("failed to build config from kubeconfig: %w", err)


there should be another option to parse the necessary information from the auth metadata. If it is structured, there probably is the correct struct already hidden somewhere, without the need to use a temporary file

Removed temporary file usage.

aaronschweig · 2025-06-12T11:38:19Z

gateway/manager/targetcluster/cluster.go

+	if tc.metadata != nil && tc.metadata.Path != "" {
+		path = tc.metadata.Path
+	}
+	return fmt.Sprintf("http://localhost:%s/%s/graphql", appCfg.Gateway.Port, path)


this is only valid for locally running services. If this GetEndpoint is needed for something else than logging, this is not the correct approach

now it logs new ednpoint with localhost only in case of local_development=true.
Otherwise it logs /workspace/graphql

gateway/manager/targetcluster/connection.go

aaronschweig · 2025-06-12T11:39:28Z

gateway/manager/targetcluster/connection.go

+	}
+
+	if metadata.Auth.Kubeconfig == "" {
+		return nil, fmt.Errorf("kubeconfig data is empty")


no interpolation made, so errors.New is sufficient

I removed that file completely during simplification. And I am very happy about it

aaronschweig · 2025-06-12T11:40:44Z

gateway/manager/targetcluster/metadata.go

+// ClusterMetadata represents cluster connection information embedded in schema files
+// Simplified for standard Kubernetes clusters with kubeconfig authentication
+type ClusterMetadata struct {
+	Host string        `json:"host"`
+	Path string        `json:"path"`
+	Auth *AuthMetadata `json:"auth,omitempty"`
+}
+
+// AuthMetadata contains kubeconfig authentication for standard Kubernetes clusters
+type AuthMetadata struct {
+	Type       string `json:"type"`       // Only "kubeconfig" is supported
+	Kubeconfig string `json:"kubeconfig"` // base64 encoded kubeconfig
+}
+
+// FileData represents the data extracted from a schema file
+type FileData struct {
+	Definitions map[string]interface{} `json:"definitions"`
+	Metadata    *ClusterMetadata       `json:"x-cluster-metadata"`
+}


not the biggest fan of type only files - it makes most sense to colocate them to their usage if possible

This is also removed

listener/reconciler/clusteraccess/strategy.go

aaronschweig · 2025-06-12T11:44:20Z

listener/reconciler/types.go

Same as above, I don't like the separate type files - please colocate to their usage if possible

listener/kcp/reconciler_factory.go

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

…into feat/mutli-cluster

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

n3rdc4ptn · 2025-07-02T08:36:27Z

Created new issue with this configuration after discussion with Artem.
#271

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

…into feat/mutli-cluster

* fix: auth issues --------- Signed-off-by: Artem Shcherbatiuk <[email protected]>

…aphql-gateway into feat/mutli-cluster

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

n3rdc4ptn · 2025-07-10T07:17:04Z

Great work, already works like a charm for me testing it out.
Two things which came up during testing for me:

HostRef

How hard would it be, to also make the host parameter not only specifiable through the normal spec, but also through a reference from a ConfigMap or Secret?

hostRef:
  configMapRef:

  secretRef:

or similar. Open for improvement here regarding the spec.

Not a deal breaker for us, just a nice to have. Also I not thought about this in my initial architecture proposal.

Status

I saw that there is a condition parameter in the ClusterAccessStatus Type but I not saw anything getting reported back after the resource got reconciled.
It would be helpful if there could be some kind of status parameter, which reports if the definitions file was generated successfully by the listener.
Maybe like ready: true to indicate, that the new endpoint is now available by the gateway.

@aaronschweig @nexus49 @vertex451

aaronschweig · 2025-07-10T07:39:37Z

@n3rdc4ptn the status could be a quick win, since this is normally supported out of the box by our lifecycle, the other one, I would argue, can be done in a separate PR.

For the status, we just need to enable WithConditionManagement() and then add conditions to the ClusterAccessStatus @vertex451

n3rdc4ptn · 2025-07-10T07:51:50Z

@n3rdc4ptn the status could be a quick win, since this is normally supported out of the box by our lifecycle, the other one, I would argue, can be done in a separate PR.

sounds great :) Thanks for the quick response. Should I create an issue for it? @aaronschweig

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

vertex451 · 2025-07-10T11:15:38Z

Added condition management.

P.S. The only thing - it just confirms that spec file is generated, but doesn't guarantee that endpoint is available - for this Gateway must parse the spec file and and create the endpoint.

n3rdc4ptn · 2025-07-10T11:15:52Z

@n3rdc4ptn the status could be a quick win, since this is normally supported out of the box by our lifecycle, the other one, I would argue, can be done in a separate PR.

sounds great :) Thanks for the quick response. Should I create an issue for it? @aaronschweig

added issue: #276

n3rdc4ptn · 2025-07-10T11:20:38Z

P.S. The only thing - it just confirms that spec file is generated, but doesn't guarantee that endpoint is available - for this Gateway must parse the spec file and and create the endpoint.

From our side, this is enough for now. We are optimistic, that the Gateway will do its job.
@aaronschweig what do you think? Required for this to be complete that the Gateway will also report status otherwise this could be a Follow Up

feat: multi cluster support for the listener

5bfcce5

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

github-actions bot added the feature label May 30, 2025

vertex451 added 5 commits May 30, 2025 17:39

cleanup

dbade00

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

merged main

5ef479d

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

feat: gateway

208d253

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

listener and gateway

679681f

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

Merge branch 'main' of github.com:openmfp/kubernetes-graphql-gateway …

65a4195

…into feat/mutli-cluster

github-advanced-security bot found potential problems Jun 9, 2025

View reviewed changes

gateway/manager/targetcluster/connection.go Fixed Show fixed Hide fixed

vertex451 added 19 commits June 9, 2025 19:38

fix config storing

04d95c2

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

fixed certificate issue

cbe76e4

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

returned kcp direct approach to the listener

95cbd65

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

testing

757ffa0

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

script

afdef55

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

cleanup

a766071

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

cleanup

77f8e9e

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

generate

d6268cb

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

genereted fresh ClusterAccess

27e566c

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

cleanup

bc874c1

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

strategy pattern in reconciling

05f2b10

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

strategy pattern in reconciling

eb67fcd

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

fix conenction target

6765e2e

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

moved handler to router package to cluster

8caffa0

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

merged clusterRegistry and clusterRouter

b8c411b

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

removed redundant code

5147ca1

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

cleanup

df79f7f

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

simplify kcp part

4c53899

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

fixed unit tests

8506a97

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

aaronschweig requested changes Jun 12, 2025

View reviewed changes

vertex451 added 2 commits June 18, 2025 14:09

feat: use golang-commons lifecycle

5d8fc12

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

Merge branch 'main' of github.com:openmfp/kubernetes-graphql-gateway …

7c43577

…into feat/mutli-cluster

vertex451 added 3 commits July 2, 2025 10:24

returned default gateway port

4218cd5

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

replace interface with any

7af15ee

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

reduced nesting

e790cbd

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

n3rdc4ptn mentioned this pull request Jul 2, 2025

Bug: GATEWAY_SHOULDIMPERSONATE & INTROSPECTION_AUTHENTICATION seems to be ignored in multi-cluster setup #271

Closed

vertex451 added 10 commits July 2, 2025 10:43

removed buildkubernetesconfig func

32eed65

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

fix: tests

78641d3

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

tests: listener/reconciler/kcp

fa75b07

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

added subroutines.go file to testignore

e69ea05

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

linter

c368c4f

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

correct name for testignore

865c82c

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

Merge branch 'main' of github.com:openmfp/kubernetes-graphql-gateway …

03033a4

…into feat/mutli-cluster

fix: auth issues (#274)

e12f68b

* fix: auth issues --------- Signed-off-by: Artem Shcherbatiuk <[email protected]>

Merge branch 'feat/mutli-cluster' of github.com:openmfp/kubernetes-gr…

96be88b

…aphql-gateway into feat/mutli-cluster

removed listener/reconciler/kcp from testignore

e5937b4

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

vertex451 changed the title ~~feat: multi cluster support for the listener~~ feat: multi cluster support Jul 9, 2025

vertex451 added 2 commits July 9, 2025 13:51

remove single cluster

369e747

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

removed singlenode mode

653de3d

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

vertex451 added 2 commits July 10, 2025 10:17

moved rest Config builder to common package

7867374

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

feat: condition management

b02ad95

On-behalf-of: @SAP [email protected] Signed-off-by: Artem Shcherbatiuk <[email protected]>

aaronschweig approved these changes Jul 10, 2025

View reviewed changes

vertex451 merged commit 089a1a3 into main Jul 10, 2025
11 of 12 checks passed

vertex451 deleted the feat/mutli-cluster branch July 10, 2025 11:26

vertex451 added this to the 2025.Q3 milestone Jul 10, 2025

feat: multi cluster support #240

feat: multi cluster support #240

Uh oh!

Conversation

vertex451 commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Gateway

Listener

Testing

Uh oh!

Uh oh!

aaronschweig left a comment • edited by vertex451 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vertex451 Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vertex451 Jun 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

n3rdc4ptn commented Jul 2, 2025

Uh oh!

n3rdc4ptn commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

HostRef

Status

Uh oh!

aaronschweig commented Jul 10, 2025

Uh oh!

n3rdc4ptn commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vertex451 commented Jul 10, 2025

Uh oh!

n3rdc4ptn commented Jul 10, 2025

Uh oh!

n3rdc4ptn commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vertex451 commented May 30, 2025 •

edited

Loading

aaronschweig left a comment •

edited by vertex451

Loading

vertex451 Jun 20, 2025 •

edited

Loading

vertex451 Jun 20, 2025 •

edited

Loading

n3rdc4ptn commented Jul 10, 2025 •

edited

Loading

n3rdc4ptn commented Jul 10, 2025 •

edited

Loading

n3rdc4ptn commented Jul 10, 2025 •

edited

Loading