Skip to content

feat: added multi-cluster runtime #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Oct 10, 2025
Merged

feat: added multi-cluster runtime #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
6f6ca05
feat: added multi-cluster runtime
OlegErshov Oct 1, 2025
b6fa6db
chore: removed commented code
OlegErshov Oct 1, 2025
4252595
chore: refactored imports
OlegErshov Oct 2, 2025
d39792c
feat: used mccontext and added condition manager
OlegErshov Oct 2, 2025
b8ecc8d
chore: removed completed todo
OlegErshov Oct 2, 2025
a791d78
feat: added initilizer's clean up using initializing provider, remove…
OlegErshov Oct 3, 2025
8bb6eff
feat: made initializer name configurable
OlegErshov Oct 3, 2025
0d464da
Merge branch 'main' into feat/multi-cluster-runtime
OlegErshov Oct 3, 2025
eeee7ce
fix: fixed merge errors
OlegErshov Oct 3, 2025
20edb47
chore: showcase mcruntime integration
aaronschweig Oct 7, 2025
a9436cd
fix: startup of manager
aaronschweig Oct 7, 2025
825f8cb
fix.
aaronschweig Oct 7, 2025
8ef79da
chore: updated model generated controller
OlegErshov Oct 7, 2025
1006d1e
Update internal/subroutine/remove_initializer.go
aaronschweig Oct 7, 2025
88c5326
feat: used multi-cluster runtime in operator contoller
OlegErshov Oct 8, 2025
32b9602
updated go version
OlegErshov Oct 8, 2025
790482c
updated go version in docker file
OlegErshov Oct 8, 2025
e1e48c3
feat: removed specific lc clients
OlegErshov Oct 9, 2025
37e4cd3
feat: updated tests
OlegErshov Oct 9, 2025
51ba5a7
chore: format fix
OlegErshov Oct 9, 2025
7b58513
Merge branch 'main' into feat/multi-cluster-runtime
OlegErshov Oct 9, 2025
b6d3bfa
fix: fixed merge errors
OlegErshov Oct 9, 2025
da0c6e5
improve test coverage
OlegErshov Oct 9, 2025
51299d3
improve tracehold for remove initializer subroutine
OlegErshov Oct 9, 2025
98124c4
fix: fixed errors related to pointing wrong logical cluster
OlegErshov Oct 10, 2025
26f5b3c
fix: fixed tests
OlegErshov Oct 10, 2025
c719876
fix: removed wrong file
OlegErshov Oct 10, 2025
24128f8
feat: support additional redirectURLs for the clients (#99)
aaronschweig Oct 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .mockery.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,17 @@ packages:
outpkg: mocks
interfaces:
Client:

sigs.k8s.io/multicluster-runtime/pkg/manager:
config:
dir: internal/subroutine/mocks
outpkg: mocks
interfaces:
Manager:

sigs.k8s.io/controller-runtime/pkg/cluster:
config:
dir: internal/subroutine/mocks
outpkg: mocks
interfaces:
Cluster:
2 changes: 1 addition & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build the manager binary
FROM golang:1.24.6-bullseye AS builder
FROM golang:1.25.2-bookworm AS builder
ARG TARGETOS
ARG TARGETARCH

Expand Down
34 changes: 30 additions & 4 deletions cmd/initializer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,24 +6,29 @@ import (

helmv2 "github.com/fluxcd/helm-controller/api/v2"
sourcev1 "github.com/fluxcd/source-controller/api/v1"

"github.com/kcp-dev/logicalcluster/v3"
"github.com/kcp-dev/multicluster-provider/initializingworkspaces"
pmcontext "github.com/platform-mesh/golang-commons/context"
"github.com/spf13/cobra"
"k8s.io/apimachinery/pkg/runtime"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
"k8s.io/client-go/rest"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/kcp"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"

"github.com/platform-mesh/security-operator/internal/controller"
mcmanager "sigs.k8s.io/multicluster-runtime/pkg/manager"
)

var initializerCmd = &cobra.Command{
Use: "initializer",
Short: "FGA initializer for the organization workspacetype",
RunE: func(cmd *cobra.Command, args []string) error {
ctx, _, shutdown := pmcontext.StartContext(log, appCfg, defaultCfg.ShutdownTimeout)
defer shutdown()

mgrCfg := ctrl.GetConfigOrDie()

Expand All @@ -50,7 +55,17 @@ var initializerCmd = &cobra.Command{
}
mgrOpts.LeaderElectionConfig = inClusterCfg
}
mgr, err := kcp.NewClusterAwareManager(mgrCfg, mgrOpts)

provider, err := initializingworkspaces.New(mgrCfg, initializingworkspaces.Options{
InitializerName: appCfg.InitializerName,
Scheme: mgrOpts.Scheme,
})
if err != nil {
log.Error().Err(err).Msg("unable to construct cluster provider")
os.Exit(1)
Comment on lines +64 to +65
Copy link

@vertex451 vertex451 Oct 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Lets replace it witth log.Fatal() which does basically the same(plus some additional flushing), but it will have more severe log level and is more relevant to this situation.

}

mgr, err := mcmanager.New(mgrCfg, provider, mgrOpts)
if err != nil {
setupLog.Error(err, "Failed to create manager")
os.Exit(1)
Expand All @@ -60,7 +75,7 @@ var initializerCmd = &cobra.Command{
utilruntime.Must(sourcev1.AddToScheme(runtimeScheme))
utilruntime.Must(helmv2.AddToScheme(runtimeScheme))

orgClient, err := logicalClusterClientFromKey(mgr, log)(logicalcluster.Name("root:orgs"))
orgClient, err := logicalClusterClientFromKey(mgr.GetLocalManager(), log)(logicalcluster.Name("root:orgs"))
if err != nil {
setupLog.Error(err, "Failed to create org client")
os.Exit(1)
Expand All @@ -78,7 +93,12 @@ var initializerCmd = &cobra.Command{
os.Exit(1)
}

if err := controller.NewLogicalClusterReconciler(log, mgrCfg, mgr.GetClient(), orgClient, appCfg, inClusterClient).SetupWithManager(mgr, defaultCfg, log); err != nil {
if appCfg.IDP.AdditionalRedirectURLs == nil {
appCfg.IDP.AdditionalRedirectURLs = []string{}
}

if err := controller.NewLogicalClusterReconciler(log, orgClient, appCfg, inClusterClient, mgr).
SetupWithManager(mgr, defaultCfg); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "LogicalCluster")
os.Exit(1)
}
Expand All @@ -92,6 +112,12 @@ var initializerCmd = &cobra.Command{
os.Exit(1)
}

go func() {
if err := provider.Run(ctx, mgr); err != nil {
log.Fatal().Err(err).Msg("unable to run provider")
}
}()

setupLog.Info("starting manager")

return mgr.Start(ctrl.SetupSignalHandler())
Expand Down
37 changes: 29 additions & 8 deletions cmd/model_generator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@ package cmd
import (
"context"
"crypto/tls"
"fmt"

"github.com/kcp-dev/multicluster-provider/apiexport"
platformeshcontext "github.com/platform-mesh/golang-commons/context"
appsv1 "k8s.io/api/apps/v1"

Expand All @@ -13,9 +15,9 @@ import (
"k8s.io/client-go/rest"
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/kcp"
"sigs.k8s.io/controller-runtime/pkg/manager"
"sigs.k8s.io/controller-runtime/pkg/metrics/server"
mcmanager "sigs.k8s.io/multicluster-runtime/pkg/manager"

securityv1alpha1 "github.com/platform-mesh/security-operator/api/v1alpha1"
"github.com/platform-mesh/security-operator/internal/controller"
Expand Down Expand Up @@ -56,18 +58,31 @@ var modelGeneratorCmd = &cobra.Command{
}
mgrOpts.LeaderElectionConfig = inClusterCfg
}
runtimeScheme := runtime.NewScheme()
utilruntime.Must(appsv1.AddToScheme(runtimeScheme))
utilruntime.Must(securityv1alpha1.AddToScheme(runtimeScheme))

mgr, err := kcp.NewClusterAwareManager(cfg, mgrOpts)
if mgrOpts.Scheme == nil {
log.Error().Err(fmt.Errorf("scheme should not be nil")).Msg("scheme should not be nil")
return fmt.Errorf("scheme should not be nil")
}
Comment on lines +61 to +68
Copy link

@coderabbitai coderabbitai bot Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Redundant scheme initialization and check.

A new runtimeScheme is created and populated at lines 61-63, but it's never assigned to mgrOpts.Scheme. Line 38 already sets mgrOpts.Scheme = scheme (presumably a package-level variable). The nil-check at lines 65-68 validates mgrOpts.Scheme, not runtimeScheme.

Apply this diff to fix the scheme initialization:

-		runtimeScheme := runtime.NewScheme()
-		utilruntime.Must(appsv1.AddToScheme(runtimeScheme))
-		utilruntime.Must(securityv1alpha1.AddToScheme(runtimeScheme))
-
 		if mgrOpts.Scheme == nil {
 			log.Error().Err(fmt.Errorf("scheme should not be nil")).Msg("scheme should not be nil")
 			return fmt.Errorf("scheme should not be nil")
 		}

Or, if you intended to use the new scheme:

 		runtimeScheme := runtime.NewScheme()
 		utilruntime.Must(appsv1.AddToScheme(runtimeScheme))
 		utilruntime.Must(securityv1alpha1.AddToScheme(runtimeScheme))
+		mgrOpts.Scheme = runtimeScheme

 		if mgrOpts.Scheme == nil {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
runtimeScheme := runtime.NewScheme()
utilruntime.Must(appsv1.AddToScheme(runtimeScheme))
utilruntime.Must(securityv1alpha1.AddToScheme(runtimeScheme))
mgr, err := kcp.NewClusterAwareManager(cfg, mgrOpts)
if mgrOpts.Scheme == nil {
log.Error().Err(fmt.Errorf("scheme should not be nil")).Msg("scheme should not be nil")
return fmt.Errorf("scheme should not be nil")
}
if mgrOpts.Scheme == nil {
log.Error().Err(fmt.Errorf("scheme should not be nil")).Msg("scheme should not be nil")
return fmt.Errorf("scheme should not be nil")
}
🤖 Prompt for AI Agents 
In cmd/model_generator.go around lines 61-68, the code creates and populates a
new runtimeScheme but never assigns it to mgrOpts.Scheme while the nil-check
validates mgrOpts.Scheme; either remove the redundant runtimeScheme creation
(lines 61-63) and keep the existing mgrOpts.Scheme set earlier, or if you
intended to use the new scheme, assign runtimeScheme to mgrOpts.Scheme before
the nil-check and then add the API registrations to that scheme so the check and
registrations operate on the same object.


provider, err := apiexport.New(cfg, apiexport.Options{
Scheme: mgrOpts.Scheme,
})
if err != nil {
setupLog.Error(err, "unable to setup manager")
log.Error().Err(err).Msg("Failed to create apiexport provider")
return err
}

runtimeScheme := runtime.NewScheme()
utilruntime.Must(appsv1.AddToScheme(runtimeScheme))
utilruntime.Must(securityv1alpha1.AddToScheme(runtimeScheme))
mgr, err := mcmanager.New(cfg, provider, mgrOpts)
if err != nil {
log.Error().Err(err).Msg("Failed to create manager")
return err
}

if err := controller.NewAPIBindingReconciler(mgr.GetClient(), log, logicalClusterClientFromKey(mgr, log)).
SetupWithManager(mgr, log, defaultCfg); err != nil {
if err := controller.NewAPIBindingReconciler(log, mgr).
SetupWithManager(mgr, defaultCfg); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "Resource")
return err
}
Expand All @@ -81,6 +96,12 @@ var modelGeneratorCmd = &cobra.Command{
return err
}

go func() {
if err := provider.Run(ctx, mgr); err != nil {
log.Fatal().Err(err).Msg("unable to run provider")
}
}()

setupLog.Info("starting manager")
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
setupLog.Error(err, "problem running manager")
Expand Down
42 changes: 33 additions & 9 deletions cmd/operator.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import (
apisv1alpha1 "github.com/kcp-dev/kcp/sdk/apis/apis/v1alpha1"
kcpcorev1alpha1 "github.com/kcp-dev/kcp/sdk/apis/core/v1alpha1"
"github.com/kcp-dev/logicalcluster/v3"
"github.com/kcp-dev/multicluster-provider/apiexport"
accountsv1alpha1 "github.com/platform-mesh/account-operator/api/v1alpha1"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
Expand All @@ -23,8 +24,9 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/kcp"

metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
mcmanager "sigs.k8s.io/multicluster-runtime/pkg/manager"

openfgav1 "github.com/openfga/api/proto/openfga/v1"
platformeshcontext "github.com/platform-mesh/golang-commons/context"
Expand All @@ -35,15 +37,16 @@ import (
kcptenancyv1alphav1 "github.com/kcp-dev/kcp/sdk/apis/tenancy/v1alpha1"
corev1alpha1 "github.com/platform-mesh/security-operator/api/v1alpha1"
"github.com/platform-mesh/security-operator/internal/controller"
"github.com/platform-mesh/security-operator/internal/subroutine"
// +kubebuilder:scaffold:imports
)

var (
scheme = runtime.NewScheme()
)

func logicalClusterClientFromKey(mgr ctrl.Manager, log *logger.Logger) subroutine.NewLogicalClusterClientFunc {
type NewLogicalClusterClientFunc func(clusterKey logicalcluster.Name) (client.Client, error)

func logicalClusterClientFromKey(mgr ctrl.Manager, log *logger.Logger) NewLogicalClusterClientFunc {
return func(clusterKey logicalcluster.Name) (client.Client, error) {
cfg := rest.CopyConfig(mgr.GetConfig())

Expand All @@ -55,6 +58,8 @@ func logicalClusterClientFromKey(mgr ctrl.Manager, log *logger.Logger) subroutin

parsed.Path = fmt.Sprintf("/clusters/%s", clusterKey)

log.Info().Msg(fmt.Sprintf("HOST from logical cluster client from key -- %s", parsed.String()))

cfg.Host = parsed.String()

return client.New(cfg, client.Options{
Expand Down Expand Up @@ -110,9 +115,22 @@ var operatorCmd = &cobra.Command{
mgrOpts.LeaderElectionConfig = inClusterCfg
}

mgr, err := kcp.NewClusterAwareManager(cfg, mgrOpts)
if mgrOpts.Scheme == nil {
log.Error().Err(fmt.Errorf("scheme should not be nil")).Msg("scheme should not be nil")
return fmt.Errorf("scheme should not be nil")
}

provider, err := apiexport.New(cfg, apiexport.Options{
Scheme: mgrOpts.Scheme,
})
if err != nil {
setupLog.Error(err, "unable to construct cluster provider")
return err
}

mgr, err := mcmanager.New(cfg, provider, mgrOpts)
if err != nil {
log.Error().Err(err).Msg("unable to start manager")
setupLog.Error(err, "Failed to create manager")
return err
}

Expand All @@ -124,14 +142,14 @@ var operatorCmd = &cobra.Command{

fga := openfgav1.NewOpenFGAServiceClient(conn)

if err = controller.NewStoreReconciler(log, mgr.GetClient(), fga, logicalClusterClientFromKey(mgr, log)).
SetupWithManager(mgr, defaultCfg, log); err != nil {
if err = controller.NewStoreReconciler(log, fga, mgr).
SetupWithManager(mgr, defaultCfg); err != nil {
log.Error().Err(err).Str("controller", "store").Msg("unable to create controller")
return err
}
if err = controller.
NewAuthorizationModelReconciler(log, mgr.GetClient(), fga, logicalClusterClientFromKey(mgr, log)).
SetupWithManager(mgr, defaultCfg, log); err != nil {
NewAuthorizationModelReconciler(log, fga, mgr).
SetupWithManager(mgr, defaultCfg); err != nil {
log.Error().Err(err).Str("controller", "authorizationmodel").Msg("unable to create controller")
return err
}
Expand All @@ -146,6 +164,12 @@ var operatorCmd = &cobra.Command{
return err
}

go func() {
if err := provider.Run(ctx, mgr); err != nil {
log.Fatal().Err(err).Msg("unable to run provider")
}
}()
Comment on lines +167 to +171
Copy link

@coderabbitai coderabbitai bot Oct 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid fatal exit when the provider stops.

Line 169 calls log.Fatal, which terminates the process. provider.Run returns context.Canceled on graceful shutdown, so this turns every normal exit into a fatal crash with a non-zero status. Switch to a non-fatal log (and ignore context.Canceled) so shutdown remains clean.

-	go func() {
-		if err := provider.Run(ctx, mgr); err != nil {
-			log.Fatal().Err(err).Msg("unable to run provider")
-		}
-	}()
+	go func() {
+		if err := provider.Run(ctx, mgr); err != nil && !errors.Is(err, context.Canceled) {
+			log.Error().Err(err).Msg("provider exited with error")
+		}
+	}()

(Remember to import "errors".)

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
go func() {
if err := provider.Run(ctx, mgr); err != nil {
log.Fatal().Err(err).Msg("unable to run provider")
}
}()
go func() {
if err := provider.Run(ctx, mgr); err != nil && !errors.Is(err, context.Canceled) {
log.Error().Err(err).Msg("provider exited with error")
}
}()
🤖 Prompt for AI Agents 
In cmd/operator.go around lines 167 to 171, replace the log.Fatal call inside
the goroutine so that when provider.Run returns nil or context.Canceled the
process does not exit with a fatal status: check the error returned, if
errors.Is(err, context.Canceled) (import "errors" and keep context imported) or
err == nil then log an Info/Debug message and return; otherwise log an Error
(non-fatal) with Err(err). Ensure the goroutine does not call log.Fatal so
graceful shutdowns are treated as normal exits.


setupLog.Info("starting manager")
if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
log.Error().Err(err).Msg("problem running manager")
Expand Down
Loading