[Multi_K8s-Plugin] Canary Rollout

[mohammedfirdouss]

What this PR does: Implements the K8S_CANARY_ROLLOUT stage for the kubernetes_multicluster plugin. When this stage runs it deploys the new version as a small canary replica set (configurable by count or percentage) to all targeted clusters in parallel, while leaving the primary workload untouched.

Why we need it: The multicluster plugin previously only supported K8S_MULTI_SYNC (all-or-nothing rollout) and K8S_MULTI_ROLLBACK. There was no way to progressively test a new version across clusters before promoting it. K8S_CANARY_ROLLOUT fills that gap and is the first step toward a full canary/baseline/primary pipeline strategy for multi-cluster deployments.

Which issue(s) this PR fixes: Part of #6446

Fixes #

Does this PR introduce a user-facing change?:
Yes — users can now add a K8S_CANARY_ROLLOUT stage to their app.pipecd.yaml pipeline when using the kubernetes_multicluster plugin.

• How are users affected by this change: Users who do not use a pipeline are unaffected. Users who configure a pipeline can now add K8S_CANARY_ROLLOUT with replicas, suffix, createService, and patches options.

• Is this breaking change: No

• How to migrate (if breaking change): N/A

[mohammedfirdouss]

@khanhtc1202 I have a question, am I meant to create example yml files for the pipecd-config, service and deployment to test fully? or it depends on how the user wants to setup their application.

[Copilot]

Pull request overview

Implements a new K8S_CANARY_ROLLOUT stage for the kubernetes_multicluster plugin to deploy a canary variant across all targeted clusters in parallel.

Changes:

• Add canary rollout stage plumbing (stage constant + plugin dispatch) and stage options in config.
• Add manifest helpers to duplicate/generate variant resources (workloads/services/configmaps/secrets) and apply them as canary.
• Add testdata and unit tests for single-cluster, multi-cluster, and failure scenarios.

Reviewed changes

Copilot reviewed 11 out of 12 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
pkg/app/pipedv1/plugin/kubernetes_multicluster/provider/resource.go	Adds Service kind constant for resource classification.
pkg/app/pipedv1/plugin/kubernetes_multicluster/provider/manifest.go	Adds manifest utilities (DeepCopyWithName, NestedString, IsService).
pkg/app/pipedv1/plugin/kubernetes_multicluster/go.mod	Bumps plugin dependencies (pipecd, SDK) and yaml library; adds new indirect deps.
pkg/app/pipedv1/plugin/kubernetes_multicluster/go.sum	Updates sums for bumped/added dependencies.
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/testdata/canary/deployment.yaml	Adds canary-related test manifest.
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/testdata/canary/app.pipecd.yaml	Adds test app config used by canary tests.
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/plugin.go	Dispatches the new K8S_CANARY_ROLLOUT stage to implementation.
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/pipeline.go	Registers the new stage constant and description.
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/misc.go	Adds helpers for generating/deleting variant resources (service/workload/config/secret).
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/canary_test.go	Adds tests for canary rollout execution paths.
pkg/app/pipedv1/plugin/kubernetes_multicluster/deployment/canary.go	Implements canary manifest generation, patching, and parallel rollout per target.
pkg/app/pipedv1/plugin/kubernetes_multicluster/config/application.go	Adds stage options structs and service reference; adds custom JSON unmarshal w/ defaults.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Warashi]

Sorry, I merged #6518, which caused a merge conflict with this PR.

I have a question, am I meant to create example yml files for the pipecd-config, service and deployment to test fully? or it depends on how the user wants to setup their application.

You need to create configs if you want to test on your local piped.
On the other hand, you don't need to do so if you think the test code is enough to check behavior.
It's better to check the behavior on your local, but it's not mandatory, I think. It's best to be able to check the expected behavior with the reproducible test code.

From another point of view, it would be better to demonstrate how to use this plugin by including example configs in the repository.

[mohammedfirdouss]

Sorry, I merged #6518, which caused a merge conflict with this PR.

I have a question, am I meant to create example yml files for the pipecd-config, service and deployment to test fully? or it depends on how the user wants to setup their application.

You need to create configs if you want to test on your local piped. On the other hand, you don't need to do so if you think the test code is enough to check behavior. It's better to check the behavior on your local, but it's not mandatory, I think. It's best to be able to check the expected behavior with the reproducible test code.

From another point of view, it would be better to demonstrate how to use this plugin by including example configs in the repository.

@Warashi Thanks for the feedback! I've added example configs mirroring the structure used in the single-cluster plugin. I also added tests for both paths — one that verifies the canary Service is created with the variant selector injected, and one that verifies no Service is created when createService is false. Will resolve the conflicts now.

[codecov]

Codecov Report

❌ Patch coverage is 42.78960% with 242 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.41%. Comparing base (25601c8) to head (95ea945).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...lugin/kubernetes_multicluster/deployment/canary.go	42.74%	132 Missing and 14 partials ⚠️
.../plugin/kubernetes_multicluster/deployment/misc.go	45.85%	74 Missing and 11 partials ⚠️
...lugin/kubernetes_multicluster/provider/manifest.go	0.00%	8 Missing ⚠️
...lugin/kubernetes_multicluster/deployment/plugin.go	0.00%	2 Missing ⚠️
...ugin/kubernetes_multicluster/config/application.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #6583       +/-   ##
===========================================
+ Coverage   28.79%   45.41%   +16.61%     
===========================================
  Files         575       86      -489     
  Lines       60858     8466    -52392     
===========================================
- Hits        17526     3845    -13681     
+ Misses      41997     4386    -37611     
+ Partials     1335      235     -1100     

Flag	Coverage Δ
.	?
.-pkg-app-pipedv1-plugin-analysis	32.43% <ø> (ø)
.-pkg-app-pipedv1-plugin-ecs	?
.-pkg-app-pipedv1-plugin-kubernetes	?
.-pkg-app-pipedv1-plugin-kubernetes_multicluster	63.48% <42.78%> (-3.82%)	⬇️
.-pkg-app-pipedv1-plugin-scriptrun	54.83% <ø> (ø)
.-pkg-app-pipedv1-plugin-terraform	?
.-pkg-app-pipedv1-plugin-wait	?
.-pkg-app-pipedv1-plugin-waitapproval	52.71% <ø> (ø)
.-pkg-plugin-sdk	50.34% <ø> (+0.19%)	⬆️
.-tool-actions-gh-release	19.23% <ø> (ø)
.-tool-actions-plan-preview	25.51% <ø> (ø)
.-tool-codegen-protoc-gen-auth	0.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mohammedfirdouss]

@Warashi I intend to create seperate individual branches for all deployment/development stages of this plugin. What best practice would you give here, eg. Should Canary have both rollout and clean stages in one branch or seperate branches etc.

[Warashi]

@Warashi I intend to create seperate individual branches for all deployment/development stages of this plugin. What best practice would you give here, eg. Should Canary have both rollout and clean stages in one branch or seperate branches etc.

Yes, I want each stages implementations to be separate PRs, i.e. separate branches.
Thank you for your confirmination.

[Warashi]

Thank you! LGTM
As I wrote above, please open another PR for Canary Clean stage.

[mohammedfirdouss]

@Warashi I intend to create seperate individual branches for all deployment/development stages of this plugin. What best practice would you give here, eg. Should Canary have both rollout and clean stages in one branch or seperate branches etc.

Yes, I want each stages implementations to be separate PRs, i.e. separate branches. Thank you for your confirmination.

Alright no problem, it is most likely going to cause a bit conflicts, but no issues.