Adding proposal for workload affinity and anti-affinity support #6685
Conversation
Signed-off-by: mszacillo <[email protected]>
karmada-bot
added
the
kind/feature
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @mszacillo, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a comprehensive proposal to enhance Karmada's scheduling capabilities by adding support for workload affinity and anti-affinity rules to the PropagationPolicy API. This feature aims to address advanced scheduling requirements, such as ensuring high availability for critical data pipelines by spreading duplicate workloads across different clusters (anti-affinity) and co-locating related smaller training jobs or interdependent services within the same cluster to minimize latency (affinity). The proposal outlines the necessary API extensions and architectural changes to the Karmada scheduler to implement these new rules.
Highlights
- API Extension for Workload Affinity/Anti-Affinity: Proposes new WorkloadAntiAffinity and WorkloadAffinity fields within the PropagationPolicy's Placement struct, along with WorkloadAffinityTerm and WeightedWorkloadAffinityTerm definitions, mirroring Kubernetes pod affinity concepts.
- Enhanced High Availability: Introduces anti-affinity rules to enable users to spread duplicate data processing pipelines across different clusters, ensuring continuous operation even if one cluster experiences issues.
- Optimized Workload Co-location: Provides affinity rules to allow users to schedule related workloads, such as distributed training jobs or interdependent services (e.g., Flink with Kafka, Ray with Redis), to the same cluster for improved performance and reduced network latency.
- Scheduler and Cache Modifications: Details required changes to ClusterInfo and ClusterSnapshot to track affinity metadata, and outlines modifications to the Karmada scheduler's internal cache and the addition of a new Workload Affinity Filter Plugin to enforce these rules.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
karmada-bot
added
the
size/L
There was a problem hiding this comment.
Code Review
This pull request introduces a detailed proposal for adding workload affinity and anti-affinity support to Karmada. The proposal is well-structured, covering user stories, API changes, and a high-level implementation plan. The design thoughtfully borrows concepts from Kubernetes pod affinity/anti-affinity, which promotes consistency. My review includes a few minor suggestions to fix typos and a copy-paste error in comments within the proposal document to improve clarity.
|
|
||
| Karmada should support being able to schedule these types of duplicate pipelines, by making sure that the applications are never scheduled to the same cluster (which would break HA gaurantees). | ||
|
|
||
| In the picture below, we can see two workloads which represent two duplicate pipelines scheduled for HA purposes. Karmada will make sure that during the scheduling phase, it will not schedule the application to a cluster if a workload with a `karmada.io/group:` label and matching key exists: |
There was a problem hiding this comment.
There seems to be a typo in the label name. The label karmada.io/group: includes a trailing colon, which is likely unintentional. It should probably be karmada.io/group.
| In the picture below, we can see two workloads which represent two duplicate pipelines scheduled for HA purposes. Karmada will make sure that during the scheduling phase, it will not schedule the application to a cluster if a workload with a `karmada.io/group:` label and matching key exists: | |
| In the picture below, we can see two workloads which represent two duplicate pipelines scheduled for HA purposes. Karmada will make sure that during the scheduling phase, it will not schedule the application to a cluster if a workload with a `karmada.io/group` label and matching key exists: |
| // Placement represents the rule for select clusters. | ||
| type Placement struct { | ||
|
|
||
| // WorkloadAffinity represents the inter-workload anti-affinity scheduling policies. |
There was a problem hiding this comment.
There's a typo in the comment. It should refer to WorkloadAntiAffinity to match the field it's describing on line 83.
| // WorkloadAffinity represents the inter-workload anti-affinity scheduling policies. | |
| // WorkloadAntiAffinity represents the inter-workload anti-affinity scheduling policies. |
| // A label query over the set of namespaces that the term applies to. | ||
| // The term is applied to the union of the namespaces selected by this field | ||
| // and the ones listed in the namespaces field. | ||
| // null selector and null or empty namespaces list means "this workloads's namespace". |
There was a problem hiding this comment.
There is a typo in the comment: workloads's should be workload's.
| // null selector and null or empty namespaces list means "this workloads's namespace". | |
| // null selector and null or empty namespaces list means "this workload's namespace". |
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6685 +/- ##
=======================================
Coverage 45.37% 45.37%
=======================================
Files 688 688
Lines 56567 56567
=======================================
Hits 25666 25666
+ Misses 29298 29297 -1
- Partials 1603 1604 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Thank you @mszacillo . We can try to start it in release-1.16. |
|
Hi @kevin-wangzefeng, we went over this proposal (specifically going over the use-cases and the API design) during the community meeting today. I think the next step here is agreeing on the API changes, and once that is solidified, starting to determine an implementation strategy. Was hoping you could take a look, given your contributions to this feature in Kubernetes. Thank you!! |
| // Placement represents the rule for select clusters. | ||
| type Placement struct { | ||
|
|
||
| // WorkloadAffinity represents the inter-workload anti-affinity scheduling policies. |
|
|
||
| ### PropagationPolicy | ||
|
|
||
| ```yaml |
There was a problem hiding this comment.
| ```yaml | |
| ```golang |
same as below.
What type of PR is this?
/kind feature
What this PR does / why we need it:
This PR submits a proposal for adding workload affinity and workload anti-affinity as rules that the user can define on the PropagationPolicy. These rules would allow Karmada to support more complex scheduling challenges such as: