Develop a custom ansible strategy plugin to perform graceful upgrade in a true rolling fashion

[VannTen]

What would you like to be added

Basically: ansible-collections/community.general#10920

(Upstreaming the strategy into community.general would also work)

Once we switch to collection only (#12227 ) we'll be able to handle ansible plugins more easily.

WARNING: This is not an easy task, and requires to dive deep into Ansible internals and do a fair bit of Python. This not at all a "good first issue"

Why is this needed

There is 2 purpose for this strategy:

• faster cluster upgrade, because there is no longer a "all nodes in a batch wait for all other nodes" bottleneck.
• make possible cluster upgrades with bigger "batch size" (currently, a PodDisruptionBudbget can block your upgrade if enoug eligible nodes for a deployment are in the same batch. With that scenario, some of the nodes where the eviction succeded would finish instead of waiting of the rest of the batch, allowing pods to be scheduled on them and the other node drain to succeed).

/lifecycle frozen
/help

[k8s-ci-robot]

@VannTen:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

What would you like to be added

Basically: ansible-collections/community.general#10920

(Upstreaming the strategy into community.general would also work)

Once we switch to collection only (#12227 ) we'll be able to handle ansible plugins more easily.

WARNING: This is not an easy task, and requires to dive deep into Ansible internals and do a fair bit of Python. This not at all a "good first issue"

Why is this needed

There is 2 purpose for this strategy:

• faster cluster upgrade, because there is no longer a "all nodes in a batch wait for all other nodes" bottleneck.
• make possible cluster upgrades with bigger "batch size" (currently, a PodDisruptionBudbget can block your upgrade if enoug eligible nodes for a deployment are in the same batch. With that scenario, some of the nodes where the eviction succeded would finish instead of waiting of the rest of the batch, allowing pods to be scheduled on them and the other node drain to succeed).

/lifecycle frozen
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[spline2hg]

Hi, I understand this is a complex issue, but I’d like to give it a try. I’ll get back to you if I get stuck. Thanks!

[spline2hg]

Hi @VannTen, I’ve been looking into the strategy methods and wanted to clarify my understanding of the problem.

Currently, we use the linear strategy with serial to handle batches. which make the nodes wait for all other nodes in the batch to complete before moving on.

I was wondering why the free strategy isn’t sufficient for this use case to make this more asynchronous. Is it mainly because:

• free relies on the global forks setting, so we can’t control concurrency per play (or differentiate masters vs workers) the way serial allows?
• free provides task-based parallelism rather than a host-based . since it does not follow a windowed model like serial, allowing newer hosts to start tasks before earlier hosts have completed all tasks?

Is the goal here basically to get host_pinned behavior with the ability to define the "concurrency" at the play level instead of the global config?

[VannTen]

Is the goal here basically to get host_pinned behavior with the ability to define the "concurrency" at the play level instead of the global config?

Yes mostly. This should allow for the following:

Given an inventory of five hosts A B C D Eand a "concurrency" of 2;
Play start for A and B, no synchronization (like free).
A finishes the play, C starts (B & C are running)

(Let's suppose B take a very long time because the drain is long)
C finishes the play, D starts (B & D are running)
B finishes the play, E starts (D & E are running).

Does that make it clear ?

Further possible refinements on the strategy:

• Allow different groups with different "inner concurrency" (for example, if we only want a dedicated node group to upgrade at most 2 at a time, but the rest of the nodes up to 10 at a time).

[spline2hg]

Does that make it clear ?

Yes, this makes sense.
I can start by an initial implementation, should it be in this repository or community.general?

[VannTen]

Probably here, as testing with kubespray will be harder if you use a community.general fork (I think. You can put a git-ref in collections deps ? Not sure).

Do note that AFAICT, it's very inconvenient for custom ansible plugins to work well in both non-collection and collection mode, you have to use both unqualified and qualified names conditionally and it's a bit of a mess.

So for testing in CI we'd need #12732