Schedule for disruption

[phajduk]

I try to achieve maintenance window configuration using Karpenter automated nodes recycling. Mostly to recreate them during weekends.

I have the following NodePool disruption:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  disruption:
      consolidationPolicy: WhenUnderutilized
      expireAfter: 1h
      budgets:
      - nodes: "25%"
        schedule: "0 15 * * 6"
        duration: 1h

I assumed the above configuration would expire instance once per hour, with the limitation that it's going to happen only in the allowed budget schedule which is Saturday at 15:00.

expireAfter: 1h means the host is considered expired 1h after creation, but it should be postponed until the budget window that allows to drop 25% of worker nodes in the specified schedule window. However, it seems it doesn't respect the budget, and nodes are recycled outside the mentioned schedule.
Workaround I've found is:

    disruption:
      consolidationPolicy: WhenUnderutilized
      expireAfter: 1h
      budgets:
      - nodes: "0%"
        schedule: "0 * * * mon-fri,sun"
        duration: 1h
      - nodes: "25%"
        schedule: "0 15 * * 6"
        duration: 1h

but it doesn't seem right. We should be able to whitelist "maintenance window" without specyfing "non maintenance window", right?

[Bryce-Soghigian]

I find this pattern emerges more where we have specific periods we tolerate a lot of disruption, then all other periods we don't. The current behavior is if a budget is specified, we allow unbounded disruptions when the specifed budgets are not active.

If no budget is specified in the nodepool we default to 10% disruption. Are you saying you would like to see a default of 0 rather than unbounded disruption?

Would love to get your feedback on some of the new equations I am defining for calculating the disruption budget mapping. See: https://github.com/kubernetes-sigs/karpenter/pull/936/files#diff-5865e4a75963415d65723ff4c6329f04561414abcd1511b9ebf7b47fc02b7f30R250

[phajduk]

@Bryce-Soghigian
Thank you for your response. I will contribute in that RFC.

Are you saying you would like to see a default of 0 rather than unbounded disruption?

Yes, that's my assumption. I feel like right now we blacklist when disruption can't happen. So algorithm calculates the lowest possible disruption so the blacklisted period will take precedence. Assuming it's default 10%, then in the blacklisted period it will be 0%.

My case is the opposite. I want to whitelist when disruption may happen. By default I would expect to have 0% disruption and only in specific maintenance windows disruption should happen. The problem is, the current algorithm takes the lowest possible value from budgets so in that case I probably get 0% all the time.

Do I understand this correctly? My ultimate solution with the current Karpenter logic needs to define all periods (which is 90% of the week) when disruption can't happen and put explicitly another budget with the schedule when it's truly allowed.

[Bryce-Soghigian]

My case is the opposite. I want to whitelist when disruption may happen.

This seems like an interesting semantic. Maybe one could define take the max or the min specified in the disruption controls.

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: worker-pool 
spec: # This is not a complete NodePool Spec.
  disruption:
    consolidationPolicy: WhenEmpty
    expireAfter: Never 
    budgetSettings: 
       unspecifed: "0", "100%" the user could pick the rule they want for undefined. 
       extrema: max || min Uer could say if they want this budget below to resolve to the max value in the budget or the min value in the budget. 
    budgets:
    - nodes: 0
    - nodes: "100%"
    

[Bryce-Soghigian]

The problem is, the current algorithm takes the lowest possible value from budgets so in that case I probably get 0% all the time.

This is correct. It doesn't allow for easy override of saying, during the last weekend of the month, allow 50% disruption. I do like the idea of extending the semantics to allow this kind of case.

[phajduk]

Thanks @Bryce-Soghigian.

[gurpalw]

i'm also looking to create a budget that blocks disruption except for a specific maintenance window. have been unable to do it so far.

[phajduk]

@gurpalw the workaround I use which is aligned with karpenter approach:

    {
      nodes    = "0%"
      schedule = "0 * * * mon-fri,sun"
      duration = "1h"
    },
    {
      nodes    = "0%"
      schedule = "0 0 * * sat"
      duration = "15h"
    },
    {
      nodes    = "25%" <- maintenance
      schedule = "0 15 * * sat"
      duration = "2h"
    },
    {
      nodes    = "0%"
      schedule = "0 17 * * sat"
      duration = "8h"
    }

[gurpalw]

thanks, it's a shame it has to be so convoluted.