feat(recommender): add OOMMinBumpUp&OOMBumpUpRatio to CRD

[omerap12]

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR adds new options to the Vertical Pod Autoscaler (VPA) to better handle Out of Memory (OOM) events:
It adds two new settings to the VPA configuration:

• OOMBumpUpRatio: How much to increase memory after an OOM event
• OOMMinBumpUp: The smallest amount to increase memory after an OOM event

These settings can be set for each container within a VPA's resource policy.
If not set for a specific container, it will use default values from the VPA recommender.

Example:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: oom-test-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: oom-test
  updatePolicy:
    updateMode: Auto
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
     oomBumpUpRatio: "3.5"
     oomMinBumpUp: "100Mi"

Which issue(s) this PR fixes:

part of #7650

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Added OOMBumpUpRatio and OOMMinBumpUp options to VPA for customizing memory increase after OOM events.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

VPA now supports OOMBumpUpRatio and OOMMinBumpUp for fine-tuning memory recommendations after OOM events, configurable globally or per-VPA.

[omerap12]

We might want to create a proper AEP for this, but this is the general direction I'm thinking. I can open additional issues to track the specific flags we’d like to support for this type of configuration.
What do you think?
cc @voelzmo @raywainman
(Wanted to loop in Adrian as well, but he's currently traveling :) )
/hold
/kind api-change

[voelzmo]

Hey @omerap12 thanks for the PR!

I agree it makes sense to be able to configure the OOM bump behavior on VPA level. There's a few questions on how to implement this, though:

• I'm not sure if we want this to be a configuration on Container level or on Pod level, i.e. should this apply to all Containers controlled by a certain VPA or should this rather be something that's controlled per individual Container? I think so far we've been mostly offering configurations on Container level, probably this would also apply here. Or do we have some indication that people who want to configure custom OOM bumps want to do this for all Containers of a Pod in the same way?
• I don't think we should introduce a new configuration type recommenderConfig. Technically, all of these properties are configuration options of the recommender (histogram decay options, maxAllowed, minAllowed, which resources to include in the recommendations, etc), so this doesn't seem like a reasonable way to group things. If we agree to make this configuration Container specific, I'd rather add this to the ContainerResourcePolicy
• Currently, OOM bump configuration is part of the AggregationsConfig, as it is assumed to be globally configured, like all the other options in there. This config is only initialized once, in the main.go:
  

  autoscaler/vertical-pod-autoscaler/pkg/recommender/main.go

  Line 224 in 3e92831

  model.InitializeAggregationsConfig(model.NewAggregationsConfig(*memoryAggregationInterval, *memoryAggregationIntervalCount, *memoryHistogramDecayHalfLife, *cpuHistogramDecayHalfLife, *oomBumpUpRatio, *oomMinBumpUp))

  • If we, however, want to make this configurable per VPA, I'd rather opt for pushing this configuration down, rather than adding some if-else to the cluster_feeder resulting in having to find the correct VPA for a Pod every time we add an OOM sample
  • IMHO, a possible place to put this configuration options would be the aggregate_container_state, where we already have the necessary methods to re-load the ContainerResourcePolicy options on VPA updates and then read this in cluster.go, right before we add the OOM sample to the ContainerAggregation:
    

    autoscaler/vertical-pod-autoscaler/pkg/recommender/model/cluster.go

    Line 262 in 3e92831

    err := containerState.RecordOOM(timestamp, requestedMemory)

WDYT?

[omerap12]

Hey @omerap12 thanks for the PR!

I agree it makes sense to be able to configure the OOM bump behavior on VPA level. There's a few questions on how to implement this, though:

• I'm not sure if we want this to be a configuration on Container level or on Pod level, i.e. should this apply to all Containers controlled by a certain VPA or should this rather be something that's controlled per individual Container? I think so far we've been mostly offering configurations on Container level, probably this would also apply here. Or do we have some indication that people who want to configure custom OOM bumps want to do this for all Containers of a Pod in the same way?

• I don't think we should introduce a new configuration type recommenderConfig. Technically, all of these properties are configuration options of the recommender (histogram decay options, maxAllowed, minAllowed, which resources to include in the recommendations, etc), so this doesn't seem like a reasonable way to group things. If we agree to make this configuration Container specific, I'd rather add this to the ContainerResourcePolicy

• Currently, OOM bump configuration is part of the AggregationsConfig, as it is assumed to be globally configured, like all the other options in there. This config is only initialized once, in the main.go:
  

  autoscaler/vertical-pod-autoscaler/pkg/recommender/main.go

  Line 224 in 3e92831

  model.InitializeAggregationsConfig(model.NewAggregationsConfig(*memoryAggregationInterval, *memoryAggregationIntervalCount, *memoryHistogramDecayHalfLife, *cpuHistogramDecayHalfLife, *oomBumpUpRatio, *oomMinBumpUp))

  • If we, however, want to make this configurable per VPA, I'd rather opt for pushing this configuration down, rather than adding some if-else to the cluster_feeder resulting in having to find the correct VPA for a Pod every time we add an OOM sample
  • IMHO, a possible place to put this configuration options would be the aggregate_container_state, where we already have the necessary methods to re-load the ContainerResourcePolicy options on VPA updates and then read this in cluster.go, right before we add the OOM sample to the ContainerAggregation:
    

    autoscaler/vertical-pod-autoscaler/pkg/recommender/model/cluster.go

    Line 262 in 3e92831

    err := containerState.RecordOOM(timestamp, requestedMemory)

WDYT?

Thanks for the input!

1. You're right. it makes sense to keep this as a per-container configuration, in line with most of our existing settings.
2. The recommenderConfig was just part of my initial POC, so with (1) in mind, we definitely don’t need it.
3. Agreed. Thanks for pointing out the relevant spot in the code!

So yep, I agree with all of your suggestions :)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: omerap12

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• vertical-pod-autoscaler/OWNERS [omerap12]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[omerap12]

/remove area provider/cluster-api
/remove area/cluster-autoscaler

[omerap12]

/remove-area provider/cluster-api
/remove-area cluster-autoscaler

[adrianmoisey]

/label api-review

[adrianmoisey]

This piece of code made me wonder if we should try move closer to how k/k validates resources. That's something for another day though

[omerap12]

Yeah.. we should create a seperate issue to discuss it.

[liggitt]

trying to kick the api-review label to exercise project automation
/remove-label api-review

[liggitt]

/label api-review

[soltysh]

/hold
just to make sure this doesn't merge before going through proper API review

[soltysh]

Here are several questions from API shadow review pov, and reading through AEP-8026:

1. How does setting these two work with the global flags? The document does not have clear answer for that, but skimming this PR it seems you're leaving the flags as is, and adding this capability at VPA level.
2. What alternatives were considered for placing both of these at container level? I don't see any discussion why it's worth setting at VPA level, pod level or container level. Which one is desired and which one makes more sense. I'd suggest thinking through questions do you really want to set it per container, and not per pod, or even per entire workload/app? What benefits you gain or loose by placing that kind of granular control at various levels.
3. Why you need to have 2 separate fields for something that seems to express the same thing (partially at least). Why not just a single OOBumpUp defined as resource.Quantity isn't sufficient? In which case, you could express both ratio as 1.2 or percentage, or explicit bytes value.
   When you're considering exposing some of the internal configs you don't always have to expose them literally, something simpler might be sufficient here.
4. What happens when both of these will be set? I haven't found clear answer in the doc to that question, either. Only When both oomBumpUpRatio = 1 and oomMinBumpUp = 0, OOM-based memory increases are completely disabled. Are they mutually exclusive, additive, averaged?

[omerap12]

1. I thought I had written this in the AEP - I need to check. The idea is that this setting will override the one in the global config. If the user adds it to their VPA object, those values will be applied. otherwise, the global flags will be used.
2. Good point. We will discuss it.
3. The core idea is not to introduce a new configuration for the VPA, but simply to expose these settings in the VPA object. I agree that this makes sense (and I’m a little embarrassed I didn’t notice it myself).
4. This hasn't change - we are taking the max between those two:

   autoscaler/vertical-pod-autoscaler/pkg/recommender/model/container.go

   Lines 192 to 193 in f1251a1

   	memoryNeeded := ResourceAmountMax(memoryUsed+MemoryAmountFromBytes(GetAggregationsConfig().OOMMinBumpUp),
   	ScaleResource(memoryUsed, GetAggregationsConfig().OOMBumpUpRatio))

[soltysh]

1. I thought I had written this in the AEP - I need to check. The idea is that this setting will override the one in the global config. If the user adds it to their VPA object, those values will be applied. otherwise, the global flags will be used.

Please update the AEP, and link here the update.

2. Good point. We will discuss it.

Again, please make sure to update the AEP, accordingly.

3. The core idea is not to introduce a new configuration for the VPA, but simply to expose these settings in the VPA object. I agree that this makes sense (and I’m a little embarrassed I didn’t notice it myself).

Again, document 😅

4. This hasn't change - we are taking the max between those two:

   autoscaler/vertical-pod-autoscaler/pkg/recommender/model/container.go

   Lines 192 to 193 in f1251a1

   	memoryNeeded := ResourceAmountMax(memoryUsed+MemoryAmountFromBytes(GetAggregationsConfig().OOMMinBumpUp),
   	ScaleResource(memoryUsed, GetAggregationsConfig().OOMBumpUpRatio))

I believe, once you reconsider question 3, you'll have a better answer and just make sure to write it down.

[omerap12]

Yeah, we submitted a PR to update the AEP here: #8505
I'll ping you once @adrianmoisey will check that out first

[liggitt]

/assign @soltysh @liggitt

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

@omerap12: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-autoscaling-e2e-vpa-full	697934c	link	true	/test pull-autoscaling-e2e-vpa-full

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.