restated pods don't have CPUBoosts applied

[camaeel]

Community Note

• Please vote on this issue by adding a 👍 reaction
  to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do
  not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Debug Output

Panic Output

Steps to Reproduce

1. Create deployment like this:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: boost-test
     name: boost-test
     namespace: boost-test
   spec:
     replicas: 4
     selector:
       matchLabels:
         app: boost-test
     template:
       metadata:
         labels:
           app: boost-test
       spec:
         affinity:
           podAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               - labelSelector:
                   matchExpressions:
                     - key: app
                       operator: In
                       values:
                         - boost-test
                 topologyKey: kubernetes.io/hostname
         containers:
           - image: nginx
             name: nginx
             resources:
               requests:
                 cpu: 100m
                 memory: 20Mi
               limits:
                 memory: 100Mi
             livenessProbe:
               httpGet:
                 port: 80
                 path: /indexxx.html
               initialDelaySeconds: 40
               periodSeconds: 5
               failureThreshold: 3
               successThreshold: 1
               timeoutSeconds: 2
             readinessProbe:
               httpGet:
                 port: 80
                 path: /index.html
               initialDelaySeconds: 20
               periodSeconds: 5
               failureThreshold: 3
               successThreshold: 1
               timeoutSeconds: 2
   

   and CPUBoost:

   apiVersion: autoscaling.x-k8s.io/v1alpha1
   kind: StartupCPUBoost
   metadata:
     name: boost-cpu
     namespace: boost-test
   selector:
   #  matchLabels: {}
   #  matchExpressions:
   #    - key: app
   #      operator: In
   #      values: [ "boost-test" ]
     matchLabels:
       app: boost-test
   spec:
     resourcePolicy:
       containerPolicies:
         - containerName: nginx
           fixedResources:
             requests: "900m"
     durationPolicy:
       podCondition:
         type: Ready
         status: "True"
   

2. Observe pod requests:

   watch "kubectl get pods -ojson | jq -r  '.items[] | .spec.containers[0].resources.requests'"
   

Expected Behavior

Operator could add resource boosts to pods that were restarted.

Actual Behavior

When pod is restarted app inside goes through initialization and startup again, so it will consume more memory, but this is not reflected in CPU requests/limits, and may end up with starvation.

References

• #0000

[mikouaj]

@camaeel I understand the issue but I'm not sure if increasing CPU resources on POD restart will make a big difference.
The application will be already starting once resources get increased. The resource resize operation may take some time itself.

Also, for JVM applications, the container detection mechanism won't reflect increased resources (i.e. for setting up number of active processor count) as resource increase will kick in once application will be starting.

Can you elaborate more how do you see this ?

[camaeel]

I see your point.
For the limits it won't make a big difference. But if I increase CPU request, scheduling of new pods on this node will be blocked until request is decreased again.

[AndreiDoroftei]

Would really benefit from this feature.

[nenych]

CPU boost on pod restart is the most wanted feature, without it seems as half working thing. If my service restarts on oom, or for any other reason, I will have 0 use on this addon.