v0.12.1

This is a new minor release of NRI Reference Plugins. It brings a few minor new features, an overhauled documentation for the resource policy plugins, some bugfixes and dependency updates.

What's New

Balloons Policy

• Scheduling Class is a is a list of scheduling related parameters organized in classes. These parameters tune containers when they are created but do not affect already running containers. The class of a container is defined with the schedulingClass: <name> option in container's balloon type, and can be overridden by scheduling-class.resource-policy.nri.io pod annotation. Each class in the list has following properties.
  • name is the name of the scheduling class.
  • policy is the Linux scheduling policy. Supported policies are: none, other, fifo, rr, batch, idle, and deadline.

• priority is the scheduling priority. Refer to sched_setscheduler(2) documentation for valid values depending on the policy.
• flags is a list of scheduling flags. Supported flags are: reset-on-fork, reclaim, dl-overrun, keep-policy, keep-params, util-clamp-min, util-clamp-max.
• nice: nice value for the container process.
• runtime: runtime value for deadline scheduling policy (in microseconds).
• deadline: deadline value for deadline scheduling policy (in microseconds).
• period: period value for deadline scheduling policy (in microseconds).
• ioClass: IO class for the container process. Supported classes are: none, rt for realtime, be for best-effort, and idle.
• ioPriority: IO priority for the container process. Refer to ionice(1) documentation for valid values.
  For example the following configuration fragment defines two balloon types, high-priority and low-priority, assigning them to the critical and background scheduling classes, with the former using a realtime scheduling policy and elevated I/O priority, and the latter using an idle scheduling policy.

balloonTypes:
- name: high-priority
  schedulingClass: critical
  ...
- name: low-priority
  schedulingClass: background
  ...
schedulingClasses:
- name: critical
  policy: rr
  priority: 50
  ioClass: rt
  ioPriority: 0
- name: background
  policy: idle
  ioClass: idle

• Excluded CPU Set notation for allowed and reserved resources allows one to specify the allowed and reserved CPU set by specifying which CPUs should be excluded from the set instead of giving explicitly the included ones. This can be especially useful in a hybrid node cluster when the number of CPUs present varies between nodes. For the available resources exclusion is with respect to the CPUs present in the system. For reserved resources exclusion is with respect to the CPU present in the available resource set.

Topology Aware

• Scheduling Class A container can be assigned to a known 'scheduling class' by name using the scheduling-class.resource-policy.nri.io effective annotation key. The value of the annotation is the name of the class for the container or the pod. The class itself needs to be defined in the active policy configuration using the schedulingClasses configuration option. For instance the following Helm configuration fragment defines two classes, realtime and idle with the corresponding scheduling and I/O priority attributes.

config:
  reservedResources:
    cpu: 2
  ...
  schedulingClasses:
  - name: realtime
    policy: fifo # SCHED_FIFO
    priority: 42

  - name: idle
    policy: idle # SCHED_IDLE
    nice: 17
    ioClass: be
    ioPriority: 6
...

The following pod annotation will then assign the container c0 to the realtime class:

metadata:
  annotations:
    scheduling-class.resource-policy.nri.io/container.c0: realtime

• Inherited Scheduling Classes The new NamespaceSchedulingClasses and PodQoSSchedulingClasses configuration CR options can be used to set up default scheduling classes per namespace and Pod QoS class. If a container is not assigned explicitly to any scheduling class by annotation, it inherits the default scheduling class for its namespace or Pod QoS class, in this order of precedence, if either or both is set.
• Excluded CPU Set notation for allowed and reserved resources allows one to specify the allowed and reserved CPU set by specifying which CPUs should be excluded from the set instead of giving explicitly the included ones. This can be especially useful in a hybrid node cluster when the number of CPUs present varies between nodes. For the available resources exclusion is with respect to the CPUs present in the system. For reserved resources exclusion is with respect to the CPU present in the available resource set.

Resource Annotator Webhook

• Cert-Manager support It is now possible to generate and inject HTTPS certificates for the webhook using cert-manager. Here is a sample session of setting up the webhook for cert-manager injection.

# Install cert-manager, if you don't have it yet.
$ helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
       --version v1.19.2 --namespace cert-manager --create-namespace \
       --set crds.enabled=true --set crds.keep=false

# Bootstrap a local issuer for cert-manager if you don't have one yet.
$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-root-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: internal-root-ca
  secretName: internal-root-ca-secret
  issuerRef:
    name: selfsigned-cluster-issuer
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: console-ca-issuer
  namespace: cert-manager
spec:
  ca:
    secretName: internal-root-ca-secret
EOF
$ kubectl apply -f ca-bootstrap.yaml
$ kubectl wait --for=condition=Ready=True clusterissuer/console-ca-issuer

# Install the chart referring it to the certificate issuer.
$ helm install -n kube-system nri-webhook nri-plugins/nri-resource-annotator \
      --set image.tag=v0.12.0 --set image.pullPolicy=IfNotPresent \
      --set service.certificateIssuer=console-ca-issuer

This should set up the resource annotator with a cert-manager issued
certificate.

What's Changed

• helm: adjust resource-annotator helm chart for better consistency with the rest. by @klihub in #624
• scripts: add resource-annotator to artifact checker script. by @klihub in #623
• helm,resource-annotator: add chart support for cert-manager. by @klihub in #625
• balloons: support configuring Linux scheduling parameters by @askervin in #618
• topology-aware: initial scheduling class support. by @klihub in #626
• resmgr: update topology change on reconfigure by @askervin in #629
• balloons: fix insufficient memory on reconfigurations by @askervin in #630
• doc: balloons documentation rewrite + cookbook by @askervin in #627
• topology-aware: add 'excluded cpuset' config notation by @klihub in #631
• operator: bypass ansible-operator proxy for Helm deployments by @fmuyassarov in #633
• e2e: be more specific when looking for known container processes with pgrep. by @klihub in #636
• docs: restructure topology-aware documentation. by @klihub in #628

Full Changelog: v0.12.0...v0.12.1