[3] Add Safeguard for pods that prevent CAPI cluster-autoscaler

[anders-elastisys]

Proposed feature

The cluster-autoscaler documentation contains a list of what types of pods can prevent the autoscaler from scaling down nodes. Would like some OPA Gatkeeper constraints that warns or restricts (depending on the constraint) Application Developers when they try to deploy such pods in environments with cluster-autoscaler (e.g. CAPI). We currently already have a lot of safeguards that would prevent Application Developers from deploying such pods, e.g. constraint for hostPath, but at least non-memory emptyDir are still allowed which can prevent the autoscaler. If this gets added, the public documentation should be updated with the new safeguard(s) and also this notice should probably be updated as well.

Proposed alternatives

• Don't add safeguards for this
• Add some operator that labels preventing pods with safe-to-evict labels automatically in Application Developer namespaces

Additional context

https://github.com/elastisys/ck8s-issue-tracker/issues/313

Definition of done

• Added safeguards for Application Developer pods as to not prevent cluster-autoscaler from scaling down clusters
• The public documentation is updated with the Safeguard(s)