CKS Enhancements (apache#9102)

CKS Enhancements:

* Ability to specify different compute or service offerings for different types of CKS cluster nodes – worker, master or etcd

* Ability to use CKS ready custom templates for CKS cluster nodes

* Add and Remove external nodes to and from a kubernetes cluster

Co-authored-by: nvazquez <[email protected]>

* Update remove node timeout global setting

* CKS/NSX : Missing variables in worker nodes

* CKS: Fix ISO attach logic

* CKS: Fix ISO attach logic

* address comment

* Fix Port - Node mapping when cluster is scaled in the presence of external node(s)

* CKS: Externalize control and worker node setup wait time and installation attempts

* Fix logger

* Add missing headers and fix end of line on files

* CKS Mark Nodes for Manual Upgrade and Filter Nodes to add to CKS cluster from the same network

* Add support to deploy CKS cluster nodes on hosts dedicated to a domain

---------

Co-authored-by: Pearl Dsilva <[email protected]>

* Support unstacked ETCD

---------

Co-authored-by: nvazquez <[email protected]>

* Fix CKS cluster scaling and minor UI improvement

* Reuse k8s cluster public IP for etcd nodes and rename etcd nodes

* Fix DNS resolver issue

* Update UDP active monitor to ICMP

* Add hypervisor type to CKS cluster creation to fix CKS cluster creation when External hosts added

* Fix build

* Fix logger

* Modify hypervisor param description in the create CKS cluster API

* CKS delete fails when external nodes are present

* CKS delete fails when external nodes are present

* address comment

* Improve network rules cleanup on failure adding external nodes to CKS cluster

* UI: Fix etcd template was not honoured

* UI: Fix etcd template was not honoured

* Refactor

* CKS: Exclude etcd nodes when calculating port numbers

* Fix network cleanup in case of CKS cluster failure

* Externalize retries and inverval for NSX segment deletion

* Fix CKS scaling when external node(s) present in the cluster

* CKS: Fix port numbers displayed against ETCD nodes

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* Add node version details to every node of k8s cluster - as we now support manual upgrade

* update column name

* CKS: Exclude etcd nodes when calculating port numbers

* update param name

* update param

* UI: Fix CKS cluster creation templates listing for non admins

* CKS: Prevent etcd node start port number to coincide with k8s cluster start port numbers

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* CKS: Set default kubernetes cluster node version to the kubernetes cluster version on upgrade

* consolidate query

* Fix upgrade logic

---------

Co-authored-by: nvazquez <[email protected]>

* Fix CKS cluster version upgrade

* CKS: Fix etcd port numbers being skipped

* Fix CKS cluster with etcd nodes on VPC

* Move schema and upgrade for 4.20

* Fix logger

* Fix after rebasing

* Add support for using different CNI plugins with CKS

* Add support for using different CNI plugins with CKS

* remove unused import

* Add UI support and list cni config API

* necessary UI changes

* add license

* changes to support external cni

* UI changes

* Fix NPE on restarting VPC with additional public IPs

* fix merge conflict

* add asnumber to create k8s svc layer

* support cni framework to use as-numbers

* update code

* condition to ignore undefined jinja template variables

* CKS: Do not pass AS number when network ID is passed

* Fix deletion of Userdata / CNI Configuration in projects

* CKS: Add CNI configuration details to the response and UI

* Explicit events for registering cni configuration

* Add Delete cni configuration API

* Fix CKS deployment when using VPC tiers with custom ACLs

* Fix DNS list on VR

* CKS: Use Network offering of the network passed during CKS cluster creation to get the AS number

* CKS cluster with guest IP

* Fix: Use control node guest IP as join IP for external nodes addition

* Fix DNS resolver issue

* Improve etcd indexing - start from 1

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* CKS: Add external node to a CKS cluster deployed with etcd node(s) successfully

* simplify logic

* Tweak setup-kube-system script for baremetal external nodes

* Consider cordoned nodes while getting ready nodes

* Fix CKS cluster scale calculations

* Set token TTL to 0 (no expire) for external etcd

* Fix missing quotes

* Fix build

* Revert PR 9133

* Add calico commands for ens35 interface

* Address review comments: plan CKS cluster deployment based on the node type

* Add qemu-guest-agent dependency for kvm based templates

* Add marvin test for CKS clusters with different offerings per node type

* Remove test tag

* Add marvin test and fix update template for cks and since annotations

* Fix marvin test for adding and removing external nodes

* Fix since version on API params

* Address review comments

* Fix unit test

* Address review comments

* UI: Make CKS public templates visible to non-admins on CKS cluster creation

* Fix linter

* Fix merge error

* Fix positional parameters on the create kubernetes ISO script and make the ETCD version optional

* fix etcd port displayed

* Further improvements to CKS  (#118)

* Multiple nics support on Ubuntu template

* Multiple nics support on Ubuntu template

* supports allocating IP to the nic when VM is added to another network - no delay

* Add option to select DNS or VR IP as resolver on VPC creation

* Add API param and UI to select option

* Add column on vpc and pass the value on the databags for CsDhcp.py to fix accordingly

* Externalize the CKS Configuration, so that end users can tweak the configuration before deploying the cluster

* Add new directory to c8 packaging for CKS config

* Remove k8s configuration from resources and make it configurable

* Revert "Remove k8s configuration from resources and make it configurable"

This reverts commit d5997033ebe4ba559e6478a64578b894f8e7d3db.

* copy conf to mgmt server and consume them from there

* Remove node from cluster

* Add missing /opt/bin directory requrired by external nodes

* Login to a specific Project view

* add indents

* Fix CKS HA clusters

* Fix build

---------

Co-authored-by: Nicolas Vazquez <[email protected]>

* Add missing headers

* Fix linter

* Address more review comments

* Fix unit test

* Fix scaling case for the same offering

* Revert "Login to a specific Project view"

This reverts commit 95e3756.

* Revert "Fix CKS HA clusters" (#120)

This reverts commit 8dac16a.

* Apply suggestions from code review about user data

Co-authored-by: Suresh Kumar Anaparti <[email protected]>

* Update api/src/main/java/org/apache/cloudstack/api/command/user/userdata/BaseRegisterUserDataCmd.java

Co-authored-by: Suresh Kumar Anaparti <[email protected]>

* Refactor column names and schema path

* Fix scaling for non existing previous offering per node type

* Update node offering entry if there was an existing offering but a global service offering has been provided on scale

---------

Co-authored-by: Pearl Dsilva <[email protected]>
Co-authored-by: Daan Hoogland <[email protected]>
Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Author: 4 people

api/src/main/java/com/cloud/event/EventTypes.java

@@ -293,6 +293,8 @@ public class EventTypes {
 
     //registering userdata events
     public static final String EVENT_REGISTER_USER_DATA = "REGISTER.USER.DATA";
+    public static final String EVENT_REGISTER_CNI_CONFIG = "REGISTER.CNI.CONFIG";
+    public static final String EVENT_DELETE_CNI_CONFIG = "DELETE.CNI.CONFIG";
 
     //register for user API and secret keys
     public static final String EVENT_REGISTER_FOR_SECRET_API_KEY = "REGISTER.USER.KEY";

plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/KubernetesCluster.java renamed to api/src/main/java/com/cloud/kubernetes/cluster/KubernetesCluster.java

@@ -44,6 +44,8 @@ enum Event {
         AutoscaleRequested,
         ScaleUpRequested,
         ScaleDownRequested,
+        AddNodeRequested,
+        RemoveNodeRequested,
         UpgradeRequested,
         OperationSucceeded,
         OperationFailed,
@@ -59,6 +61,8 @@ enum State {
         Stopped("All resources for the Kubernetes cluster are destroyed, Kubernetes cluster may still have ephemeral resource like persistent volumes provisioned"),
         Scaling("Transient state in which resources are either getting scaled up/down"),
         Upgrading("Transient state in which cluster is getting upgraded"),
+        Importing("Transient state in which additional nodes are added as worker nodes to a cluster"),
+        RemovingNodes("Transient state in which additional nodes are removed from a cluster"),
         Alert("State to represent Kubernetes clusters which are not in expected desired state (operationally in active control place, stopped cluster VM's etc)."),
         Recovering("State in which Kubernetes cluster is recovering from alert state"),
         Destroyed("End state of Kubernetes cluster in which all resources are destroyed, cluster will not be usable further"),
@@ -96,6 +100,17 @@ enum State {
             s_fsm.addTransition(State.Upgrading, Event.OperationSucceeded, State.Running);
             s_fsm.addTransition(State.Upgrading, Event.OperationFailed, State.Alert);
 
+            s_fsm.addTransition(State.Running, Event.AddNodeRequested, State.Importing);
+            s_fsm.addTransition(State.Alert, Event.AddNodeRequested, State.Importing);
+            s_fsm.addTransition(State.Importing, Event.OperationSucceeded, State.Running);
+            s_fsm.addTransition(State.Importing, Event.OperationFailed, State.Running);
+            s_fsm.addTransition(State.Alert, Event.OperationSucceeded, State.Running);
+
+            s_fsm.addTransition(State.Running, Event.RemoveNodeRequested, State.RemovingNodes);
+            s_fsm.addTransition(State.Alert, Event.RemoveNodeRequested, State.RemovingNodes);
+            s_fsm.addTransition(State.RemovingNodes, Event.OperationSucceeded, State.Running);
+            s_fsm.addTransition(State.RemovingNodes, Event.OperationFailed, State.Running);
+
             s_fsm.addTransition(State.Alert, Event.RecoveryRequested, State.Recovering);
             s_fsm.addTransition(State.Recovering, Event.OperationSucceeded, State.Running);
             s_fsm.addTransition(State.Recovering, Event.OperationFailed, State.Alert);
@@ -142,4 +157,13 @@ enum State {
     Long getMaxSize();
     Long getSecurityGroupId();
     ClusterType getClusterType();
+    Long getControlNodeServiceOfferingId();
+    Long getWorkerNodeServiceOfferingId();
+    Long getEtcdNodeServiceOfferingId();
+    Long getControlNodeTemplateId();
+    Long getWorkerNodeTemplateId();
+    Long getEtcdNodeTemplateId();
+    Long getEtcdNodeCount();
+    Long getCniConfigId();
+    String getCniConfigDetails();
 }

api/src/main/java/com/cloud/kubernetes/cluster/KubernetesServiceHelper.java

@@ -18,14 +18,23 @@
 
 import org.apache.cloudstack.acl.ControlledEntity;
 
+import java.util.Map;
+
 import com.cloud.user.Account;
 import com.cloud.uservm.UserVm;
 import com.cloud.utils.component.Adapter;
 
 public interface KubernetesServiceHelper extends Adapter {
 
+    enum KubernetesClusterNodeType {
+        CONTROL, WORKER, ETCD, DEFAULT
+    }
+
     ControlledEntity findByUuid(String uuid);
     ControlledEntity findByVmId(long vmId);
     void checkVmCanBeDestroyed(UserVm userVm);
+    boolean isValidNodeType(String nodeType);
+    Map<String, Long> getServiceOfferingNodeTypeMap(Map<String, Map<String, String>> serviceOfferingNodeTypeMap);
+    Map<String, Long> getTemplateNodeTypeMap(Map<String, Map<String, String>> templateNodeTypeMap);
     void cleanupForAccount(Account account);
 }

api/src/main/java/com/cloud/network/NetworkService.java

@@ -268,4 +268,6 @@ Network createPrivateNetwork(String networkName, String displayText, long physic
     InternalLoadBalancerElementService getInternalLoadBalancerElementByNetworkServiceProviderId(long networkProviderId);
     InternalLoadBalancerElementService getInternalLoadBalancerElementById(long providerId);
     List<InternalLoadBalancerElementService> getInternalLoadBalancerElements();
+
+    boolean handleCksIsoOnNetworkVirtualRouter(Long virtualRouterId, boolean mount) throws ResourceUnavailableException;
 }

api/src/main/java/com/cloud/network/vpc/Vpc.java

@@ -105,4 +105,6 @@ public enum State {
     String getIp6Dns1();
 
     String getIp6Dns2();
+
+    boolean useRouterIpAsResolver();
 }

api/src/main/java/com/cloud/network/vpc/VpcService.java

@@ -48,17 +48,17 @@ public interface VpcService {
      * @param vpcName
      * @param displayText
      * @param cidr
-     * @param networkDomain TODO
+     * @param networkDomain   TODO
      * @param ip4Dns1
      * @param ip4Dns2
-     * @param displayVpc TODO
+     * @param displayVpc      TODO
+     * @param useVrIpResolver
      * @return
      * @throws ResourceAllocationException TODO
      */
     Vpc createVpc(long zoneId, long vpcOffId, long vpcOwnerId, String vpcName, String displayText, String cidr, String networkDomain,
                   String ip4Dns1, String ip4Dns2, String ip6Dns1, String ip6Dns2, Boolean displayVpc, Integer publicMtu, Integer cidrSize,
-                  Long asNumber, List<Long> bgpPeerIds)
-            throws ResourceAllocationException;
+                  Long asNumber, List<Long> bgpPeerIds, Boolean useVrIpResolver) throws ResourceAllocationException;
 
     /**
      * Persists VPC record in the database

api/src/main/java/com/cloud/server/ManagementService.java

@@ -70,8 +70,10 @@
 import org.apache.cloudstack.api.command.user.ssh.DeleteSSHKeyPairCmd;
 import org.apache.cloudstack.api.command.user.ssh.ListSSHKeyPairsCmd;
 import org.apache.cloudstack.api.command.user.ssh.RegisterSSHKeyPairCmd;
+import org.apache.cloudstack.api.command.user.userdata.DeleteCniConfigurationCmd;
 import org.apache.cloudstack.api.command.user.userdata.DeleteUserDataCmd;
 import org.apache.cloudstack.api.command.user.userdata.ListUserDataCmd;
+import org.apache.cloudstack.api.command.user.userdata.RegisterCniConfigurationCmd;
 import org.apache.cloudstack.api.command.user.userdata.RegisterUserDataCmd;
 import org.apache.cloudstack.api.command.user.vm.GetVMPasswordCmd;
 import org.apache.cloudstack.api.command.user.vmgroup.UpdateVMGroupCmd;
@@ -393,17 +395,23 @@ public interface ManagementService {
      *            The api command class.
      * @return The list of userdatas found.
      */
-    Pair<List<? extends UserData>, Integer> listUserDatas(ListUserDataCmd cmd);
+    Pair<List<? extends UserData>, Integer> listUserDatas(ListUserDataCmd cmd, boolean forCks);
+
+    /**
+     * Registers a cni configuration.
+     *
+     * @param cmd    The api command class.
+     * @return A VO with the registered user data.
+     */
+    UserData registerCniConfiguration(RegisterCniConfigurationCmd cmd);
 
     /**
      * Registers a userdata.
      *
-     * @param cmd
-     *            The api command class.
+     * @param cmd    The api command class.
      * @return A VO with the registered userdata.
      */
     UserData registerUserData(RegisterUserDataCmd cmd);
-
     /**
      * Deletes a userdata.
      *
@@ -413,6 +421,14 @@ public interface ManagementService {
      */
     boolean deleteUserData(DeleteUserDataCmd cmd);
 
+    /**
+     * Deletes user data.
+     *
+     * @param cmd
+     *            The api command class.
+     * @return True on success. False otherwise.
+     */
+    boolean deleteCniConfiguration(DeleteCniConfigurationCmd cmd);
     /**
      * Search registered key pairs for the logged in user.
      *

api/src/main/java/com/cloud/template/TemplateApiService.java

@@ -62,10 +62,23 @@ public interface TemplateApiService {
     VirtualMachineTemplate prepareTemplate(long templateId, long zoneId, Long storageId);
 
 
+    /**
+     * Detach ISO from VM
+     * @param vmId id of the VM
+     * @param isoId id of the ISO (when passed). If it is not passed, it will get it from user_vm table
+     * @param extraParams forced, isVirtualRouter
+     * @return true when operation succeeds, false if not
+     */
+    boolean detachIso(long vmId, Long isoId, Boolean... extraParams);
 
-    boolean detachIso(long vmId, boolean forced);
-
-    boolean attachIso(long isoId, long vmId, boolean forced);
+    /**
+     * Attach ISO to a VM
+     * @param isoId id of the ISO to attach
+     * @param vmId id of the VM to attach the ISO to
+     * @param extraParams: forced, isVirtualRouter
+     * @return true when operation succeeds, false if not
+     */
+    boolean attachIso(long isoId, long vmId, Boolean... extraParams);
 
     /**
      * Deletes a template

api/src/main/java/com/cloud/template/VirtualMachineTemplate.java

@@ -145,6 +145,8 @@ public enum TemplateFilter {
 
     boolean isDeployAsIs();
 
+    boolean isForCks();
+
     Long getUserDataId();
 
     UserData.UserDataOverridePolicy getUserDataOverridePolicy();

api/src/main/java/com/cloud/user/UserData.java

@@ -29,4 +29,5 @@ public enum UserDataOverridePolicy {
     String getUserData();
 
     String getParams();
+    boolean isForCks();
 }