You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/en/blog/_posts/2024-02-22-diy-create-your-own-cloud-with-kubernetes-part-2/index.md
+21-18Lines changed: 21 additions & 18 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,13 +13,13 @@ In this article, we'll show you a few various virtualization technologies in Kub
13
13
14
14
We will talk about technologies such as KubeVirt, LINSTOR, and Kube-OVN.
15
15
16
-
But first, let's explain why virtual machines are needed for, and why can't you just use docker containers for building cloud?
16
+
But first, let's explain what virtual machines are needed for, and why can't you just use docker containers for building cloud?
17
17
The reason is that containers do not provide a sufficient level of isolation.
18
18
Although the situation improves year by year, we often encounter vulnerabilities that allow escaping the container sandbox and elevating privileges in the system.
19
19
20
-
On the other hand, Kubernetes was not originally designed to be multi-tenant system, meaning the basic usage pattern involves creating a separate Kubernetes cluster for every independent project and development team.
20
+
On the other hand, Kubernetes was not originally designed to be a multi-tenant system, meaning the basic usage pattern involves creating a separate Kubernetes cluster for every independent project and development team.
21
21
22
-
Virtual machines are the primary means of isolating tenants from each other in a cloud. In virtual machines, users are allowed to run any code with administrative rights, but this don't affect other tenants or the cloud system itself. In other words, virtual machines allows to achieve [hard multi-tenancy isolation](/docs/concepts/security/multi-tenancy/#isolation), and run in environments where tenants do not trust each other.
22
+
Virtual machines are the primary means of isolating tenants from each other in a cloud environment. In virtual machines, users can execute code and programs with administrative privilege, but this doesn't affect other tenants or the environment itself. In other words, virtual machines allow to achieve [hard multi-tenancy isolation](/docs/concepts/security/multi-tenancy/#isolation), and run in environments where tenants do not trust each other.
23
23
24
24
## Virtualization technologies in Kubernetes
25
25
@@ -30,16 +30,16 @@ are the most popular ones. But you should know that they work differently.
30
30
31
31
{{< figure src="kata-containers.svg" caption="A diagram showing how container isolation is ensured by running containers in virtual machines with Kata Containers" alt="A diagram showing how container isolation is ensured by running containers in virtual machines with Kata Containers" >}}
32
32
33
-
**KubeVirt** allows to run of traditional virtual machines using the Kubernetes API. KubeVirt virtual machines are run as regular linux processes in containers. In other words, in KubeVirt a container used simple as a sandbox for running virtual machine (QEMU) processes.
34
-
This can be clearly seen by looking at how live migration of virtual machines is implemented in KubeVirt. When migration is needed, the virtual machine moved from one container to another.
33
+
**KubeVirt** allows running traditional virtual machines using the Kubernetes API. KubeVirt virtual machines are run as regular linux processes in containers. In other words, in KubeVirt, a container is used as a sandbox for running virtual machine (QEMU) processes.
34
+
This can be clearly seen in the figure below, by looking at how live migration of virtual machines is implemented in KubeVirt. When migration is needed, the virtual machine moves from one container to another.
35
35
36
36
{{< figure src="kubevirt-migration.svg" caption="A diagram showing live migration of a virtual machine from one container to another in KubeVirt" alt="A diagram showing live migration of a virtual machine from one container to another in KubeVirt" >}}
37
37
38
38
There is also an alternative project - [Virtink](https://github.com/smartxworks/virtink), which implements lightweight virtualization using [Cloud-Hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) and is initially focused on running virtual Kubernetes clusters using the Cluster API.
39
39
40
40
Considering our goals, we decided to use KubeVirt as the most popular project in this area. Besides we have extensive expertise and already made a lot of contributions to KubeVirt.
41
41
42
-
KubeVirt is [easy to install](https://kubevirt.io/user-guide/operations/installation/) and out-of-box allows you to run virtual machines using [containerDisk](https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#containerdisk) feature - this allows you to store and distribute VM images directly as OCI images from container image registry.
42
+
KubeVirt is [easy to install](https://kubevirt.io/user-guide/operations/installation/) and allows you to run virtual machines out-of-the-box using [containerDisk](https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#containerdisk) feature - this allows you to store and distribute VM images directly as OCI images from container image registry.
43
43
Virtual machines with containerDisk are well suited for creating Kubernetes worker nodes and other VMs that do not require state persistence.
44
44
45
45
For managing persistent data, KubeVirt offers a separate tool, Containerized Data Importer (CDI). It allows for cloning PVCs and populating them with data from base images. The CDI is necessary if you want to automatically provision persistent volumes for your virtual machines, and it is also required for the KubeVirt CSI Driver, which is used to handle persistent volumes claims from tenant Kubernetes clusters.
@@ -50,13 +50,13 @@ But at first, you have to decide where and how you will store these data.
50
50
51
51
With the introduction of the CSI (Container Storage Interface), a wide range of technologies that integrate with Kubernetes has become available.
52
52
In fact, KubeVirt fully utilizes the CSI interface, aligning the choice of storage for virtualization closely with the choice of storage for Kubernetes itself.
53
-
However, there are nuances, which you need to consider. Unlike containers, which typically use a standard filesystem, block devices are more efficient for virtual machine.
53
+
However, there are nuances, which you need to consider. Unlike containers, which typically use a standard filesystem, block devices are more efficient for virtual machine.
54
54
55
-
Although the CSI interface in Kubernetes allows the request of both types of volumes either a filesystem or block devices—it's important to verify that your storage backend supports this.
55
+
Although the CSI interface in Kubernetes allows the request of both types of volumes: filesystems and block devices, it's important to verify that your storage backend supports this.
56
56
57
-
Using block devices for virtual machines eliminates the need for an additional abstraction layer, such as a filesystem, that makes it more performant and in most cases enables the use of the ReadWriteMany mode. This mode allows concurrent access to the volume from multiple nodes, which is a critical feature for enabling the live migration of virtual machines in KubeVirt.
57
+
Using block devices for virtual machines eliminates the need for an additional abstraction layer, such as a filesystem, that makes it more performant and in most cases enables the use of the _ReadWriteMany_ mode. This mode allows concurrent access to the volume from multiple nodes, which is a critical feature for enabling the live migration of virtual machines in KubeVirt.
58
58
59
-
The storage system can be external or internal (in the case of hyper-converged infrastructure). Using external storage in many cases makes the whole system more stable, as your data is stored on separately from compute nodes.
59
+
The storage system can be external or internal (in the case of hyper-converged infrastructure). Using external storage in many cases makes the whole system more stable, as your data is stored separately from compute nodes.
60
60
61
61
{{< figure src="storage-external.svg" caption="A diagram showing external data storage communication with the compute nodes" alt="A diagram showing external data storage communication with the compute nodes" >}}
62
62
@@ -68,22 +68,25 @@ On the other hand, hyper-converged systems are often implemented using local sto
68
68
69
69
{{< figure src="storage-clustered.svg" caption="A diagram showing clustered data storage running on the compute nodes" alt="A diagram showing clustered data storage running on the compute nodes" >}}
70
70
71
-
A hyper-converged system has its advantages, e.g. data locality: when your data is stored locally, access to such data is faster, but there are disadvantages as such a system is usually more difficult to manage and maintain.
71
+
A hyper-converged system has its advantages. For example, data locality: when your data is stored locally, access to such data is faster. But there are disadvantages as such a system is usually more difficult to manage and maintain.
72
72
73
73
At Ænix, we wanted to provide a ready-to-use solution that could be used without the need to purchase and setup an additional external storage, and that was optimal in terms of speed and resource utilization. LINSTOR became that solution.
74
74
The time-tested and industry-popular technologies such as LVM and ZFS as backend gives confidence that data is securely stored. DRBD-based replication is incredible fast and consumes a small amount of computing resources.
75
75
76
76
For installing LINSTOR in Kubernetes, there is the Piraeus project, which already provides a ready-made block storage to use with KubeVirt.
77
77
78
-
> **Note**: In case you are using Talos Linux, as we described in the [previous article](/blog/2024/01/22/diy-create-your-own-cloud-with-kubernetes-part-1/), you will need to enable the necessary kernel modules in advance, and configure piraeus as described in the [instruction](https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/how-to/talos.md).
78
+
{{< note >}}
79
+
In case you are using Talos Linux, as we described in the [previous article](/blog/2024/01/22/diy-create-your-own-cloud-with-kubernetes-part-1/), you will need to enable the necessary kernel modules in advance, and configure piraeus as described in the [instruction](https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/how-to/talos.md).
80
+
{{< /note >}}
81
+
79
82
80
83
## Networking for Kubernetes VMs
81
84
82
-
Despite having the similar interface - CNI. The network architecture in Kubernetes is actually more complex and typically consists of many independent components that are not directly connected to each other. In fact you can split Kubernetes networking to four layers, which are described below.
85
+
Despite having the similar interface - CNI, The network architecture in Kubernetes is actually more complex and typically consists of many independent components that are not directly connected to each other. In fact you can split Kubernetes networking into four layers, which are described below.
83
86
84
87
### Node Network (Data Center Network)
85
88
86
-
The network through which nodes are interconnected with each other. This network is usually not managed by Kubernetes, but it is an important because without it, nothing would work. In practice, the bare metal infrastructure usually have more than one of such networks e.g. one for node-to-node communication, second for storage replication, third for external access, etc.
89
+
The network through which nodes are interconnected with each other. This network is usually not managed by Kubernetes, but it is an important one because, without it, nothing would work. In practice, the bare metal infrastructure usually has more than one of such networks e.g. one for node-to-node communication, second for storage replication, third for external access, etc.
87
90
88
91
{{< figure src="net-nodes.svg" caption="A diagram showing the role of the node network (data center network) on the Kubernetes networking scheme" alt="A diagram showing the role of the node network (data center network) on the Kubernetes networking scheme" >}}
89
92
@@ -97,9 +100,9 @@ This is the network provided by your CNI plugin. The task of the CNI plugin is t
97
100
98
101
In practice, your cluster can have several CNI plugins managed by [Multus](https://github.com/k8snetworkplumbingwg/multus-cni). This approach is often used in virtualization solutions based on KubeVirt - [Rancher](https://www.rancher.com/) and [OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift/virtualization). The primary CNI plugin is used for integration with Kubernetes services, while additional CNI plugins are used to implement private networks (VPC) and integration with the physical networks of your data center.
99
102
100
-
The [default CNI-plugins](https://github.com/containernetworking/plugins/tree/main/plugins) can be used to connect bridges or physical interfaces. Additionally, there are specialized plugins such as [macvtap-cni](https://github.com/kubevirt/macvtap-cni) which designed to provide a more performance.
103
+
The [default CNI-plugins](https://github.com/containernetworking/plugins/tree/main/plugins) can be used to connect bridges or physical interfaces. Additionally, there are specialized plugins such as [macvtap-cni](https://github.com/kubevirt/macvtap-cni) which are designed to provide more performance.
101
104
102
-
One additional aspect to keep in mind when running virtual machines in Kubernetes is the need for IPAM (IP Address Management), especially for secondary interfaces provided by Multus. This is commonly managed by a DHCP server operating within your infrastructure. Additionally, the allocation of MAC addresses for virtual machines, that can be managed by [Kubemacpool](https://github.com/k8snetworkplumbingwg/kubemacpool).
105
+
One additional aspect to keep in mind when running virtual machines in Kubernetes is the need for IPAM (IP Address Management), especially for secondary interfaces provided by Multus. This is commonly managed by a DHCP server operating within your infrastructure. Additionally, the allocation of MAC addresses for virtual machines can be managed by [Kubemacpool](https://github.com/k8snetworkplumbingwg/kubemacpool).
103
106
104
107
Although in our platform, we decided to go another way and fully rely on [Kube-OVN](https://www.kube-ovn.io/). This CNI plugin is based on OVN (Open Virtual Network) which was originally developed for OpenStack and it provides a complete network solution for virtual machines in Kubernetes, features Custom Resources for managing IPs and MAC addresses, supports live migration with preserving IP addresses between the nodes, and enables the creation of VPCs for physical network separation between tenants.
105
108
@@ -116,15 +119,15 @@ The same approach is also commonly used with virtual machines in clouds despite
116
119
117
120
118
121
The implementation of the services network in Kubernetes is handled by the services network plugin, The standard implementation is called **kube-proxy** and is used in most clusters.
119
-
But nowadays this functionality might be provided as part of the CNI plugin. The most advanced implementation is offered by the [Cilium](https://cilium.io/) project, which can be run in kube-proxy replacement mode.
122
+
But nowadays, this functionality might be provided as part of the CNI plugin. The most advanced implementation is offered by the [Cilium](https://cilium.io/) project, which can be run in kube-proxy replacement mode.
120
123
121
124
Cilium is based on the eBPF technology, which allows for efficient offloading of the Linux networking stack, thereby improving performance and security compared to traditional methods based on iptables.
122
125
123
126
In practice, Cilium and Kube-OVN can be easily [integrated](https://kube-ovn.readthedocs.io/zh-cn/stable/en/advance/with-cilium/) to provide a unified solution that offers seamless multi-tenant networking for virtual machines, as well as advanced network policies and combined services network functionality.
124
127
125
128
### External Traffic Load Balancer
126
129
127
-
At this stage, you already have everything need to run virtual machines in Kubernetes. But there is actually one more thing.
130
+
At this stage, you already have everything needed to run virtual machines in Kubernetes. But there is actually one more thing.
128
131
You still need to access your services from outside your cluster, and an external load balancer will help you with organizing this.
129
132
130
133
For bare metal Kubernetes clusters, there are several load balancers available: [MetalLB](https://metallb.universe.tf/), [kube-vip](https://kube-vip.io/), [LoxiLB](https://www.loxilb.io/), also [Cilium](https://docs.cilium.io/en/latest/network/lb-ipam/) and [Kube-OVN](https://kube-ovn.readthedocs.io/zh-cn/latest/en/guide/loadbalancer-service/) provides built-in implementation.
0 commit comments