Skip to content

Commit 2b15a67

Browse files
kvapsGauravpadam
andcommitted
Apply suggestions from code review
Co-authored-by: Gaurav Padam <[email protected]>
1 parent 791435e commit 2b15a67

File tree

1 file changed

+21
-18
lines changed
  • content/en/blog/_posts/2024-02-22-diy-create-your-own-cloud-with-kubernetes-part-2

1 file changed

+21
-18
lines changed

content/en/blog/_posts/2024-02-22-diy-create-your-own-cloud-with-kubernetes-part-2/index.md

Lines changed: 21 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,13 @@ In this article, we'll show you a few various virtualization technologies in Kub
1313

1414
We will talk about technologies such as KubeVirt, LINSTOR, and Kube-OVN.
1515

16-
But first, let's explain why virtual machines are needed for, and why can't you just use docker containers for building cloud?
16+
But first, let's explain what virtual machines are needed for, and why can't you just use docker containers for building cloud?
1717
The reason is that containers do not provide a sufficient level of isolation.
1818
Although the situation improves year by year, we often encounter vulnerabilities that allow escaping the container sandbox and elevating privileges in the system.
1919

20-
On the other hand, Kubernetes was not originally designed to be multi-tenant system, meaning the basic usage pattern involves creating a separate Kubernetes cluster for every independent project and development team.
20+
On the other hand, Kubernetes was not originally designed to be a multi-tenant system, meaning the basic usage pattern involves creating a separate Kubernetes cluster for every independent project and development team.
2121

22-
Virtual machines are the primary means of isolating tenants from each other in a cloud. In virtual machines, users are allowed to run any code with administrative rights, but this don't affect other tenants or the cloud system itself. In other words, virtual machines allows to achieve [hard multi-tenancy isolation](/docs/concepts/security/multi-tenancy/#isolation), and run in environments where tenants do not trust each other.
22+
Virtual machines are the primary means of isolating tenants from each other in a cloud environment. In virtual machines, users can execute code and programs with administrative privilege, but this doesn't affect other tenants or the environment itself. In other words, virtual machines allow to achieve [hard multi-tenancy isolation](/docs/concepts/security/multi-tenancy/#isolation), and run in environments where tenants do not trust each other.
2323

2424
## Virtualization technologies in Kubernetes
2525

@@ -30,16 +30,16 @@ are the most popular ones. But you should know that they work differently.
3030

3131
{{< figure src="kata-containers.svg" caption="A diagram showing how container isolation is ensured by running containers in virtual machines with Kata Containers" alt="A diagram showing how container isolation is ensured by running containers in virtual machines with Kata Containers" >}}
3232

33-
**KubeVirt** allows to run of traditional virtual machines using the Kubernetes API. KubeVirt virtual machines are run as regular linux processes in containers. In other words, in KubeVirt a container used simple as a sandbox for running virtual machine (QEMU) processes.
34-
This can be clearly seen by looking at how live migration of virtual machines is implemented in KubeVirt. When migration is needed, the virtual machine moved from one container to another.
33+
**KubeVirt** allows running traditional virtual machines using the Kubernetes API. KubeVirt virtual machines are run as regular linux processes in containers. In other words, in KubeVirt, a container is used as a sandbox for running virtual machine (QEMU) processes.
34+
This can be clearly seen in the figure below, by looking at how live migration of virtual machines is implemented in KubeVirt. When migration is needed, the virtual machine moves from one container to another.
3535

3636
{{< figure src="kubevirt-migration.svg" caption="A diagram showing live migration of a virtual machine from one container to another in KubeVirt" alt="A diagram showing live migration of a virtual machine from one container to another in KubeVirt" >}}
3737

3838
There is also an alternative project - [Virtink](https://github.com/smartxworks/virtink), which implements lightweight virtualization using [Cloud-Hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor) and is initially focused on running virtual Kubernetes clusters using the Cluster API.
3939

4040
Considering our goals, we decided to use KubeVirt as the most popular project in this area. Besides we have extensive expertise and already made a lot of contributions to KubeVirt.
4141

42-
KubeVirt is [easy to install](https://kubevirt.io/user-guide/operations/installation/) and out-of-box allows you to run virtual machines using [containerDisk](https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#containerdisk) feature - this allows you to store and distribute VM images directly as OCI images from container image registry.
42+
KubeVirt is [easy to install](https://kubevirt.io/user-guide/operations/installation/) and allows you to run virtual machines out-of-the-box using [containerDisk](https://kubevirt.io/user-guide/virtual_machines/disks_and_volumes/#containerdisk) feature - this allows you to store and distribute VM images directly as OCI images from container image registry.
4343
Virtual machines with containerDisk are well suited for creating Kubernetes worker nodes and other VMs that do not require state persistence.
4444

4545
For managing persistent data, KubeVirt offers a separate tool, Containerized Data Importer (CDI). It allows for cloning PVCs and populating them with data from base images. The CDI is necessary if you want to automatically provision persistent volumes for your virtual machines, and it is also required for the KubeVirt CSI Driver, which is used to handle persistent volumes claims from tenant Kubernetes clusters.
@@ -50,13 +50,13 @@ But at first, you have to decide where and how you will store these data.
5050

5151
With the introduction of the CSI (Container Storage Interface), a wide range of technologies that integrate with Kubernetes has become available.
5252
In fact, KubeVirt fully utilizes the CSI interface, aligning the choice of storage for virtualization closely with the choice of storage for Kubernetes itself.
53-
However, there are nuances, which you need to consider. Unlike containers, which typically use a standard filesystem, block devices are more efficient for virtual machine .
53+
However, there are nuances, which you need to consider. Unlike containers, which typically use a standard filesystem, block devices are more efficient for virtual machine.
5454

55-
Although the CSI interface in Kubernetes allows the request of both types of volumes either a filesystem or block devicesit's important to verify that your storage backend supports this.
55+
Although the CSI interface in Kubernetes allows the request of both types of volumes: filesystems and block devices, it's important to verify that your storage backend supports this.
5656

57-
Using block devices for virtual machines eliminates the need for an additional abstraction layer, such as a filesystem, that makes it more performant and in most cases enables the use of the ReadWriteMany mode. This mode allows concurrent access to the volume from multiple nodes, which is a critical feature for enabling the live migration of virtual machines in KubeVirt.
57+
Using block devices for virtual machines eliminates the need for an additional abstraction layer, such as a filesystem, that makes it more performant and in most cases enables the use of the _ReadWriteMany_ mode. This mode allows concurrent access to the volume from multiple nodes, which is a critical feature for enabling the live migration of virtual machines in KubeVirt.
5858

59-
The storage system can be external or internal (in the case of hyper-converged infrastructure). Using external storage in many cases makes the whole system more stable, as your data is stored on separately from compute nodes.
59+
The storage system can be external or internal (in the case of hyper-converged infrastructure). Using external storage in many cases makes the whole system more stable, as your data is stored separately from compute nodes.
6060

6161
{{< figure src="storage-external.svg" caption="A diagram showing external data storage communication with the compute nodes" alt="A diagram showing external data storage communication with the compute nodes" >}}
6262

@@ -68,22 +68,25 @@ On the other hand, hyper-converged systems are often implemented using local sto
6868

6969
{{< figure src="storage-clustered.svg" caption="A diagram showing clustered data storage running on the compute nodes" alt="A diagram showing clustered data storage running on the compute nodes" >}}
7070

71-
A hyper-converged system has its advantages, e.g. data locality: when your data is stored locally, access to such data is faster, but there are disadvantages as such a system is usually more difficult to manage and maintain.
71+
A hyper-converged system has its advantages. For example, data locality: when your data is stored locally, access to such data is faster. But there are disadvantages as such a system is usually more difficult to manage and maintain.
7272

7373
At Ænix, we wanted to provide a ready-to-use solution that could be used without the need to purchase and setup an additional external storage, and that was optimal in terms of speed and resource utilization. LINSTOR became that solution.
7474
The time-tested and industry-popular technologies such as LVM and ZFS as backend gives confidence that data is securely stored. DRBD-based replication is incredible fast and consumes a small amount of computing resources.
7575

7676
For installing LINSTOR in Kubernetes, there is the Piraeus project, which already provides a ready-made block storage to use with KubeVirt.
7777

78-
> **Note**: In case you are using Talos Linux, as we described in the [previous article](/blog/2024/01/22/diy-create-your-own-cloud-with-kubernetes-part-1/), you will need to enable the necessary kernel modules in advance, and configure piraeus as described in the [instruction](https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/how-to/talos.md).
78+
{{< note >}}
79+
In case you are using Talos Linux, as we described in the [previous article](/blog/2024/01/22/diy-create-your-own-cloud-with-kubernetes-part-1/), you will need to enable the necessary kernel modules in advance, and configure piraeus as described in the [instruction](https://github.com/piraeusdatastore/piraeus-operator/blob/v2/docs/how-to/talos.md).
80+
{{< /note >}}
81+
7982

8083
## Networking for Kubernetes VMs
8184

82-
Despite having the similar interface - CNI. The network architecture in Kubernetes is actually more complex and typically consists of many independent components that are not directly connected to each other. In fact you can split Kubernetes networking to four layers, which are described below.
85+
Despite having the similar interface - CNI, The network architecture in Kubernetes is actually more complex and typically consists of many independent components that are not directly connected to each other. In fact you can split Kubernetes networking into four layers, which are described below.
8386

8487
### Node Network (Data Center Network)
8588

86-
The network through which nodes are interconnected with each other. This network is usually not managed by Kubernetes, but it is an important because without it, nothing would work. In practice, the bare metal infrastructure usually have more than one of such networks e.g. one for node-to-node communication, second for storage replication, third for external access, etc.
89+
The network through which nodes are interconnected with each other. This network is usually not managed by Kubernetes, but it is an important one because, without it, nothing would work. In practice, the bare metal infrastructure usually has more than one of such networks e.g. one for node-to-node communication, second for storage replication, third for external access, etc.
8790

8891
{{< figure src="net-nodes.svg" caption="A diagram showing the role of the node network (data center network) on the Kubernetes networking scheme" alt="A diagram showing the role of the node network (data center network) on the Kubernetes networking scheme" >}}
8992

@@ -97,9 +100,9 @@ This is the network provided by your CNI plugin. The task of the CNI plugin is t
97100

98101
In practice, your cluster can have several CNI plugins managed by [Multus](https://github.com/k8snetworkplumbingwg/multus-cni). This approach is often used in virtualization solutions based on KubeVirt - [Rancher](https://www.rancher.com/) and [OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift/virtualization). The primary CNI plugin is used for integration with Kubernetes services, while additional CNI plugins are used to implement private networks (VPC) and integration with the physical networks of your data center.
99102

100-
The [default CNI-plugins](https://github.com/containernetworking/plugins/tree/main/plugins) can be used to connect bridges or physical interfaces. Additionally, there are specialized plugins such as [macvtap-cni](https://github.com/kubevirt/macvtap-cni) which designed to provide a more performance.
103+
The [default CNI-plugins](https://github.com/containernetworking/plugins/tree/main/plugins) can be used to connect bridges or physical interfaces. Additionally, there are specialized plugins such as [macvtap-cni](https://github.com/kubevirt/macvtap-cni) which are designed to provide more performance.
101104

102-
One additional aspect to keep in mind when running virtual machines in Kubernetes is the need for IPAM (IP Address Management), especially for secondary interfaces provided by Multus. This is commonly managed by a DHCP server operating within your infrastructure. Additionally, the allocation of MAC addresses for virtual machines, that can be managed by [Kubemacpool](https://github.com/k8snetworkplumbingwg/kubemacpool).
105+
One additional aspect to keep in mind when running virtual machines in Kubernetes is the need for IPAM (IP Address Management), especially for secondary interfaces provided by Multus. This is commonly managed by a DHCP server operating within your infrastructure. Additionally, the allocation of MAC addresses for virtual machines can be managed by [Kubemacpool](https://github.com/k8snetworkplumbingwg/kubemacpool).
103106

104107
Although in our platform, we decided to go another way and fully rely on [Kube-OVN](https://www.kube-ovn.io/). This CNI plugin is based on OVN (Open Virtual Network) which was originally developed for OpenStack and it provides a complete network solution for virtual machines in Kubernetes, features Custom Resources for managing IPs and MAC addresses, supports live migration with preserving IP addresses between the nodes, and enables the creation of VPCs for physical network separation between tenants.
105108

@@ -116,15 +119,15 @@ The same approach is also commonly used with virtual machines in clouds despite
116119

117120

118121
The implementation of the services network in Kubernetes is handled by the services network plugin, The standard implementation is called **kube-proxy** and is used in most clusters.
119-
But nowadays this functionality might be provided as part of the CNI plugin. The most advanced implementation is offered by the [Cilium](https://cilium.io/) project, which can be run in kube-proxy replacement mode.
122+
But nowadays, this functionality might be provided as part of the CNI plugin. The most advanced implementation is offered by the [Cilium](https://cilium.io/) project, which can be run in kube-proxy replacement mode.
120123

121124
Cilium is based on the eBPF technology, which allows for efficient offloading of the Linux networking stack, thereby improving performance and security compared to traditional methods based on iptables.
122125

123126
In practice, Cilium and Kube-OVN can be easily [integrated](https://kube-ovn.readthedocs.io/zh-cn/stable/en/advance/with-cilium/) to provide a unified solution that offers seamless multi-tenant networking for virtual machines, as well as advanced network policies and combined services network functionality.
124127

125128
### External Traffic Load Balancer
126129

127-
At this stage, you already have everything need to run virtual machines in Kubernetes. But there is actually one more thing.
130+
At this stage, you already have everything needed to run virtual machines in Kubernetes. But there is actually one more thing.
128131
You still need to access your services from outside your cluster, and an external load balancer will help you with organizing this.
129132

130133
For bare metal Kubernetes clusters, there are several load balancers available: [MetalLB](https://metallb.universe.tf/), [kube-vip](https://kube-vip.io/), [LoxiLB](https://www.loxilb.io/), also [Cilium](https://docs.cilium.io/en/latest/network/lb-ipam/) and [Kube-OVN](https://kube-ovn.readthedocs.io/zh-cn/latest/en/guide/loadbalancer-service/) provides built-in implementation.

0 commit comments

Comments
 (0)