You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Harvester now offers the capability to install a [Container Storage Interface (CSI)](https://kubernetes-csi.github.io/docs/introduction.html)in your Harvester cluster. This allows you to leverage external storage for the Virtual Machine's non-system data disk, allowing you to use different drivers tailored for specific needs, whether for performance optimization or seamless integration with your existing in-house storage solutions.
13
+
Harvester now supports provisioning of root volumes and data volumes using external [Container Storage Interface (CSI)](https://kubernetes-csi.github.io/docs/introduction.html)drivers. This enhancement allows you to select drivers that meet specific requirements, such as performance optimization or seamless integration with existing internal storage solutions.
14
14
15
-
:::note
15
+
:::important
16
+
17
+
Harvester has validated the following CSI drivers:
The validated CSI drivers have the following capabilities:
16
25
17
-
The Virtual Machine (VM) image provisioner in Harvester still relies on Longhorn. Before version 1.2.0, Harvester exclusively supported Longhorn for storing VM data and did not offer support for external storage as a destination for VM data.
26
+
| Storage Solution | VM Image | VM Rook Disk | VM Data Disk | Volume Export To VM Image | VM Template Generator | VM Live Migration | VM Snapshot | VM Backup |
For the Harvester functions to work well, the third-party CSI driver needs to have the following capabilities:
24
-
- Support expansion
25
-
- Support snapshot
26
-
- Support clone
27
-
- Support block device
28
-
- Support Read-Write-Many (RWX), for [Live Migration](../vm/live-migration.md)
37
+
To enable Harvester to function well, use CSI drivers that support the following capabilities:
38
+
- Volume expansion (online resizing)
39
+
- Snapshot creation (volume and virtual machine snapshots)
40
+
- Cloning (volume and virtual machine clones)
41
+
- Usage of Read-Write-Many (RWX) volumes for [Live Migration](../vm/live-migration.md)
29
42
30
43
## Create Harvester cluster
31
44
@@ -51,24 +64,192 @@ With the kubeconfig of the Harvester cluster, you can install the third-party CS
51
64
52
65
Before you can make use of Harvester's **Backup & Snapshot** features, you need to set up some essential configurations through the Harvester [csi-driver-config](../advanced/settings.md#csi-driver-config) setting. Follow these steps to make these configurations:
53
66
67
+
:::note
68
+
69
+
Backup currently only works with the Longhorn v1 Data Engine. If you are using other storage providers, you can skip the **Backup VolumeSnapshot Class Name** configuration.
70
+
71
+
For more information, see [VM Backup Compatibility](../../versioned_docs/version-v1.4/advanced/csidriver.md#vm-backup-compatibility).
72
+
73
+
:::
74
+
54
75
1. Login to the Harvester UI, then navigate to **Advanced** > **Settings**.
55
76
1. Find and select **csi-driver-config**, and then select **⋮** > **Edit Setting** to access the configuration options.
56
77
1. Set the **Provisioner** to the third-party CSI driver in the settings.
57
78
1. Next, Configure the **Volume Snapshot Class Name**. This setting points to the name of the `VolumeSnapshotClass` used for creating volume snapshots or VM snapshots.
Once the CSI driver is installed and the Harvester cluster is configured, an external storage solution can be used in tasks that involve storage management.
85
+
86
+
### Virtual Machine Image Creation
87
+
88
+
You can use an external storage solution to store and manage virtual machine images.
89
+
90
+
When [uploading a virtual machine image](../image/upload-image.md) using the Harvester UI (**Image > Create**), you must select the StorageClass for the external storage solution on the **Storage** tab. In the following example, the StorageClass is **nfs-csi**.
Your virtual machines can use root and data volumes in external storage.
101
+
102
+
When [creating a virtual machine](../vm/create-vm.md) using the Harvester UI (**Virtual Machine > Create**), you must perform the following actions on the **Volumes** tab:
103
+
104
+
- Select a virtual machine image stored in the external storage solution, and then configure the required settings.
After successfully configuring these settings, you can utilize the third-party StorageClass. You can apply the third-party StorageClass when creating an empty volume or adding a new block volume to a VM, enhancing your Harvester cluster's storage capabilities.
115
+
You can create volumes in your external storage solution.
64
116
65
-
With these configurations in place, your Harvester cluster is ready to make the most of the third-party storage integration.
117
+
When [creating a volume](../volume/create-volume.md) using the Harvester UI (**Volumes > Create**), you must perform the following actions:
You can now use the CDI API to create custom [storage profiles](https://github.com/kubevirt/containerized-data-importer/blob/main/doc/storageprofile.md) that simplify definition of data volumes. Storage profiles allow multiple data volumes to share the same provisioner settings.
129
+
130
+
The following is an example of an LVM storage profile:
131
+
132
+
```yaml
133
+
apiVersion: cdi.kubevirt.io/v1beta1
134
+
kind: StorageProfile
135
+
metadata:
136
+
name: lvm-node-1-striped
137
+
spec:
138
+
claimPropertySets:
139
+
- accessModes:
140
+
- ReadWriteOnce
141
+
volumeMode: Block
142
+
status:
143
+
claimPropertySets:
144
+
- accessModes:
145
+
- ReadWriteOnce
146
+
volumeMode: Block
147
+
cloneStrategy: snapshot
148
+
dataImportCronSourceFormat: pvc
149
+
provisioner: lvm.driver.harvesterhci.io
150
+
snapshotClass: lvm-snapshot
151
+
storageClass: lvm-node-1-striped
152
+
```
153
+
154
+
For more information, see [Storage Profiles](https://github.com/kubevirt/containerized-data-importer/blob/main/doc/storageprofile.md) in the CDI documentation.
155
+
156
+
You can define the above fields to override the default configuration showing on the status.
157
+
158
+
### Limitations
159
+
160
+
- Backup support is currently limited to Longhorn V1 Data Engine volumes. Harvester is unable to create backups of volumes in external storage.
161
+
162
+
- There is a limitation in the CDI which prevents Harvester from converting attached PVCs to virtual machine images. Before exporting a volume in external storage, ensure that the PVC is not attached to workloads. This prevents the resulting image from getting stuck in the *Exporting* state.
You can deploy the NFS CSI driver only when the NFS server is already installed and running.
171
+
172
+
If the server is already running, check the `squash` option. You must disable squashing of remote root users (`no_root_squash` or `no_all_squash`) because KubeVirt needs the QEMU UID/GID to ensure that the volume can be synced properly.
173
+
174
+
:::
175
+
176
+
1. Install the driver using the `csi-driver-nfs` Helm chart.
For more information about parameters, see [Driver Parameters: Storage Class Usage](https://github.com/kubernetes-csi/csi-driver-nfs/blob/master/docs/driver-parameters.md) in the Kubernetes NFS CSI Driver documentation.
185
+
```yaml
186
+
apiVersion: storage.k8s.io/v1
187
+
kind: StorageClass
188
+
metadata:
189
+
name: nfs-csi
190
+
provisioner: nfs.csi.k8s.io
191
+
parameters:
192
+
server: <your-nfs-server-ip>
193
+
share: <your-nfs-share>
194
+
# csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
Once created, you can use the StorageClass to create virtual machine images, root volumes, and data volumes.
70
205
71
206
## References
72
207
73
208
-[Use Rook Ceph External Storage with Harvester](https://harvesterhci.io/kb/use_rook_ceph_external_storage)
74
-
-[Using NetApp Storage on Harvester](https://harvesterhci.io/kb/install_netapp_trident_csi)
209
+
-[Using NetApp Storage on Harvester](https://harvesterhci.io/kb/install_netapp_trident_csi)
210
+
-[Third Party Storage Support](https://github.com/harvester/harvester/blob/master/enhancements/20250203-third-party-storage-support.md)
211
+
212
+
## Known Issues
213
+
214
+
### 1. Infinite Image Download Loop
215
+
216
+
The image download process loops endlessly when the StorageClass for the image uses the LVM CSI driver. This issue is related to the scratch volume, which is created by CDI and is used to temporarily store the image data. When the issue exists in your environment, you might find the following error messages in `importer-prime-xxx` pod logs:
217
+
218
+
```
219
+
E0418 01:59:51.843459 1 util.go:98] Unable to write file from dataReader: write /scratch/tmpimage: no space left on device
220
+
E0418 01:59:51.861235 1 data-processor.go:243] write /scratch/tmpimage: no space left on device
The message `no space left on device` indicates that the filesystem created using the scratch volume is not enough to store the image data. CDI creates the scratch volume based on the size of the target volume, but some space is lost to filesystem overhead. The default overhead value is `0.055` (equivalent to 5.5%), which is sufficient in most cases. However, if the image size is less than 1 GB and its virtual size is very close to the image size, the default overhead is likely to be insufficient.
240
+
241
+
The workaround is to increase the filesystem overhead to 20% using the following command:
The image should be downloaded once the filesystem overhead is increased.
248
+
249
+
:::note
250
+
251
+
Increasing the overhead value does not affect the image PVC size. The scratch volume is deleted after the image is imported.
252
+
253
+
:::
254
+
255
+
Related issue: [#7993](https://github.com/harvester/harvester/issues/7993) (See this [comment](https://github.com/harvester/harvester/issues/7993#issuecomment-2790260841).)
Copy file name to clipboardExpand all lines: docs/image/image-security.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,6 +19,12 @@ _Available as of v1.4.0_
19
19
20
20
Harvester allows you to encrypt and decrypt virtual machine images. The encryption mechanism utilizes the Linux kernel module dm_crypt and the command-line utility cryptsetup.
21
21
22
+
:::note
23
+
24
+
This feature only supports the Longhorn V1 Data Engine. You cannot encrypt and decrypt images that are stored in other storage solutions.
Copy file name to clipboardExpand all lines: docs/vm/backup-restore.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -26,6 +26,8 @@ VM backups are created from the **Virtual Machines** page. The VM backup volumes
26
26
27
27
A backup target must be set up. For more information, see [Configure Backup Target](#configure-backup-target). If the backup target has not been set, you’ll be prompted with a message to do so.
28
28
29
+
Backup support is currently limited to Longhorn V1 Data Engine volumes. Harvester is unable to create backups of volumes in external storage.
Copy file name to clipboardExpand all lines: docs/vm/create-vm.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -152,6 +152,7 @@ A disk can be made accessible via the following types:
152
152
| cd-rom | This type will expose the volume as a cd-rom drive to the VM. It is read-only by default. |
153
153
154
154
A volume's [StorageClass](../advanced/storageclass.md) can be specified when adding a new empty volume; for other volumes (such as VM images), the `StorageClass` is defined during image creation.
155
+
If you are using external storage, ensure that the correct **StorageClass** and **Volume Mode** are selected. For example, a volume with the **nfs-csi** StorageClass must use the **Filesystem** volume mode.
Copy file name to clipboardExpand all lines: docs/vm/edit-vm.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -49,6 +49,12 @@ For more details about the network implementation, please refer to the [Networki
49
49
50
50
You can add additional volumes to the VM after booting. You can also expand the size of the volume after shutting down the VM, click the VM and go to the `Volumes` tab, edit the size of the expanded volume. After restarting the VM and waiting for the resize to complete, your disk will automatically finish expanding.
51
51
52
+
:::note
53
+
54
+
If you are using external storage, ensure that the storage provider supports volume expansion before you resize volumes.
Copy file name to clipboardExpand all lines: docs/vm/live-migration.md
+1Lines changed: 1 addition & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,6 +23,7 @@ Live migration means moving a virtual machine to a different host without downti
23
23
- Live migration is not allowed when the virtual machine has any volume of the `CD-ROM` type. Such volumes should be ejected before live migration.
24
24
- Live migration is not allowed when the virtual machine has any volume of the `Container Disk` type. Such volumes should be removed before live migration.
25
25
- Live migration is not allowed when the virtual machine has any `PCIDevice` passthrough enabled. Such devices need to be removed before live migration.
26
+
- Live migration is not allowed when the volumeAccessMode of any volume in the virtual machine is `ReadWriteOnce`. Such volumes should be removed before live migration.
Copy file name to clipboardExpand all lines: docs/volume/export-volume.md
+6Lines changed: 6 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,6 +13,12 @@ description: Export volume to image from the Volume page.
13
13
14
14
You can select and export an existing volume to an image by following the steps below:
15
15
16
+
:::note
17
+
18
+
If the volume is in external storage, ensure that it is not attached to workloads (virtual machines and pods) before starting the export process. This prevents the resulting image from getting stuck in the *Exporting* state.
19
+
20
+
:::
21
+
16
22
1. Click the `⋮` button and select the `Export Image` option.
0 commit comments