Skip to content

Commit de1e17e

Browse files
YuryHrytsukYuryHrytsuk
authored
Kubernetes: add local storage (#1100)
* Update longhorn README Document how to perform (kubernetes) node maintenance * Update Longhorn README: disks config and maintenance * Kubernets add local storage Use topolvm as the most mature local storage csi. * Update longhorn readme
1 parent 0387102 commit de1e17e

File tree

3 files changed

+177
-1
lines changed

3 files changed

+177
-1
lines changed

‎charts/longhorn/README.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,15 @@ Source:
2727

2828
### How to configure disks for LH
2929

30-
As of now, we follow the same approach we use for `/docker` folder (via ansible playbook) but we use `/longhorn` folder name
30+
Manual configuration performed (to be moved to ansible)
31+
1. Create partition on the disk
32+
* e.g. via using `fdisk` https://phoenixnap.com/kb/linux-create-partition
33+
2. Format partition as XFS
34+
* `sudo mkfs.xfs -f /dev/sda1`
35+
3. Mount partition `sudo mount -t xfs /dev/sda1 /longhorn`
36+
4. Persist mount in `/etc/fstab` by adding line
37+
* `UUID=<partition's uuid> /longhorn xfs pquota 0 0`
38+
* UUID can be received from `lsblk -f`
3139

3240
Issue asking LH to clearly document requirements: https://github.com/longhorn/longhorn/issues/11125
3341

@@ -54,3 +62,22 @@ Insights into LH's performance:
5462

5563
Resource requirements:
5664
* https://github.com/longhorn/longhorn/issues/1691
65+
66+
### (Kubernetes) Node maintenance
67+
68+
https://longhorn.io/docs/1.8.1/maintenance/maintenance/
69+
70+
Note: you can use Longhorn GUI to perform some operations
71+
72+
### Zero downtime updating longhorn disks (procedure)
73+
Notes:
74+
* Update one node at a time so that other nodes can still serve data
75+
76+
1. Go to LH GUI and select a Node
77+
1. Disable scheduling
78+
2. Request eviction
79+
1. Remove disk from the node
80+
* If remove icon is disabled, disable eviction on disk to enable the remove button
81+
2. Perform disks updates on the node
82+
3. Make sure LH didn't pick up wrongly configured disk in the meantime and remove the wrong disk if it did so
83+
4. Wait till LH automatically adds the disk to the Node

‎charts/topolvm/README.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
## topolvm components and architecture
2+
See diagram https://github.com/topolvm/topolvm/blob/topolvm-chart-v15.5.5/docs/design.md
3+
4+
## Preqrequisites
5+
`topolvm` does not automatically creates Volume Groups (specified in device-classes). This needs to be configured additionally (e.g. manually, via ansible, ...)
6+
7+
Manual example (Ubuntu 22.04):
8+
1. Create partition to use later (`sudo fdisk /dev/sda`)
9+
2. Create PV (`sudo pvcreate /dev/sda2`)
10+
* Prerequisite: `sudo apt install lvm2`
11+
3. Create Volume group (`sudo vgcreate topovg-sdd /dev/sda2`)
12+
* Note: Volume group's name must correspond to the setting of `volume-group` inside `lvmd.deviceClasses`
13+
4. Check volume group (`sudo vgdisplay`)
14+
15+
Source: https://github.com/topolvm/topolvm/blob/topolvm-chart-v15.5.5/docs/getting-started.md#prerequisites
16+
17+
## Deleting PV(C)s with `retain` reclaim policy
18+
1. Delete release (e.g. helm uninstall -n test test)
19+
2. Find LogicalVolume CR (`kubectl get logicalvolumes.topolvm.io`
20+
3. Delete LogicalVolume CR (`kubectl delete logicalvolumes.topolvm.io <lv-name>`)
21+
4. Delete PV (`kubectl delete PV <pv-name>`)
22+
23+
## Backup / Snapshotting
24+
1. Only possible while using thin provisioning
25+
2. We use thick (non-thin provisioned) volumes --> no snapshot support
26+
27+
Track this feature request for changes https://github.com/topolvm/topolvm/issues/1070
28+
29+
Note: there might be alternative not documented ways (e.g. via Velero)
30+
31+
## Resizing PVs
32+
1. Update storage capacity in configuration
33+
2. Deploy changes
34+
35+
Note: storage size can only be increased. Otherwise, one gets `Forbidden: field can not be less than previous value` error
36+
37+
## Node maintenance
38+
39+
Read https://github.com/topolvm/topolvm/blob/topolvm-chart-v15.5.5/docs/node-maintenance.md
40+
41+
## Using topolvm. Notes
42+
* `topolvm` may not work with pods that define `spec.nodeName` Use node affinity instead
43+
https://github.com/topolvm/topolvm/blob/main/docs/faq.md#the-pod-does-not-start-when-nodename-is-specified-in-the-pod-spec

‎charts/topolvm/values.yaml.gotmpl‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
lvmd:
2+
# set up lvmd service with DaemonSet
3+
managed: true
4+
5+
# device classes (VGs) need to be created outside of topolvm (e.g. manually, via ansible, ...)
6+
deviceClasses:
7+
- name: ssd
8+
volume-group: topovg-sdd
9+
default: true
10+
spare-gb: 5
11+
12+
storageClasses:
13+
- name: {{ .Values.topolvmStorageClassName }}
14+
storageClass:
15+
# Want to use non-default device class?
16+
# See configuration example in
17+
# https://github.com/topolvm/topolvm/blob/topolvm-chart-v15.5.5/docs/snapshot-and-restore.md#set-up-a-storage-class
18+
19+
fsType: xfs
20+
isDefaultClass: false
21+
# volumeBindingMode can be either WaitForFirstConsumer or Immediate. WaitForFirstConsumer is recommended because TopoLVM cannot schedule pods wisely if volumeBindingMode is Immediate.
22+
volumeBindingMode: WaitForFirstConsumer
23+
allowVolumeExpansion: true
24+
# NOTE: On removal requires manual clean up of PVs, LVMs
25+
# and Logical Volumes (CR logicalvolumes.topolvm.io).
26+
# Removal Logical Volume (CR) would clean up the LVM on the node,
27+
# but PV has still to be removed manually.
28+
# Read more: https://github.com/topolvm/topolvm/blob/topolvm-chart-v15.5.5/docs/advanced-setup.md#storageclass
29+
reclaimPolicy: Retain
30+
31+
resources:
32+
topolvm_node:
33+
requests:
34+
memory: 100Mi
35+
cpu: 100m
36+
limits:
37+
memory: 500Mi
38+
cpu: 500m
39+
40+
topolvm_controller:
41+
requests:
42+
memory: 50Mi
43+
cpu: 50m
44+
limits:
45+
memory: 200Mi
46+
cpu: 200m
47+
48+
lvmd:
49+
requests:
50+
memory: 100Mi
51+
cpu: 100m
52+
limits:
53+
memory: 500Mi
54+
cpu: 500m
55+
56+
csi_registrar:
57+
requests:
58+
cpu: 25m
59+
memory: 10Mi
60+
limits:
61+
cpu: 200m
62+
memory: 200Mi
63+
64+
csi_provisioner:
65+
requests:
66+
memory: 50Mi
67+
cpu: 50m
68+
limits:
69+
memory: 200Mi
70+
cpu: 200m
71+
72+
csi_resizer:
73+
requests:
74+
memory: 50Mi
75+
cpu: 50m
76+
limits:
77+
memory: 200Mi
78+
cpu: 200m
79+
80+
csi_snapshotter:
81+
requests:
82+
memory: 50Mi
83+
cpu: 50m
84+
limits:
85+
memory: 200Mi
86+
cpu: 200m
87+
88+
liveness_probe:
89+
requests:
90+
cpu: 25m
91+
memory: 10Mi
92+
limits:
93+
cpu: 200m
94+
memory: 200Mi
95+
96+
# https://github.com/topolvm/topolvm/blob/topolvm-chart-v15.5.5/docs/topolvm-scheduler.md
97+
scheduler:
98+
# start simple
99+
enabled: false
100+
101+
cert-manager:
102+
# start simple
103+
enabled: false
104+
105+
snapshot:
106+
enabled: true

0 commit comments

Comments
 (0)