Skip to content

Commit ec5f2ab

Browse files
japokornrichm
japokorn
and
richm
authored
LVMVDO support (#181)
* LVMVDO support - added support for LVM VDO - new accepted pool options: - vdo_compression - vdo_deduplication - vdo_size - added tests for LVM VDO - added support for COPR repositories * remove extraneous blank lines at end of file remove extraneous blank lines at end of file * use pipefail; reduce line length use pipefail; reduce line length * fix line length issues; use multiline when fix line length issues; use multiline `when` * make sure all tasks are named make sure all tasks are named * fix line length using multiline when; remove extra blank lines fix line length using multiline `when`; remove extra blank lines * wrap lines to shorten line length wrap lines to shorten line length Co-authored-by: Richard Megginson <richm@stanfordalumni.org>
1 parent b3b4561 commit ec5f2ab

13 files changed

+301
-6
lines changed

‎README.md‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,8 @@ must contain only a single item.
9494
##### `size`
9595
The `size` specifies the size of the file system. The format for this is intended to
9696
be human-readable, e.g.: "10g", "50 GiB".
97+
When using `compression` or `deduplication`, `size` can be set higher than actual available space,
98+
e.g.: 3 times the size of the volume, based on duplicity and/or compressibility of stored data.
9799

98100
__NOTE__: The requested volume size may be reduced as necessary so the volume can
99101
fit in the available pool space, but only if the required reduction is
@@ -157,6 +159,30 @@ This integer specifies the LUKS key size (in bits).
157159
##### `encryption_luks_version`
158160
This integer specifies the LUKS version to use.
159161

162+
##### `deduplication`
163+
This specifies whether or not the Virtual Data Optimizer (VDO) will be used.
164+
When set, duplicate data stored on storage volume will be
165+
deduplicated resulting in more storage capacity.
166+
Can be used together with `compression` and `vdo_pool_size`.
167+
Volume has to be part of the LVM `storage_pool`.
168+
Limit one VDO `storage_volume` per `storage_pool`.
169+
Underlying volume has to be at least 9 GB (bare minimum is around 5 GiB).
170+
171+
##### `compression`
172+
This specifies whether or not the Virtual Data Optimizer (VDO) will be used.
173+
When set, data stored on storage volume will be
174+
compressed resulting in more storage capacity.
175+
Volume has to be part of the LVM `storage_pool`.
176+
Can be used together with `deduplication` and `vdo_pool_size`.
177+
Limit one VDO `storage_volume` per `storage_pool`.
178+
179+
##### `vdo_pool_size`
180+
When Virtual Data Optimizer (VDO) is used, this specifies the actual size the volume
181+
will take on the device. Virtual size of VDO volume is set by `size` parameter.
182+
`vdo_pool_size` format is intended to be human-readable,
183+
e.g.: "30g", "50GiB".
184+
Default value is equal to the size of the volume.
185+
160186
#### `storage_safe_mode`
161187
When true (the default), an error will occur instead of automatically removing existing devices and/or formatting.
162188

‎defaults/main.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,3 +56,7 @@ storage_volume_defaults:
5656
encryption_cipher: null
5757
encryption_key_size: null
5858
encryption_luks_version: null
59+
60+
compression: null
61+
deduplication: null
62+
vdo_pool_size: null

‎library/blivet.py‎

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -320,6 +320,8 @@ def required_packages(self):
320320
packages.extend(fmt.packages)
321321
if self._volume.get('encryption'):
322322
packages.extend(get_format('luks').packages)
323+
if self._volume.get('compression') or self._volume.get('deduplication'):
324+
packages.extend(['vdo', 'kmod-kvdo'])
323325
return packages
324326

325327
@property
@@ -645,10 +647,41 @@ def _create(self):
645647
fmt = self._get_format()
646648
trim_percent = (1.0 - float(parent.free_space / size)) * 100
647649
log.debug("size: %s ; %s", size, trim_percent)
648-
if size > parent.free_space:
650+
651+
create_vdo = self._volume['deduplication'] or self._volume['compression']
652+
653+
if create_vdo:
654+
pool_size = parent.free_space if size > parent.free_space else size
655+
if self._volume['vdo_pool_size']:
656+
pool_size = Size(self._volume['vdo_pool_size'])
657+
if pool_size > parent.free_space:
658+
if trim_percent > MAX_TRIM_PERCENT:
659+
raise BlivetAnsibleError("specified 'vdo_pool_size' for volume '%s' "
660+
"exceeds available space in pool '%s' (%s)" % (pool_size,
661+
parent.name,
662+
parent.free_space))
663+
else:
664+
log.info("adjusting %s size from %s to %s to fit in %s free space",
665+
self._volume['name'],
666+
size,
667+
parent.free_space,
668+
parent.name)
669+
pool_size = parent.free_space
670+
671+
try:
672+
vdopool = self._blivet.new_lv(name='vdopool', vdo_pool=True,
673+
parents=[parent], compression=self._volume['compression'],
674+
deduplication=self._volume['deduplication'],
675+
size=pool_size)
676+
except Exception as e:
677+
raise BlivetAnsibleError("failed to set up VDO pool '%s': %s" % (self._volume['name'], str(e)))
678+
679+
self._blivet.create_device(vdopool)
680+
elif size > parent.free_space:
649681
if trim_percent > MAX_TRIM_PERCENT:
650-
raise BlivetAnsibleError("specified size for volume '%s' exceeds available space in pool '%s' (%s)"
651-
% (size, parent.name, parent.free_space))
682+
raise BlivetAnsibleError("specified size for volume '%s' exceeds available space in pool '%s' (%s)" % (size,
683+
parent.name,
684+
parent.free_space))
652685
else:
653686
log.info("adjusting %s size from %s to %s to fit in %s free space",
654687
self._volume['name'],
@@ -658,8 +691,16 @@ def _create(self):
658691
size = parent.free_space
659692

660693
try:
661-
device = self._blivet.new_lv(name=self._volume['name'],
662-
parents=[parent], size=size, fmt=fmt)
694+
if create_vdo:
695+
device = self._blivet.new_lv(name=self._volume['name'], vdo_lv=create_vdo,
696+
parents=[vdopool if create_vdo else parent],
697+
size=size, fmt=fmt)
698+
else:
699+
# This is here for backwards compatibility. Until 8.4 blivet does not support
700+
# vdo_lv optional parameter
701+
device = self._blivet.new_lv(name=self._volume['name'],
702+
parents=[vdopool if create_vdo else parent],
703+
size=size, fmt=fmt)
663704
except Exception as e:
664705
raise BlivetAnsibleError("failed to set up volume '%s': %s" % (self._volume['name'], str(e)))
665706

@@ -818,7 +859,6 @@ def required_packages(self):
818859

819860
if self._pool.get('encryption'):
820861
packages.extend(get_format('luks').packages)
821-
822862
return packages
823863

824864
@property

‎tasks/enable_copr.yml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
---
2+
- name: make sure COPR is not already enabled
3+
shell: |
4+
set -euo pipefail
5+
{{ ansible_pkg_mgr }} repolist | \
6+
grep -c `echo {{ repo.repository }} | tr / :` || true
7+
args:
8+
warn: false
9+
register: copr_present
10+
changed_when: false
11+
12+
- name: get list of COPRs to be enabled
13+
command:
14+
cmd: "{{ ansible_pkg_mgr }} -y copr enable {{ repo.repository }}"
15+
warn: false
16+
when: copr_present.stdout == "0"

‎tasks/enable_coprs.yml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
---
2+
- name: check if the COPR support packages should be installed
3+
set_fact:
4+
install_copr: yes
5+
loop: "{{ _storage_copr_packages|default([]) }}"
6+
loop_control:
7+
loop_var: repo
8+
when:
9+
- _storage_copr_packages is defined
10+
- copr_packages is defined
11+
- repo.packages|intersect(copr_packages)|length
12+
13+
- name: make sure COPR support packages are present
14+
package:
15+
name: "{{ _storage_copr_support_packages }}"
16+
when: install_copr is defined and install_copr | bool
17+
18+
- name: enable COPRs
19+
include_tasks: "enable_copr.yml"
20+
loop: "{{ _storage_copr_packages|default([]) }}"
21+
loop_control:
22+
loop_var: repo
23+
when:
24+
- _storage_copr_packages is defined
25+
- copr_packages is defined
26+
- repo.packages|intersect(copr_packages)|length

‎tasks/main-blivet.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,11 @@
2626
packages_only: true
2727
register: package_info
2828

29+
- name: enable copr repositories if needed
30+
include_tasks: "enable_coprs.yml"
31+
vars:
32+
copr_packages: "{{ package_info.packages }}"
33+
2934
- name: make sure required packages are installed
3035
package:
3136
name: "{{ package_info.packages }}"

‎tests/test-verify-pool-members.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,9 @@
5757
- name: Check member encryption
5858
include_tasks: verify-pool-members-encryption.yml
5959

60+
- name: Check VDO
61+
include_tasks: verify-pool-members-vdo.yml
62+
6063
- set_fact:
6164
_storage_test_expected_pv_type: null
6265
_storage_test_expected_pv_count: null

‎tests/tests_create_lvmvdo_then_remove.yml‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
---
2+
- hosts: all
3+
become: true
4+
vars:
5+
storage_safe_mode: false
6+
mount_location: '/opt/test1'
7+
volume_group_size: '10g'
8+
volume_size: '12g'
9+
pool_size: '9g'
10+
11+
tasks:
12+
- include_role:
13+
name: linux-system-roles.storage
14+
15+
- name: Gather package facts
16+
package_facts:
17+
# gather information about packages
18+
19+
- name: Set blivet package name
20+
set_fact:
21+
blivet_pkg_name: "{{ ansible_facts.packages | select('search','blivet') | select('search', 'python') | list }}"
22+
23+
- name: Set blivet package version
24+
set_fact:
25+
blivet_pkg_version: "{{ ansible_facts.packages[blivet_pkg_name[0]][0]['version'] + '-' + ansible_facts.packages[blivet_pkg_name[0]][0]['release'] }}"
26+
27+
- block:
28+
- include_tasks: get_unused_disk.yml
29+
vars:
30+
min_size: "{{ volume_group_size }}"
31+
max_return: 1
32+
33+
- name: Create LVM VDO volume under volume group 'pool1'
34+
include_role:
35+
name: linux-system-roles.storage
36+
vars:
37+
storage_pools:
38+
- name: pool1
39+
disks: "{{ unused_disks }}"
40+
volumes:
41+
- name: volume1
42+
compression: true
43+
deduplication: true
44+
vdo_pool_size: "{{ pool_size }}"
45+
size: "{{ volume_size }}"
46+
mount_point: "{{ mount_location }}"
47+
48+
- include_tasks: verify-role-results.yml
49+
50+
- name: Repeat the previous invocation to verify idempotence
51+
include_role:
52+
name: linux-system-roles.storage
53+
vars:
54+
storage_pools:
55+
- name: pool1
56+
disks: "{{ unused_disks }}"
57+
volumes:
58+
- name: volume1
59+
compression: true
60+
deduplication: true
61+
vdo_pool_size: "{{ pool_size }}"
62+
size: "{{ volume_size }}"
63+
mount_point: "{{ mount_location }}"
64+
65+
- include_tasks: verify-role-results.yml
66+
67+
- name: Remove LVM VDO volume in 'pool1' created above
68+
include_role:
69+
name: linux-system-roles.storage
70+
vars:
71+
storage_pools:
72+
- name: pool1
73+
disks: "{{ unused_disks }}"
74+
state: "absent"
75+
volumes:
76+
- name: volume1
77+
compression: true
78+
deduplication: true
79+
vdo_pool_size: "{{ pool_size }}"
80+
size: "{{ volume_size }}"
81+
mount_point: "{{ mount_location }}"
82+
83+
- include_tasks: verify-role-results.yml
84+
85+
- name: Create LVM VDO volume under volume group 'pool1' (this time default size)
86+
include_role:
87+
name: linux-system-roles.storage
88+
vars:
89+
storage_pools:
90+
- name: pool1
91+
disks: "{{ unused_disks }}"
92+
volumes:
93+
- name: volume1
94+
compression: true
95+
deduplication: true
96+
size: "{{ volume_size }}"
97+
mount_point: "{{ mount_location }}"
98+
99+
- include_tasks: verify-role-results.yml
100+
101+
- name: Remove LVM VDO volume in 'pool1' created above
102+
include_role:
103+
name: linux-system-roles.storage
104+
vars:
105+
storage_pools:
106+
- name: pool1
107+
disks: "{{ unused_disks }}"
108+
state: "absent"
109+
volumes:
110+
- name: volume1
111+
compression: true
112+
deduplication: true
113+
size: "{{ volume_size }}"
114+
mount_point: "{{ mount_location }}"
115+
116+
- include_tasks: verify-role-results.yml
117+
118+
when: blivet_pkg_version is version("3.2.2-10", ">=") and ansible_facts["distribution"] != "Fedora"

‎tests/verify-pool-member-vdo.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
- name: check VDO
2+
block:
3+
- name: get information about VDO deduplication
4+
command: "vdostats --verbose {{ storage_test_pool.name }}-vdopool-vpool"
5+
register: storage_test_vdo_status
6+
changed_when: false
7+
8+
- set_fact:
9+
storage_test_vdo_dedupe_re: "{{ ('maximum dedupe queries +: +0$') }}"
10+
11+
- assert:
12+
that: "storage_test_vdo_status.stdout is regex(storage_test_vdo_dedupe_re)"
13+
msg: "VDO deduplication is on but it should not"
14+
when: not storage_test_vdo_volume.deduplication
15+
16+
- assert:
17+
that: "storage_test_vdo_status.stdout is not regex(storage_test_vdo_dedupe_re)"
18+
msg: "VDO deduplication is off but it should not"
19+
when: storage_test_vdo_volume.deduplication | bool
20+
21+
- set_fact:
22+
storage_test_vdo_compress_re: "{{ ('compressed fragments written +: +0$') }}"
23+
24+
- assert:
25+
that: "storage_test_vdo_status.stdout is regex(storage_test_vdo_compress_re)"
26+
msg: "VDO compression is on but it should not"
27+
when: not storage_test_vdo_volume.compression
28+
29+
- assert:
30+
that: "storage_test_vdo_status.stdout is not regex(storage_test_vdo_compress_re)"
31+
msg: "VDO compression is off but it should not"
32+
when: storage_test_vdo_volume.compression | bool
33+
34+
when:
35+
- storage_test_vdo_volume.deduplication != none or storage_test_vdo_volume.compression != none
36+
- storage_test_pool.state != "absent"
37+
- storage_test_vdo_volume.state != "absent"
38+
39+
- set_fact:
40+
storage_test_vdo_status: null

‎tests/verify-pool-members-vdo.yml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
- name: Validate pool member VDO settings
2+
include_tasks: verify-pool-member-vdo.yml
3+
loop: "{{ storage_test_pool.volumes }}"
4+
loop_control:
5+
loop_var: storage_test_vdo_volume
6+
when: storage_test_pool.type == 'lvm'

0 commit comments

Comments
 (0)