Skip to content

Commit d323e4c

Browse files
madhu-pillaimadhu-pillai
committed
feat: Enable CEX based LUKS encryption
 1. Configure the CEX in KVM host 2. Use the right device type ignition for MCO 3. Attach the CEX mediated device to guest vm Signed-off-by: Madhu Pillai <[email protected]>
1 parent 68d8afd commit d323e4c

20 files changed

+437
-2
lines changed

‎docs/set-variables-group-vars.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
:--- | :--- | :---
1313
**installation_type** | Can be of type kvm or lpar. Some packages will be ignored for installation in case of non lpar based installation. | kvm
1414
**controller_sudo_pass** | The password to the machine running Ansible (localhost). This will only be used for two things. To ensure you've installed the pre-requisite packages if you're on Linux, and to add the login URL to your /etc/hosts file. | Pas$w0rd!
15+
**cex_device** | Specify the storage device type used for LUKS encryption. This setting determines enable cex MCO Ignition configuration will be applied. Use in combination with the cex parameter. [dasd, fcp, virt]
1516

1617
## 2 - LPAR(s)
1718
**Variable Name** | **Description** | **Example**
@@ -403,3 +404,7 @@
403404
**zvm.interface.ip** | IP addresses for to be used for zVM nodes | 192.168.10.1
404405
**zvm.nodes.dasd.disk_id** | Disk id for dasd disk to be used for zVM node | 4404
405406
**zvm.nodes.lun** | Disk details of fcp disk to be used for zVM node | 840a
407+
408+
## Crypto Express Card based LUKS encryption specific for zKVM ( Optional )
409+
**Variable Name** | **Description** | **Example**
410+
**cex_uuid_map** | Specify guest hostname: "UUID:domain" UUID can be generated from uuidgen command and domain can be retrieved from lszcrypt | upi-cex-control-1: "68cd2d83-3eef-4e45-b22c-534f90b16cb9:00.0035"

‎inventories/default/group_vars/all.yaml.template‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
# Section 1 - Ansible Controller
77
installation_type: kvm
88
controller_sudo_pass: #X
9+
cex_device: #[dasd | fcp | virt ]
910

1011
env:
1112

@@ -250,3 +251,13 @@ abi:
250251
# (Optional) Proxy
251252
# Pls check the documentation which vars are present (include examples). If use_proxy is set to true,
252253
# than proxy_http, proxy_https and proxy_no must be set.
254+
255+
256+
# Section 15 - CEX based Luks Encryption ( Optional )
257+
cex_uuid_map:
258+
hostname:UUID:domain
259+
hostname:UUID:domain
260+
261+
# Provide control and compute hostname with uuid and domain.
262+
# UUID can be generated from `uuidgen` command and domain from lszcrypt -V
263+
# eg upi-control-1:5c84eefb-cb45-4519-86d3-ba23e65e8896:12.0001

‎playbooks/3_setup_kvm_host.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,3 +188,9 @@
188188
roles:
189189
- configure_storage
190190
- { role: macvtap, when: env.network_mode | upper != 'NAT' }
191+
192+
- hosts: kvm_host
193+
tags: setup, section_3
194+
become: true
195+
roles:
196+
- { role: configure_cex, when: cex_device is defined }

‎roles/configure_cex/tasks/main.yaml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
---
2+
3+
- name: Set cex_cards from cex_uuid_map values (uuid:domain only)
4+
set_fact:
5+
cex_cards: "{{ cex_uuid_map.values() | list | unique }}"
6+
7+
- name: Debug final list of CEX UUID assignments
8+
debug:
9+
var: cex_cards
10+
11+
- name: Create VFIO assignment script for all CEX cards
12+
template:
13+
src: assign_cards.sh.j2
14+
dest: /tmp/assign_all_cex_cards.sh
15+
mode: '0755'
16+
17+
- name: Execute VFIO assignment script
18+
shell: /tmp/assign_all_cex_cards.sh
19+
args:
20+
executable: /bin/bash
21+
22+
- name: Housekeep temporary assignment script
23+
file:
24+
path: /tmp/assign_all_cex_cards.sh
25+
state: absent
26+
27+
- name: Initialize empty cex_hostdev_map
28+
set_fact:
29+
cex_hostdev_map: {}
30+
31+
- name: Populate cex_hostdev_map with mdev format
32+
set_fact:
33+
cex_hostdev_map: "{{ cex_hostdev_map | combine({ item.key : 'mdev_' + (item.value.split(':')[0] | regex_replace('-', '_')) + '_matrix' }) }}"
34+
loop: "{{ cex_uuid_map | dict2items }}"
35+
36+
37+
- name: Save cex_hostdev_map to a file for reuse
38+
copy:
39+
dest: "/root/.cex_hostdev_map.json"
40+
content: "{{ cex_hostdev_map | to_nice_json }}"

‎roles/configure_cex/templates/assign_cards.sh.j2‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/bin/bash
2+
# Reference document for the cex configuration in zKVM
3+
# https://www.ibm.com/docs/en/linux-on-systems?topic=management-configuring-crypto-express-adapters-kvm-guests
4+
5+
# Configure each CEX card
6+
{% for entry in cex_cards %}
7+
{% set uuid = entry.split(':')[0] %}
8+
{% set matrix_val = entry.split(':')[1] %}
9+
{% set adapter = matrix_val.split('.')[0] %}
10+
{% set domain = matrix_val.split('.')[1] %}
11+
12+
uuid="{{ uuid }}"
13+
matrix_val="{{ matrix_val }}"
14+
adapter="{{ adapter }}"
15+
domain="{{ domain }}"
16+
17+
uuid_path="/sys/devices/vfio_ap/matrix/$uuid"
18+
matrix_file="$uuid_path/matrix"
19+
20+
if [ -d "$uuid_path" ]; then
21+
if grep -q "{{ matrix_val }}" "$matrix_file" 2>/dev/null; then
22+
echo "[INFO] UUID $uuid already configured with matrix {{ matrix_val }} — skipping."
23+
else
24+
echo "[WARN] UUID $uuid exists, but matrix entry '{{ matrix_val }}' not found!"
25+
echo "[WARN] Please reboot the node and try again."
26+
exit 1
27+
fi
28+
else
29+
modprobe vfio_ap
30+
echo 0x0 > /sys/bus/ap/apmask
31+
echo 0x0 > /sys/bus/ap/aqmask
32+
echo "[INFO] Creating UUID $uuid with adapter $adapter and domain $domain"
33+
echo "$uuid" > /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/create
34+
echo "0x$adapter" > "$uuid_path/assign_adapter"
35+
echo "0x$domain" > "$uuid_path/assign_domain"
36+
fi
37+
38+
{% endfor %}

‎roles/create_compute_nodes/tasks/main.yaml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
---
2+
- name: Load cex_hostdev_map from JSON
3+
set_fact:
4+
cex_hostdev_map: "{{ lookup('file', '/root/.cex_hostdev_map.json') | from_json }}"
25

36
- name: Install CoreOS on compute nodes
47
tags: create_compute_nodes
@@ -30,7 +33,10 @@
3033
--memballoon none \
3134
--graphics none \
3235
--wait=-1 \
33-
--noautoconsole
36+
--noautoconsole \
37+
{% set vm_name = env.cluster.nodes.compute.vm_name[i] %}
38+
{% set hostdev = '--hostdev ' + cex_hostdev_map[vm_name] if cex_device is defined and cex_hostdev_map is defined and vm_name in cex_hostdev_map else '' %}
39+
{{ hostdev }}
3440
timeout: 360
3541
with_sequence: start=0 end={{ (env.cluster.nodes.compute.hostname | length) - 1 }} stride=1
3642
loop_control:

‎roles/create_control_nodes/tasks/main.yaml‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,32 @@
11
---
22

3+
- name: Load cex_hostdev_map from JSON file
4+
set_fact:
5+
cex_hostdev_map: "{{ lookup('file', '/root/.cex_hostdev_map.json') | from_json }}"
6+
7+
- name: Debug rendered cex_hostdev per control node
8+
debug:
9+
msg: "VM: {{ vm_name }}, Hostdev: {{ cex_hostdev }}"
10+
vars:
11+
vm_name: "{{ env.cluster.nodes.control.vm_name[i] }}"
12+
cex_hostdev: >-
13+
{% if cex_device is defined and cex_hostdev_map is defined and vm_name in cex_hostdev_map %}
14+
--hostdev={{ cex_hostdev_map[vm_name] }}
15+
{% else %}
16+
""
17+
{% endif %}
18+
with_sequence: start=0 end={{ (env.cluster.nodes.control.hostname | length) - 1 }}
19+
loop_control:
20+
index_var: i
21+
322
- name: Create CoreOS control nodes on the the KVM host.
423
tags: create_control_nodes
24+
vars:
25+
vm_name: "{{ env.cluster.nodes.control.vm_name[i] }}"
26+
cex_hostdev: >-
27+
{% if cex_device is defined and cex_hostdev_map is defined and vm_name in cex_hostdev_map %}
28+
--hostdev={{ cex_hostdev_map[vm_name] }}
29+
{% endif %}
530
shell: |
631
virt-install \
732
--name {{ env.cluster.nodes.control.vm_name[i] }} \
@@ -29,7 +54,8 @@
2954
--graphics none \
3055
--console pty,target_type=serial \
3156
--wait=-1 \
32-
--noautoconsole
57+
--noautoconsole \
58+
{{ cex_hostdev }}
3359
timeout: 360
3460
with_sequence: start=0 end={{ (env.cluster.nodes.control.hostname | length) - 1 }} stride=1
3561
loop_control:
@@ -67,6 +93,7 @@
6793
--graphics none \
6894
--wait=-1 \
6995
--noautoconsole
96+
7097
when: env.z.high_availability == True and inventory_hostname == env.z.lpar1.hostname and env.cluster.nodes.control.vm_name[0] not in hosts_with_host_vars
7198

7299
- name: Create the second CoreOS control node on the second KVM host, if cluster is to be highly available.

‎roles/get_ocp/files/dasd-master-machineconfig.bu‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
variant: openshift
2+
version: 4.19.0
3+
metadata:
4+
name: master-luks-storage
5+
labels:
6+
machineconfiguration.openshift.io/role: master
7+
openshift:
8+
fips: true
9+
kernel_arguments:
10+
- rd.luks.key=/etc/luks/cex.key
11+
boot_device:
12+
layout: s390x-eckd
13+
luks:
14+
device: /dev/dasda
15+
cex:
16+
enabled: true

‎roles/get_ocp/files/dasd-master-machineconfig.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Generated by Butane; do not edit
2+
apiVersion: machineconfiguration.openshift.io/v1
3+
kind: MachineConfig
4+
metadata:
5+
labels:
6+
machineconfiguration.openshift.io/role: master
7+
name: master-luks-storage
8+
spec:
9+
config:
10+
ignition:
11+
version: 3.5.0
12+
storage:
13+
filesystems:
14+
- device: /dev/mapper/root
15+
format: xfs
16+
label: root
17+
wipeFilesystem: true
18+
luks:
19+
- cex:
20+
enabled: true
21+
device: /dev/dasda2
22+
label: luks-root
23+
name: root
24+
options:
25+
- --cipher
26+
- aes-cbc-essiv:sha256
27+
wipeVolume: true
28+
fips: true
29+
kernelArguments:
30+
- rd.luks.key=/etc/luks/cex.key

‎roles/get_ocp/files/dasd-worker-machineconfig.bu‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
variant: openshift
2+
version: 4.19.0
3+
metadata:
4+
name: worker-luks-storage
5+
labels:
6+
machineconfiguration.openshift.io/role: worker
7+
openshift:
8+
fips: true
9+
kernel_arguments:
10+
- rd.luks.key=/etc/luks/cex.key
11+
boot_device:
12+
layout: s390x-eckd
13+
luks:
14+
device: /dev/dasda
15+
cex:
16+
enabled: true

0 commit comments

Comments
 (0)