Skip to content

Commit 1bc184c

Browse files
iroykaufmaniroykaufman
committed
Remote attestation with PCRs and AMD SEV-SNP on GCP using RHCOS
Signed-off-by: Roy Kaufman <[email protected]>
1 parent 7378bac commit 1bc184c

File tree

20 files changed

+782
-1
lines changed

20 files changed

+782
-1
lines changed

‎.gitignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
cache/
22
coreos.key*
3-
fcos-cvm.tar*
43
*.ign
54
*.ociarchive
65
tmp/
6+
*.tar
7+
*.tar.gz

‎trustee-on-GCP/README.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Remote attestation with PCRs and AMD SEV-SNP on GCP using RHCOS
2+
3+
This guide provides step-by-step instructions for setting up remote attestation using PCRs and AMD SEV-SNP on Google Cloud Platform (GCP) with Red Hat CoreOS (RHCOS). It covers the deployment of a Trustee server and the creation of a custom RHCOS client image that communicates with the Trustee service to fetch encryption keys and decrypt the root image.
4+
5+
6+
## Prerequisites
7+
8+
1. Copy the pull secret from [Red Hat OpenShift](https://console.redhat.com/openshift/create/local) to ```~/.config/containers/auth.json``` into auths:quay.io:auth:&lt;pull_secret&gt;
9+
2. Install [gcloud](https://cloud.google.com/sdk/docs/install)
10+
3. Configure a subnet on GCP for the server and client by running ```./scripts/network_setup.sh```.
11+
12+
13+
## Deploy the trustee server (KBS)
14+
15+
1. Run ```./scripts/deploy-trustee.sh -k <SSH_KEY> -b ./trustee/trustee.bu```. This will start the KBS with the correct configuration (the name of this VM must match the hostname of the server, so it has to match `KBS_HOSTNAME` in `./scripts/rh-coreos/usr/libexec/aa-client`).
16+
2. Access the VM via SSH, then run ```sudo /usr/local/bin/populate_kbs.sh```. This will add the refrence value to Trustee.
17+
18+
## Deploy the client
19+
20+
1. Build a custom RHCOS image by running:
21+
```bash
22+
./scripts/build-rhcos-image.sh <IMAGE_NAME>
23+
```
24+
25+
2. Upload the image to GCP by running:
26+
```bash
27+
./scripts/upload_image_gcp.sh <BUCKET_NAME> <IMAGE_NAME>
28+
```
29+
30+
3. Deploy the client by running:
31+
```bash
32+
./scripts/deploy-client.sh -k <SSH_KEY> -b ./trustee/config/kbs-config.toml -n <VM_NAME> -i <IMAGE_NAME>
33+
```
34+
This will create the VM, perform attestation and decrypt the disk.
35+
36+
37+
38+
39+
## Info about the kbs and kbs-client
40+
41+
I use this version of [trustee](https://github.com/iroykaufman/trustee/tree/addtpm) and the [guest component](https://github.com/iroykaufman/guest-components/tree/TPM-as-additional-device).
42+
43+
Trustee includes [pr#851](https://github.com/confidential-containers/trustee/pull/851) with the following changes:
44+
45+
1. The guest component encrypts the public part of the AK in ASN.1 format, but trustee unmarshals it. The unmarshal part was replaced with an ASN.1 decrypt method.
46+
2. The TPM verifier does not check the nonce in the TPM because the `report_data` contains a digest of the `runtime_data` instead of the nonce. This is because the TPM is an additional device. This is a temporary solution.
47+
48+
49+
The changes in the guest component are included in this [PR#1093](https://github.com/confidential-containers/guest-components/pull/1093).
50+
51+
## Attestation Policy
52+
53+
The policy only checks hardware for both SEV-SNP and TPM.
54+
55+
## Resource Policy
56+
57+
Verify that both devices are affirming and exist.
58+

‎trustee-on-GCP/rh-coreos/Containerfile‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
FROM quay.io/rkaufman/kbs-tpm-snp:latest as kbc
2+
FROM quay.io/openshift-release-dev/ocp-v4.0-art-dev:419.96.202505021444-0-coreos
3+
4+
COPY usr /usr
5+
COPY --from=kbc /usr/local/bin/kbs-client /usr/bin/trustee-attester
6+
7+
RUN set -xeuo pipefail && \
8+
KERNEL_VERSION="$(basename $(ls -d /lib/modules/*))" && \
9+
raw_args="$(lsinitrd /lib/modules/${KERNEL_VERSION}/initramfs.img | grep '^Arguments: ' | sed 's/^Arguments: //')" && \
10+
stock_arguments=$(echo "$raw_args" | sed "s/'//g") && \
11+
echo "Using kernel: $KERNEL_VERSION" && \
12+
echo "Dracut arguments: $stock_arguments" && \
13+
mkdir -p /tmp/dracut /var/roothome && \
14+
dracut $stock_arguments && \
15+
mv -v /boot/initramfs*.img "/lib/modules/${KERNEL_VERSION}/initramfs.img" && \
16+
ostree container commit
17+
18+
LABEL com.coreos.osname="rh-coreos"

‎trustee-on-GCP/rh-coreos/luks.bu‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
variant: fcos
2+
version: 1.6.0
3+
4+
passwd:
5+
users:
6+
- name: core
7+
ssh_authorized_keys:
8+
- <KEY>
9+
10+
storage:
11+
luks:
12+
- name: root
13+
label: luks-root
14+
device: /dev/disk/by-partlabel/root
15+
clevis:
16+
custom:
17+
needs_network: false
18+
pin: tpm2
19+
config: '{"pcr_bank":"sha256","pcr_ids":"7"}'
20+
wipe_volume: true
21+
filesystems:
22+
- device: /dev/mapper/root
23+
format: xfs
24+
wipe_filesystem: true
25+
label: root
26+
files:
27+
- path: /etc/profile.d/systemd-pager.sh
28+
mode: 0644
29+
contents:
30+
inline: |
31+
# Tell systemd to not use a pager when printing information
32+
export SYSTEMD_PAGER=cat
33+
34+
systemd:
35+
units:
36+
- name: [email protected]
37+
dropins:
38+
- name: autologin-core.conf
39+
contents: |
40+
[Service]
41+
# Override Execstart in main unit
42+
ExecStart=
43+
# Add new Execstart with `-` prefix to ignore failure`
44+
ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM

‎trustee-on-GCP/rh-coreos/usr/lib/dracut/dracut.conf.d/50noxattr.conf‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
export DRACUT_NO_XATTR=1

‎trustee-on-GCP/rh-coreos/usr/lib/dracut/dracut.conf.d/50trustee.conf‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
force_drivers+=" sev-guest "

‎trustee-on-GCP/rh-coreos/usr/lib/dracut/modules.d/65aaclient/module-setup.sh‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/bin/bash
2+
3+
check() {
4+
# Always include the module for now
5+
return 0
6+
}
7+
8+
depends() {
9+
echo crypt systemd network
10+
return 0
11+
}
12+
13+
install () {
14+
inst $systemdsystemunitdir/aa-client.service
15+
inst /usr/libexec/aa-client
16+
inst /usr/bin/trustee-attester
17+
18+
inst curl
19+
inst cryptsetup
20+
inst tr
21+
inst lsblk
22+
inst mktemp
23+
inst base64
24+
inst /usr/lib/systemd/systemd-reply-password
25+
inst tpm2_evictcontrol
26+
inst tpm2_createprimary
27+
28+
systemctl -q --root "$initdir" add-wants initrd.target aa-client.service
29+
30+
# need to figure out why systemd-unit-file get x mode
31+
chmod -x $systemdsystemunitdir/aa-client.service
32+
33+
# need network -- figure out how to do it without chaning the command line
34+
echo "rd.neednet=1" > "${initdir}/etc/cmdline.d/65aa-client.conf"
35+
}

‎trustee-on-GCP/rh-coreos/usr/lib/systemd/system/aa-client.service‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
[Unit]
2+
Description=Confidential Containers Attestation Agent Client
3+
Before=cryptsetup.target
4+
Before=initrd-switch-root.target
5+
Before=ignition-complete.target
6+
Before[email protected]
7+
After=network-online.target
8+
After=dev-disk-by\x2dlabel-boot.device
9+
After=ignition-files.service
10+
Wants=network-online.target
11+
Requires=dev-disk-by\x2dlabel-boot.device
12+
13+
[Service]
14+
Type=oneshot
15+
ExecStart=/usr/libexec/aa-client
16+
# Should not be needed
17+
# SyslogLevel=debug
18+
# MountFlags=slave

‎trustee-on-GCP/rh-coreos/usr/libexec/aa-client‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
#!/bin/bash
2+
. /lib/dracut-lib.sh
3+
set -x
4+
5+
# Configuration
6+
KBS_HOSTNAME="kbs"
7+
KBS_PORT="8080"
8+
KBS_URL="http://${KBS_HOSTNAME}:${KBS_PORT}"
9+
ROOT_DEVICE=/dev/$(dmsetup info --noheadings -c -o blkdevs_used "$(blkid -L root)")
10+
CRYPT_ROOT_NAME="luks-root"
11+
BOOT_DEVICE="/dev/disk/by-label/boot"
12+
BOOT_MOUNT="/mnt/boot_partition"
13+
MARKER_FILE="${BOOT_MOUNT}/.trustee_done"
14+
ROOT_UUID_FILE="${BOOT_MOUNT}/.root_uuid"
15+
CRYPTTAB_FILE="/sysroot/etc/crypttab"
16+
MAX_RETRY_ATTEMPTS=3
17+
RETRY_DELAY=5
18+
19+
# temp configuration
20+
TEMP_DIR=$(mktemp -d /tmp/secure_attestation_XXXXXX)
21+
PASSPHRASE_FILE="${TEMP_DIR}/passphrase"
22+
OLD_PASS_FILE="${TEMP_DIR}/old_passphrase"
23+
24+
# Make sure the safety of temp files
25+
umask 077
26+
# Clean function - make sure it is always executed
27+
cleanup() {
28+
# Securely clean the sensitive data from memory
29+
if [ -n "${passphrase}" ]; then
30+
passphrase="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
31+
fi
32+
33+
# umount the mountpoint
34+
umount "${BOOT_MOUNT}" 2>/dev/null || true
35+
36+
# Delete the temp files or directories
37+
rm -rf "${TEMP_DIR}"
38+
39+
info "*** ATTESTATION SERVICE COMPLETED ***"
40+
}
41+
trap cleanup EXIT
42+
43+
fetch_passphrase() {
44+
local attempts=$MAX_RETRY_ATTEMPTS
45+
local count=0
46+
local result=""
47+
local status=0
48+
49+
while [ $count -lt $attempts ] && [ -z "$result" ]; do
50+
info "Attempt $((count+1)): Fetching passphrase from ${KBS_URL}"
51+
52+
if ! result=$(/usr/bin/trustee-attester --url "${KBS_URL}" get-resource --path default/machine/root 2>/dev/null); then
53+
status=$?
54+
info "Attestation failed with status $status"
55+
sleep $RETRY_DELAY
56+
elif [ -n "$result" ]; then
57+
info "Successfully retrieved passphrase"
58+
break
59+
else
60+
info "Empty response received"
61+
sleep $RETRY_DELAY
62+
fi
63+
64+
count=$((count+1))
65+
done
66+
67+
if [ -z "$result" ]; then
68+
info "Failed to retrieve passphrase after $attempts attempts"
69+
return 1
70+
fi
71+
72+
echo "$result"
73+
return 0
74+
}
75+
76+
replace_luks_key() {
77+
local device=$1
78+
local new_key=$2
79+
80+
info "Getting current LUKS key"
81+
. clevis-luks-common-functions
82+
local pt=$(clevis_luks_unlock_device "${device}")
83+
echo -n "${pt}" > "${OLD_PASS_FILE}"
84+
chmod 600 "${OLD_PASS_FILE}"
85+
pt="xxxxxxxxxx" # Erase the passphrase from memory
86+
87+
if ! /usr/sbin/cryptsetup --verbose open --test-passphrase "${device}" --key-file="${OLD_PASS_FILE}"; then
88+
info "Failed to verify old key"
89+
return 1
90+
fi
91+
92+
info "Replacing LUKS key"
93+
/usr/sbin/cryptsetup luksAddKey "${device}" --key-file="${OLD_PASS_FILE}" "${new_key}" || return 1
94+
/usr/sbin/cryptsetup luksKillSlot "${device}" 1 --key-file="${new_key}" || return 1
95+
96+
info "Do verification of new LUKS key"
97+
if ! /usr/sbin/cryptsetup --verbose open --test-passphrase "${device}" --key-file="${new_key}"; then
98+
info "Failed to verify new key"
99+
return 1
100+
fi
101+
102+
info "Removing LUKS token"
103+
/usr/sbin/cryptsetup token remove "${device}" --token-id 0
104+
105+
info "Create marker file"
106+
touch "${MARKER_FILE}"
107+
info "LUKS key replacement completed successfully"
108+
}
109+
110+
# Main execution
111+
info "*** ATTESTATION SERVICE FOR DISK ENCRYPTION ***"
112+
113+
# Create AK for TPM
114+
info "Creating Attestation Key (AK) in TPM"
115+
tpm2_createprimary -C e -G rsa2048:rsassa:null -g sha256 \
116+
-a "fixedtpm|fixedparent|sensitivedataorigin|userwithauth|sign"\
117+
-c ak.ctx
118+
119+
tpm2_evictcontrol -C o -c ak.ctx 0x81010002
120+
121+
122+
123+
# Check if root is already decrypted
124+
if [ -e "/dev/mapper/${CRYPT_ROOT_NAME}" ]; then
125+
info "Root device already decrypted"
126+
exit 0
127+
fi
128+
129+
# Fetch passphrase
130+
passphrase_base64=$(fetch_passphrase)
131+
[ -z "$passphrase_base64" ] && { info "No passphrase received"; exit 1; }
132+
133+
passphrase=$(echo "$passphrase_base64" | base64 -d | tr -cd '[:print:]')
134+
echo -n "$passphrase" > "${PASSPHRASE_FILE}"
135+
chmod 600 "${PASSPHRASE_FILE}"
136+
137+
# Mount boot partition
138+
mkdir -p "${BOOT_MOUNT}"
139+
if ! mount -o rw "${BOOT_DEVICE}" "${BOOT_MOUNT}" 2>/dev/null; then
140+
info "Warning: Could not mount boot partition - assuming first boot"
141+
BOOT_MOUNTED=false
142+
else
143+
BOOT_MOUNTED=true
144+
fi
145+
146+
if $BOOT_MOUNTED && [ -f "${MARKER_FILE}" ]; then
147+
info "Normal boot: Decrypting root filesystem"
148+
info "ATTESTATION SERVICE: Decrypting root filesystem with fetched key"
149+
ROOT_UUID=$(cat "${ROOT_UUID_FILE}")
150+
ROOT_DEVICE="/dev/disk/by-uuid/${ROOT_UUID}"
151+
info "Normal boot: Root device resolved as ${ROOT_DEVICE}"
152+
if /usr/sbin/cryptsetup --verbose open ${ROOT_DEVICE} ${CRYPT_ROOT_NAME} --key-file=${PASSPHRASE_FILE}; then
153+
info "ATTESTATION SERVICE: Successfully decrypted root filesystem"
154+
else
155+
info "ATTESTATION SERVICE: Failed to decrypt root filesystem"
156+
exit 1
157+
fi
158+
else
159+
info "First boot: Replacing LUKS key"
160+
info "First boot: Root device resolved as ${ROOT_DEVICE}"
161+
replace_luks_key "${ROOT_DEVICE}" "${PASSPHRASE_FILE}" || exit 1
162+
ROOT_UUID=$(blkid -s UUID -o value "${ROOT_DEVICE}")
163+
echo "$ROOT_UUID" > "${ROOT_UUID_FILE}"
164+
165+
# Prevent the system from calling [email protected]
166+
# to decrypt LUKS devices during other-boots.
167+
if [ -f "${CRYPTTAB_FILE}" ]; then
168+
info "Clearing crypttab file"
169+
echo > "${CRYPTTAB_FILE}" || info "Warning: Could not clear crypttab file"
170+
else
171+
info "Warning: Crypttab file not found at ${CRYPTTAB_FILE}"
172+
fi
173+
fi

‎trustee-on-GCP/scripts/build-rhcos-image.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/bin/bash
2+
3+
4+
5+
IMG_NAME=$1
6+
ociarchive=${IMG_NAME}.tar
7+
8+
set -xe
9+
10+
sudo setenforce 0
11+
12+
TMPDIR=$(mktemp -d)
13+
git clone --depth 1 https://github.com/coreos/custom-coreos-disk-images ${TMPDIR}
14+
15+
# Build the container image
16+
sudo podman build -t ${IMG_NAME} -f rh-coreos/Containerfile rh-coreos
17+
18+
sudo skopeo copy containers-storage:localhost/${IMG_NAME}:latest oci-archive:${ociarchive}
19+
sudo -E ${TMPDIR}/custom-coreos-disk-images.sh --platform gcp \
20+
--ociarchive ${ociarchive} \
21+
--osname rhcos
22+
rm -rf "$TMPDIR"

0 commit comments

Comments
 (0)