Skip to content

Commit 8834419

Browse files
Restructuring, rewriting.
1 parent 9b1bcd8 commit 8834419

File tree

5 files changed

+266
-559
lines changed

5 files changed

+266
-559
lines changed

content/learning-paths/servers-and-cloud-computing/cca-device-attach/1-introduction.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Together, CCA and RME deliver the infrastructure needed to build Confidential Co
1818

1919
A Realm is a protected execution environment enabled by RME. It operates independently from both the Normal World where operating systems and user applications run, and the Secure World, which hosts trusted firmware and trusted execution environments (TEEs).
2020

21-
Realms allow lower-privileged software, such as an application or a virtual machine, to protect its content and execution from attacks by higher-privileged software, such as an OS or a hypervisor. Realms provide an environment for confidential computing, without requiring the Realm owner to trust the software components that manage the resources that the Realm uses.
21+
Realms allow lower-privileged software, such as an application or a virtual machine, to protect its content and execution from attacks by higher-privileged software, such as an OS or a hypervisor. Essentially, realms provide an environment for confidential computing without requiring the Realm owner to trust the software components that manage the resources that the Realm uses.
2222

2323
While Realms are isolated by design, they still need to interact with external components to be practical - for example, accessing a network interface or communicating with a peripheral device.
2424

Lines changed: 22 additions & 282 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
title: "Explore secure device attach with VirtIO"
2+
title: "VirtIO for device attach"
33
weight: 3
44

55
### FIXED, DO NOT MODIFY
@@ -8,7 +8,7 @@ layout: learningpathall
88

99
## Overview
1010

11-
This section introduces VirtIO and Bounce Buffers in the context of CCA Realms, and explains how they enable secure data exchange between a Realm and the untrusted external world.
11+
This section introduces VirtIO in the context of CCA Realms, and explains how it enables secure data exchange between a Realm and the untrusted external world.
1212

1313
A Realm must eventually use physical devices to interact with the external world. The easiest way to achieve this is by using VirtIO, which provides a fast, high-level emulation layer. This is considered the first level of device attach, where access is mediated by the hypervisor using paravirtualized interfaces.
1414

@@ -17,296 +17,36 @@ and **D**ata **E**ncryption), where the host OS assigns a physical device to a R
1717

1818
### What is VirtIO?
1919

20-
VirtIO provides an efficient, paravirtualized I/O interface between Realms and host devices. It is an abstraction layer for virtual devices in virtualized environments. It provides standardized and efficient interfaces between guest virtual machines (VMs) and host devices, making it easier to develop paravirtualized drivers.
20+
VirtIO provides an efficient, paravirtualized I/O interface between Realms and host devices. It is an abstraction layer for virtual devices in virtualized environments. VirtIO is a standardized, efficient interface for virtual devices in virtualized environments. It lets guest operating systems use paravirtualized drivers to communicate with host-provided devices, avoiding the overhead of fully emulating physical hardware.
2121

2222
Paravirtualized means that the guest OS is aware it’s running in a virtualized environment and can use optimized drivers (VirtIO) to communicate with virtual hardware. Emulating physical hardware devices (like NICs or disks) for VMs is slow and inefficient. VirtIO allows VMs to bypass full device emulation and use streamlined drivers.
2323

24-
VirtIO is most commonly used with KVM/QEMU virtualization.
24+
VirtIO is most commonly used with KVM/QEMU virtualization. Example drivers include:
2525

26-
Example drivers include:
27-
28-
- `virtio-net`: Paravirtualized networking
29-
- `virtio-blk`: Block device (disk) access
30-
- `virtio-fs`: File sharing (host ↔ guest)
31-
- `virtio-balloon`: Dynamic memory management
32-
- `virtio-rng`: Random number source
33-
- `virtio-console`: Simple console interface
26+
- `virtio-net`: paravirtualized networking
27+
- `virtio-blk`: block storage
28+
- `virtio-fs`: file sharing between host and guest
29+
- `virtio-balloon`: dynamic memory management
30+
- `virtio-rng`: random number source
31+
- `virtio-console`: simple console interface
3432

3533
### How does VirtIO work in VMs?
3634

37-
Here is an overview of how VirtIO works in Virtual Machines:
38-
39-
1. The Host Hypervisor (for example, QEMU/KVM) exposes VirtIO backend devices.
40-
2. The guest OS loads VirtIO _frontend_ drivers (for example, `virtio_net`,
41-
`virtio_blk`) that communicate using the VirtIO protocol.
42-
3. Communication happens via shared memory (`virtqueues`) for I/O operations,
43-
avoiding full device emulation.
44-
4. Devices are exposed over the PCI or MMIO bus to the guest.
45-
46-
For example, instead of emulating an Intel e1000 NIC, the host exposes a `virtio-net` interface, and the guest OS uses the `virtio-net` driver to send and receive packets via shared buffers.
47-
48-
## Bounce buffers
49-
50-
### What are bounce buffers?
51-
52-
Bounce buffers are temporary memory buffers used when Direct Memory Access (DMA) cannot be performed directly on the original data buffer. This might be because:
53-
54-
1. The original buffer is not physically contiguous
55-
2. The buffer is in high memory or not accessible to the device
56-
3. The buffer does not meet alignment or boundary constraints required by the device
57-
58-
## Why use bounce buffers?
59-
60-
Data _bounces_ between:
61-
- The original buffer (in user or kernel space) and
62-
- The DMA-capable bounce buffer (used for I/O with the device)
63-
64-
This ensures that data transfer is possible even when the original memory isn’t suitable for DMA.
65-
66-
## CCA Realms, VirtIO and bounce buffers
67-
68-
The defining feature of a Realm is that its memory (called *Realm memory*) is cryptographically isolated from both the Normal and Secure Worlds.
69-
70-
This means that:
71-
- Realm memory is encrypted with unique keys
72-
- Non-Realm entities (host OS, hypervisor) cannot directly read or
73-
write to Realm memory
74-
- Even Direct Memory Access (DMA) from peripherals or untrusted drivers cannot
75-
access Realm data
76-
77-
This design ensures confidentiality but presents a challenge: how can a Realm securely interact with untrusted components, such as network stacks in the host OS, storage subsystems, or I/O devices managed by untrusted drivers? To exchange data securely with untrusted components (for example, network stacks, storage subsystems), Realms use *bounce buffers* as intermediaries.
78-
79-
### How bounce buffers are used with RME
80-
81-
1. Exporting Data:
82-
- A Realm application prepares some data (for example, the results of computation)
83-
- It copies this data from protected Realm memory into a bounce buffer.
84-
- The Realm notifies the untrusted host or hypervisor that the data is ready.
85-
- The host retrieves the data from the bounce buffer.
86-
87-
2. Importing Data:
88-
- The host places data (for example, input from a file or device) into a bounce buffer.
89-
- The Realm is notified and validates the source.
90-
- The Realm copies the data from the bounce buffer into its protected memory.
91-
92-
This pattern preserves confidentiality and integrity of Realm data, since:
93-
- The Realm never allows direct access to its memory.
94-
- It can validate and sanitize any data received via bounce buffers.
95-
- No sensitive data is exposed without explicit copying.
96-
97-
### Is confidentiality preserved with bounce buffers?
98-
99-
In the previous section, it was mentioned that bounce buffers preserve confidentiality. Lets dive a little deeper into that. Bounce buffers are nothing more than an explicitly shared temporary area between the Realm world and the outside world. This does indeed preserve the confidentiality of the rest of the Realm data. On the other hand, for the data being transferred, it is leaving the Realm world and will only remain confidential if it is encrypted in some way, for example, for network traffic, TLS should be used.
100-
101-
## Seeing a Realm's bounce buffers at work
102-
103-
Let's put this to work and check for ourselves that bounce buffers are used. The steps in this section will build on the Key Broker demo that was used in the [CCA
104-
Essentials learning path](/learning-paths/servers-and-cloud-computing/cca-essentials/example/),
105-
demonstrating an end-to-end attestation.
106-
107-
### Start the Key Broker Server (KBS)
108-
109-
First, pull the docker container image with the pre-built KBS, and then run the container:
110-
111-
```bash
112-
docker pull armswdev/cca-learning-path:cca-key-broker-v2
113-
docker run --rm -it armswdev/cca-learning-path:cca-key-broker-v2
114-
```
115-
116-
Now within your running docker container, get a list of network interfaces:
117-
118-
```bash
119-
ip -c a
120-
```
121-
122-
The output should look like:
123-
124-
```output
125-
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
126-
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
127-
inet 127.0.0.1/8 scope host lo
128-
valid_lft forever preferred_lft forever
129-
inet6 ::1/128 scope host
130-
valid_lft forever preferred_lft forever
131-
20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
132-
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
133-
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
134-
valid_lft forever preferred_lft forever
135-
```
136-
137-
Start the KBS on the `eth0` network interface, and replace 172.17.0.2 shown in
138-
the command below with the IP address corresponding to eth0 in the output of `ip
139-
-c a` above.
140-
141-
```bash
142-
./keybroker-server -v --addr 172.17.0.2
143-
```
144-
145-
The output should look like:
146-
147-
```output
148-
INFO starting 16 workers
149-
INFO Actix runtime found; starting in Actix runtime
150-
INFO starting service: "actix-web-service-172.17.0.2:8088", workers: 16, listening on: 172.17.0.2:8088
151-
```
152-
153-
### Get into a Realm
154-
155-
With the Key Broker Server running in one terminal, open up a new terminal in
156-
which you will run the Key Broker Client (KBC). The intent is to
157-
observe that the data transmitted over the network (thru `virtio_net`) are
158-
indeed using bounce buffers.
159-
160-
Pull the docker container image with the pre-built KBC, and then run the container:
161-
162-
```bash
163-
docker pull armswdev/cca-learning-path:cca-simulation-v2
164-
docker run --rm -it armswdev/cca-learning-path:cca-simulation-v2
165-
```
166-
167-
Within the running container, launch the `run-cca-fvp.sh` script to run the Arm
168-
CCA pre-built binaries on the FVP:
169-
170-
```bash
171-
./run-cca-fvp.sh
172-
```
173-
The `run-cca-fvp.sh` script uses the screen command to connect to the different
174-
UARTs in the FVP.
175-
176-
You should see the host Linux kernel boot on your terminal and you will be
177-
prompted to log in to the host. Enter root as the username:
178-
179-
```output
180-
[ 4.169458] Run /sbin/init as init process
181-
[ 4.273748] EXT4-fs (vda): re-mounted 64d1bcff-5d03-412c-83c6-48ec4253590e r/w. Quota mode: none.
182-
Starting syslogd: OK
183-
Starting klogd: OK
184-
Running sysctl: OK
185-
Starting network: [ 5.254843] smc91x 1a000000.ethernet eth0: link up, 10Mbps, half-duplex, lpa 0x0000
186-
udhcpc: started, v1.36.1
187-
udhcpc: broadcasting discover
188-
udhcpc: broadcasting select for 172.20.51.1, server 172.20.51.254
189-
udhcpc: lease of 172.20.51.1 obtained from 172.20.51.254, lease time 86400
190-
deleting routers
191-
adding dns 172.20.51.254
192-
OK
193-
194-
Welcome to the CCA host
195-
host login: root
196-
(host) #
197-
```
198-
199-
Change directory to `/cca` and use `lkvm` to launch a guest Linux in a Realm:
200-
```bash
201-
cd /cca
202-
./lkvm run --realm --disable-sve --irqchip=gicv3-its --firmware KVMTOOL_EFI.fd -c 1 -m 512 --no-pvtime --disk guest-disk.img --restricted_mem --virtio-transport pci --pmu --network mode=user
203-
```
204-
205-
You should see the realm boot. Note that `lkvm` is invoked with `--network
206-
mode=user`, which makes the guest see the network through a VirtIO device.
207-
208-
After boot up, which might take some time, you will be prompted to log in at the
209-
guest Linux prompt. Use root again as the username:
210-
211-
```output
212-
Starting syslogd: OK
213-
Starting klogd: OK
214-
Running sysctl: OK
215-
Starting network: udhcpc: started, v1.36.1
216-
udhcpc: broadcasting discover
217-
udhcpc: broadcasting select for 192.168.33.15, server 192.168.33.1
218-
udhcpc: lease of 192.168.33.15 obtained from 192.168.33.1, lease time 14400
219-
deleting routers
220-
adding dns 172.20.51.254
221-
OK
222-
223-
Welcome to the CCA realm
224-
realm login: root
225-
(realm) #
226-
```
227-
228-
### Observe bounce buffer usage in the realm
229-
230-
First, check that the Linux kernel has tracing support:
231-
232-
```bash { output_lines="2-46" }
233-
ls /sys/kernel/debug/tracing/events/
234-
9p i2c_slave qcom_glink
235-
alarmtimer icmp qcom_smp2p
236-
asoc initcall qdisc
237-
block interconnect ras
238-
bpf_test_run io_uring raw_syscalls
239-
bpf_trace iomap rcu
240-
bridge iommu regmap
241-
capability ipi regulator
242-
cgroup irq rpcgss
243-
chipidea jbd2 rpm
244-
clk kmem rpmh
245-
cma ksm rseq
246-
compaction kvm rtc
247-
cpuhp kyber sched
248-
cros_ec libata scmi
249-
csd lock scsi
250-
dev lockd signal
251-
devfreq maple_tree skb
252-
devlink mdio smbus
253-
dma memcg sock
254-
dma_fence migrate spi
255-
dpaa2_eth mmap spmi
256-
dpaa_eth mmap_lock sunrpc
257-
dwc3 mmc swiotlb
258-
e1000e_trace module task
259-
enable mtu3 tcp
260-
error_report musb tegra_apb_dma
261-
ext4 napi thermal
262-
fib neigh thermal_power_allocator
263-
filelock net thp
264-
filemap netfs timer
265-
fsl_edma netlink timer_migration
266-
ftrace nfs timestamp
267-
gadget nfs4 tlb
268-
gpio notifier udp
269-
gpu_mem oom ufs
270-
handshake optee vmalloc
271-
header_event page_isolation vmscan
272-
header_page page_pool watchdog
273-
hns3 pagemap workqueue
274-
huge_memory percpu writeback
275-
hugetlbfs power xdp
276-
hw_pressure printk xhci-hcd
277-
hwmon pwm
278-
i2c qcom_aoss
279-
```
280-
281-
As shown above, you should see a list of the available trace
282-
points.
283-
284-
Now, enable the kernel tracing infrastructure together with bounce buffer
285-
tracing, read the trace in the background (filtering on `keybroker-app-`) and
286-
run the Key Broker Client application in the realm, using the endpoint address
287-
that the Key Broker Server is listening on (from the other terminal):
35+
Here is an overview of how VirtIO works in virtual machines:
28836

289-
```bash
290-
echo 1 > /sys/kernel/debug/tracing/tracing_on
291-
echo 1 > /sys/kernel/debug/tracing/events/swiotlb/enable
292-
grep keybroker-app- /sys/kernel/debug/tracing/trace_pipe &
293-
keybroker-app -v --endpoint http://172.17.0.2:8088 skywalker
294-
```
37+
1. The host hypervisor (for example, QEMU/KVM) exposes VirtIO backend devices
38+
2. The guest OS loads VirtIO frontend drivers such as `virtio_net`or `virtio_blk` that communicate using the VirtIO protocol
39+
3. I/O uses shared memory `virtqueues`, which avoids full device emulation
40+
4. Devices are exposed over the PCI or MMIO bus to the guest
29541

296-
In the `keybroker-app`command above, `skywalker` is the key name that is
297-
requested from the KBS.
42+
For example, instead of emulating an Intel e1000 NIC, the host exposes a `virtio-net` interface. The guest OS uses the `virtio-net` driver to exchange packets through shared buffers.
29843

299-
The output should look like:
44+
### Key takeaways
30045

301-
```output
302-
INFO Requesting key named 'skywalker' from the keybroker server with URL http://172.17.0.2:8088/keys/v1/key/skywalker
303-
INFO Challenge (64 bytes) = [5c, ec, 1e, f5, 93, 54, 4a, 8a, ee, 2e, 46, a0, 50, 0d, 41, dd, d4, 60, b0, 58, 5b, 51, 71, 76, d1, 66, d3, b7, 38, e8, af, ae, 0a, 07, 4e, c5, 60, dc, 4a, c0, b8, 73, 98, d9, bd, af, 41, 96, 99, 6d, 74, cc, 19, 70, 24, c4, c9, 5c, 21, 61, 1a, cb, 76, 75]
304-
INFO Submitting evidence to URL http://172.17.0.2:8088/keys/v1/evidence/1928844131
305-
INFO Attestation success :-) ! The key returned from the keybroker is 'May the force be with you.'
306-
keybroker-app-143 [000] b..2. 1772.607321: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
307-
keybroker-app-143 [000] b..2. 1772.644478: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
308-
```
46+
- VirtIO provides fast I/O through paravirtualization, not hardware emulation
47+
- Shared queues reduce overhead and context switching
48+
- It is the simplest and most common first step for device attach in Realms
30949

310-
Note that the interleaving of the trace messages and KBC messages might differ
311-
from one run to another. With the `switlb_bounced` messages above you can successfully observe that the bounce buffers are being used in the realm.
50+
## Next steps
31251

52+
Learn how bounce buffers make this safe for Realms in [Bounce buffers](./bounce-buffers.md)

0 commit comments

Comments
 (0)