Merge pull request #2154 from Arnaud-de-Grandmaison-ARM/bounce-buffers

First sketch of the CCA DA LP.

Author: pareenaverma

content/learning-paths/servers-and-cloud-computing/cca-device-attach/1-introduction.md

@@ -0,0 +1,30 @@
+---
+title: "About CCA Realms"
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+Arm’s *Realm Management Extension (RME)* is a key security feature introduced in
+the Armv9-A architecture. It enables a new form of hardware-enforced isolation
+designed to support _confidential computing_. It defines the set of hardware
+features and properties that are required to comply with the Arm *Confidential
+Computing Architecture (CCA)* architecture.
+
+At the heart of RME is the concept of a *Realm*, a protected execution
+environment that operates independently from the conventional *Normal World*
+(used by operating systems and applications) and the *Secure World* (used by
+trusted firmware or TEE). Realms are managed by a new privileged entity called
+the *Realm Management Monitor (RMM)* and are enforced by the hardware via the
+*Granule Protection Table (GPT)* and *Granule Transitioning* mechanism.
+
+Realms allow lower-privileged software, such as an application or a virtual
+machine, to protect its content and execution from attacks by higher-privileged
+software, such as an OS or a hypervisor. Realms provide an environment for
+confidential computing, without requiring the Realm owner to trust the software
+components that manage the resources that the Realm uses.
+
+To be useful, a Realm has to interact with the rest of the world at some point.
+For example, a network interface is likely to be needed. This learning path will
+teach you how devices are attached and used by Realms.

content/learning-paths/servers-and-cloud-computing/cca-device-attach/2-virtio.md

@@ -0,0 +1,339 @@
+---
+title: "Device attach - part 1: virtio"
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+This section is a high level overview of VirtIO and Bounce Buffers, and how they
+relate to CCA Realms.
+
+A Realm has to use physical devices at some point to interact with the external
+and / or physical world. The easiest way to do this is by using VirtIO, which
+provides a fast high level emulation layer. This can be seen as the level-0 of
+device attach.
+
+As will be seen later in this learning path, more evolved device attach can be
+performed leveraging hardware security features like PCIe-TDISP (**T**EE
+**D**evice **I**nterface **S**ecurity **P**rotocol) and PCIe-IDE (**I**ntegrity
+and **D**ata **E**ncryption), where the host OS can assign a physical device to
+a realm, which will be able to make security measurements on the physical device
+and include it in its base and measurements.
+
+## VirtIO
+
+### What is VirtIO ?
+
+VirtIO is an abstraction layer for virtual devices in virtualized environments.
+It provides standardized and efficient interfaces between guest virtual machines
+(VMs) and host devices, making it easier to develop _paravirtualized_ drivers.
+_Paravirtualized_ means that the guest OS is aware it’s running in a virtualized
+environment and can use optimized drivers (VirtIO) to communicate with virtual
+hardware. Emulating hardware devices (like NICs or disks) for VMs is slow and
+inefficient. VirtIO provides a standardized and efficient interface that allows
+VMs to bypass full device emulation and instead use optimized drivers.
+
+VirtIO is most commonly used with KVM/QEMU virtualization. Example drivers are:
+- `virtio-net`: Paravirtualized networking
+- `virtio-blk`: Block device (disk) access
+- `virtio-fs`: File sharing (host ↔ guest)
+- `virtio-balloon`: Dynamic memory management
+- `virtio-rng`: Random number source
+- `virtio-console`: Simple console interface
+- ...
+
+### How VirtIO works in VMs
+
+1. The Host Hypervisor (e.g., QEMU/KVM) exposes VirtIO “backend” devices.
+2. The guest OS loads VirtIO _frontend_ drivers (e.g., `virtio_net`,
+   `virtio_blk`) that speak the VirtIO protocol.
+3. Communication happens via shared memory (`virtqueues`) for I/O operations,
+   avoiding full device emulation.
+4. Devices are exposed over the PCI or MMIO bus to the guest.
+
+For example, instead of emulating an Intel e1000 NIC, the host exposes a
+`virtio-net` interface to the guest OS and the guest OS uses the `virtio-net`
+driver to send/receive packets via shared buffers.
+
+## Bounce buffers
+
+### What are bounce buffers?
+
+Bounce buffers are a generic mechanism part of the Linux kernel. They are
+temporary memory buffers used in the Linux kernel to handle situations where
+direct memory access (DMA) can’t be performed directly on the original data
+buffer. This often happens because:
+1. The original buffer is not physically contiguous.
+2. The buffer is in high memory or not accessible to the device.
+3. The buffer doesn’t meet alignment or boundary requirements of the device.
+
+### Why _bounce_ buffers?
+
+Data _bounces_ between:
+- The original buffer (in user/kernel space) and
+- The DMA-capable bounce buffer (used for I/O with the device)
+
+This ensures that data transfers can still happen even when the original memory
+is not suitable or accessible for transfers.
+
+## CCA Realms, VirtIO and bounce buffers
+
+The defining feature of a Realm is that its memory (called *Realm memory*) is
+cryptographically isolated from both the Normal and Secure Worlds. This means
+that:
+- Realm memory is encrypted using keys that are unique to each Realm.
+- Non-Realm entities (like the host OS or hypervisor) cannot directly read or
+  write Realm memory.
+- Even Direct Memory Access (DMA) from peripherals or untrusted drivers cannot
+  access Realm data.
+
+This design ensures confidentiality but introduces a problem: *how can Realms
+interact with untrusted components*, such as:
+- Network stacks in the host OS,
+- Storage subsystems,
+- I/O devices managed by untrusted drivers?
+
+The solution to safely exchange data between a Realm and the outside World is to
+use _bounce buffers_ as an intermediary !
+
+### How bounce buffers are used with RME
+
+1. Exporting Data:
+   - A Realm application prepares some data (e.g., results of computation).
+   - It copies this data from protected Realm memory into a bounce buffer.
+   - The Realm notifies the untrusted host or hypervisor that the data is ready.
+   - The host retrieves the data from the bounce buffer.
+
+2. Importing Data:
+   - The host places data (e.g., input from a file or device) into a bounce buffer.
+   - The Realm is notified and validates the source.
+   - The Realm copies the data from the bounce buffer into its protected memory.
+
+This pattern preserves confidentiality and integrity of Realm data, since:
+- The Realm never allows direct access to its memory.
+- It can validate and sanitize any data received via bounce buffers.
+- No sensitive data is exposed without explicit copying.
+
+### Confidentiality preserved with bounce buffers, really?
+
+In the previous section, it was mentioned that _bounce buffers preserves
+confidentiality_, a sentence which deserves a bit more thoughts. Bounce buffers
+are nothing more than an explicitly shared temporary area between the Realm
+world and the outside world. This does indeed preserve the confidentiality of
+all the rest of the Realm data. On the other hand, for the data being
+transferred, it is leaving the Realm world and will only remain confidential if it
+is encrypted in some ways, e.g. for network traffic, TLS should be used.
+
+## Seeing a Realm's bounce buffers at work
+
+Let's put this to work and check for ourselves that bounce buffers are used. This
+will build on the Key Broker demo that was used in the [CCA
+Essentials learning path](/learning-paths/servers-and-cloud-computing/cca-essentials/example/),
+demonstrating an end-to-end attestation.
+
+### Start the **K**eyb **B**roker **Server** (KBS)
+
+First, pull the docker container image with the pre-built KBS, and then run the container:
+
+```bash
+docker pull armswdev/cca-learning-path:cca-key-broker-v2
+docker run --rm -it armswdev/cca-learning-path:cca-key-broker-v2
+```
+
+Now within your running docker container, get a list of network interfaces:
+
+```bash
+ip -c a
+```
+
+The output should look like:
+
+```output
+1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host
+       valid_lft forever preferred_lft forever
+20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
+    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
+    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
+       valid_lft forever preferred_lft forever
+```
+
+Start the KBS on the `eth0` network interface, and replace 172.17.0.2 shown in
+the command below with the IP address corresponding to eth0 in the output of `ip
+-c a` above.
+
+```bash
+./keybroker-server -v --addr 172.17.0.2
+```
+
+The output should look like:
+
+```output
+INFO starting 16 workers
+INFO Actix runtime found; starting in Actix runtime
+INFO starting service: "actix-web-service-172.17.0.2:8088", workers: 16, listening on: 172.17.0.2:8088
+```
+
+### Get into a Realm
+
+With the Key Broker Server running in one terminal, open up a new terminal in
+which you will run the **K**ey **B**roker **C**lient (KBC). The intent is to
+observe that the data transmitted over the network (thru `virtio_net`) are
+indeed using bounce buffers.
+
+Pull the docker container image with the pre-built KBC, and then run the container:
+
+```bash
+docker pull armswdev/cca-learning-path:cca-simulation-v2
+docker run --rm -it armswdev/cca-learning-path:cca-simulation-v2
+```
+
+Within the running container, launch the `run-cca-fvp.sh` script to run the Arm
+CCA pre-built binaries on the FVP:
+
+```bash
+./run-cca-fvp.sh
+```
+The `run-cca-fvp.sh` script uses the screen command to connect to the different
+UARTs in the FVP.
+
+You should see the host Linux kernel boot on your terminal and you will be
+prompted to log in to the host. Enter root as the username:
+
+```output
+[    4.169458] Run /sbin/init as init process
+[    4.273748] EXT4-fs (vda): re-mounted 64d1bcff-5d03-412c-83c6-48ec4253590e r/w. Quota mode: none.
+Starting syslogd: OK
+Starting klogd: OK
+Running sysctl: OK
+Starting network: [    5.254843] smc91x 1a000000.ethernet eth0: link up, 10Mbps, half-duplex, lpa 0x0000
+udhcpc: started, v1.36.1
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.20.51.1, server 172.20.51.254
+udhcpc: lease of 172.20.51.1 obtained from 172.20.51.254, lease time 86400
+deleting routers
+adding dns 172.20.51.254
+OK
+
+Welcome to the CCA host
+host login: root
+(host) #
+```
+
+Change directory to `/cca` and use `lkvm` to launch a guest Linux in a Realm:
+```bash
+cd /cca
+./lkvm run --realm --disable-sve --irqchip=gicv3-its --firmware KVMTOOL_EFI.fd -c 1 -m 512 --no-pvtime --disk guest-disk.img --restricted_mem --virtio-transport pci --pmu --network mode=user
+```
+
+You should see the realm boot. Note that `lkvm` is invoked with `--network
+mode=user`, which makes the guest see the network thru a VirtIO device.
+
+After boot up, which might take some time, you will be prompted to log in at the
+guest Linux prompt. Use root again as the username:
+
+```output
+Starting syslogd: OK
+Starting klogd: OK
+Running sysctl: OK
+Starting network: udhcpc: started, v1.36.1
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 192.168.33.15, server 192.168.33.1
+udhcpc: lease of 192.168.33.15 obtained from 192.168.33.1, lease time 14400
+deleting routers
+adding dns 172.20.51.254
+OK
+
+Welcome to the CCA realm
+realm login: root
+(realm) #
+```
+
+### Observe bounce buffer usage in the realm
+
+First, check that the Linux kernel has tracing support:
+
+```bash { output_lines="2-46" }
+ls /sys/kernel/debug/tracing/events/
+9p                       i2c_slave                qcom_glink
+alarmtimer               icmp                     qcom_smp2p
+asoc                     initcall                 qdisc
+block                    interconnect             ras
+bpf_test_run             io_uring                 raw_syscalls
+bpf_trace                iomap                    rcu
+bridge                   iommu                    regmap
+capability               ipi                      regulator
+cgroup                   irq                      rpcgss
+chipidea                 jbd2                     rpm
+clk                      kmem                     rpmh
+cma                      ksm                      rseq
+compaction               kvm                      rtc
+cpuhp                    kyber                    sched
+cros_ec                  libata                   scmi
+csd                      lock                     scsi
+dev                      lockd                    signal
+devfreq                  maple_tree               skb
+devlink                  mdio                     smbus
+dma                      memcg                    sock
+dma_fence                migrate                  spi
+dpaa2_eth                mmap                     spmi
+dpaa_eth                 mmap_lock                sunrpc
+dwc3                     mmc                      swiotlb
+e1000e_trace             module                   task
+enable                   mtu3                     tcp
+error_report             musb                     tegra_apb_dma
+ext4                     napi                     thermal
+fib                      neigh                    thermal_power_allocator
+filelock                 net                      thp
+filemap                  netfs                    timer
+fsl_edma                 netlink                  timer_migration
+ftrace                   nfs                      timestamp
+gadget                   nfs4                     tlb
+gpio                     notifier                 udp
+gpu_mem                  oom                      ufs
+handshake                optee                    vmalloc
+header_event             page_isolation           vmscan
+header_page              page_pool                watchdog
+hns3                     pagemap                  workqueue
+huge_memory              percpu                   writeback
+hugetlbfs                power                    xdp
+hw_pressure              printk                   xhci-hcd
+hwmon                    pwm
+i2c                      qcom_aoss
+```
+
+As shown in the above transcript, you should get a list of the available trace
+points.
+
+Now, enable the kernel tracing infrastructure together with the bounce buffer
+tracing, read the trace in the background (filtering on `keybroker-app-`) and
+run the Key Broker Client application in the realm, using the endpoint address
+that the Key Broker Server is listening on (from the other terminal):
+
+```bash
+echo 1 > /sys/kernel/debug/tracing/tracing_on
+echo 1 > /sys/kernel/debug/tracing/events/swiotlb/enable
+grep keybroker-app- /sys/kernel/debug/tracing/trace_pipe &
+keybroker-app -v --endpoint http://172.17.0.2:8088 skywalker
+```
+
+In the `keybroker-app`command above, `skywalker` is the key name that is
+requested from the KBS.
+
+You should get an output looking like:
+
+```output
+INFO Requesting key named 'skywalker' from the keybroker server with URL http://172.17.0.2:8088/keys/v1/key/skywalker
+INFO Challenge (64 bytes) = [5c, ec, 1e, f5, 93, 54, 4a, 8a, ee, 2e, 46, a0, 50, 0d, 41, dd, d4, 60, b0, 58, 5b, 51, 71, 76, d1, 66, d3, b7, 38, e8, af, ae, 0a, 07, 4e, c5, 60, dc, 4a, c0, b8, 73, 98, d9, bd, af, 41, 96, 99, 6d, 74, cc, 19, 70, 24, c4, c9, 5c, 21, 61, 1a, cb, 76, 75]
+INFO Submitting evidence to URL http://172.17.0.2:8088/keys/v1/evidence/1928844131
+INFO Attestation success :-) ! The key returned from the keybroker is 'May the force be with you.'
+   keybroker-app-143     [000] b..2.  1772.607321: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
+   keybroker-app-143     [000] b..2.  1772.644478: swiotlb_bounced: dev_name: 0000:00:00.0 dma_mask=ffffffffffffffff dev_addr=80b6717e size=66 FORCE
+```
+
+Note that the interleaving of the trace messages and KBC messages might differ
+from one run to another.