Skip to content

Commit 1a65c39

Browse files
jgunthorpejgunthorpe
committed
Merge patch series "IOMMUFD Generic interface"
Jason Gunthorpe <[email protected]> says: ================== iommufd is the user API to control the IOMMU subsystem as it relates to managing IO page tables that point at user space memory. It takes over from drivers/vfio/vfio_iommu_type1.c (aka the VFIO container) which is the VFIO specific interface for a similar idea. We see a broad need for extended features, some being highly IOMMU device specific: - Binding iommu_domain's to PASID/SSID - Userspace IO page tables, for ARM, x86 and S390 - Kernel bypassed invalidation of user page tables - Re-use of the KVM page table in the IOMMU - Dirty page tracking in the IOMMU - Runtime Increase/Decrease of IOPTE size - PRI support with faults resolved in userspace Many of these HW features exist to support VM use cases - for instance the combination of PASID, PRI and Userspace IO Page Tables allows an implementation of DMA Shared Virtual Addressing (vSVA) within a guest. Dirty tracking enables VM live migration with SRIOV devices and PASID support allow creating "scalable IOV" devices, among other things. As these features are fundamental to a VM platform they need to be uniformly exposed to all the driver families that do DMA into VMs, which is currently VFIO and VDPA. The pre-v1 series proposed re-using the VFIO type 1 data structure, however it was suggested that if we are doing this big update then we should also come with an improved data structure that solves the limitations that VFIO type1 has. Notably this addresses: - Multiple IOAS/'containers' and multiple domains inside a single FD - Single-pin operation no matter how many domains and containers use a page - A fine grained locking scheme supporting user managed concurrency for multi-threaded map/unmap - A pre-registration mechanism to optimize vIOMMU use cases by pre-pinning pages - Extended ioctl API that can manage these new objects and exposes domains directly to user space - domains are sharable between subsystems, eg VFIO and VDPA The bulk of this code is a new data structure design to track how the IOVAs are mapped to PFNs. iommufd intends to be general and consumable by any driver that wants to DMA to userspace. From a driver perspective it can largely be dropped in in-place of iommu_attach_device() and provides a uniform full feature set to all consumers. As this is a larger project this series is the first step. This series provides the iommfd "generic interface" which is designed to be suitable for applications like DPDK and VMM flows that are not optimized to specific HW scenarios. It is close to being a drop in replacement for the existing VFIO type 1 and supports existing qemu based VM flows. Several follow-on series are being prepared: - Patches integrating with qemu in native mode: https://github.com/yiliu1765/qemu/commits/qemu-iommufd-6.0-rc2 - A completed integration with VFIO now exists that covers "emulated" mdev use cases now, and can pass testing with qemu/etc in compatability mode: https://github.com/jgunthorpe/linux/commits/vfio_iommufd - A draft providing system iommu dirty tracking on top of iommufd, including iommu driver implementations: https://github.com/jpemartins/linux/commits/x86-iommufd This pairs with patches for providing a similar API to support VFIO-device tracking to give a complete vfio solution: https://lore.kernel.org/kvm/[email protected]/ - Userspace page tables aka 'nested translation' for ARM and Intel iommu drivers: https://github.com/nicolinc/iommufd/commits/iommufd_nesting - "device centric" vfio series to expose the vfio_device FD directly as a normal cdev, and provide an extended API allowing dynamically changing the IOAS binding: https://github.com/yiliu1765/iommufd/commits/iommufd-v6.0-rc2-nesting-0901 - Drafts for PASID and PRI interfaces are included above as well Overall enough work is done now to show the merit of the new API design and at least draft solutions to many of the main problems. Several people have contributed directly to this work: Eric Auger, Joao Martins, Kevin Tian, Lu Baolu, Nicolin Chen, Yi L Liu. Many more have participated in the discussions that lead here, and provided ideas. Thanks to all! The v1/v2 iommufd series has been used to guide a large amount of preparatory work that has now been merged. The general theme is to organize things in a way that makes injecting iommufd natural: - VFIO live migration support with mlx5 and hisi_acc drivers. These series need a dirty tracking solution to be really usable. https://lore.kernel.org/kvm/[email protected]/ https://lore.kernel.org/kvm/[email protected]/ - Significantly rework the VFIO gvt mdev and remove struct mdev_parent_ops https://lore.kernel.org/lkml/[email protected]/ - Rework how PCIe no-snoop blocking works https://lore.kernel.org/kvm/[email protected]/ - Consolidate dma ownership into the iommu core code https://lore.kernel.org/linux-iommu/[email protected]/ - Make all vfio driver interfaces use struct vfio_device consistently https://lore.kernel.org/kvm/[email protected]/ - Remove the vfio_group from the kvm/vfio interface https://lore.kernel.org/kvm/[email protected]/ - Simplify locking in vfio https://lore.kernel.org/kvm/[email protected]/ - Remove the vfio notifiter scheme that faces drivers https://lore.kernel.org/kvm/[email protected]/ - Improve the driver facing API for vfio pin/unpin pages to make the presence of struct page clear https://lore.kernel.org/kvm/[email protected]/ - Clean up in the Intel IOMMU driver https://lore.kernel.org/linux-iommu/[email protected]/ https://lore.kernel.org/linux-iommu/[email protected]/ https://lore.kernel.org/linux-iommu/[email protected]/ https://lore.kernel.org/linux-iommu/[email protected]/ https://lore.kernel.org/linux-iommu/[email protected]/ - Rework s390 vfio drivers https://lore.kernel.org/kvm/[email protected]/ - Normalize vfio ioctl handling https://lore.kernel.org/kvm/[email protected]/ - VFIO API for dirty tracking (aka dma logging) managed inside a PCI device, with mlx5 implementation https://lore.kernel.org/kvm/[email protected] - Introduce a struct device sysfs presence for struct vfio_device https://lore.kernel.org/kvm/[email protected]/ - Complete restructuring the vfio mdev model https://lore.kernel.org/kvm/[email protected]/ - Isolate VFIO container code in preperation for iommufd to provide an alternative implementation of it all https://lore.kernel.org/kvm/[email protected] - Simplify and consolidate iommu_domain/device compatability checking https://lore.kernel.org/linux-iommu/[email protected]/ - Align iommu SVA support with the domain-centric model https://lore.kernel.org/all/[email protected]/ This is about 233 patches applied since March, thank you to everyone involved in all this work! Currently there are a number of supporting series still in progress: - DMABUF exporter support for VFIO to allow PCI P2P with VFIO https://lore.kernel.org/r/[email protected] - Start to provide iommu_domain ops for POWER https://lore.kernel.org/all/[email protected]/ However, these are not necessary for this series to advance. Syzkaller coverage has been merged and is now running in the syzbot environment on linux-next: google/syzkaller#3515 google/syzkaller#3521 ================== Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Jason Gunthorpe <[email protected]>
2 parents 69e61ed + 57f0988 commit 1a65c39

40 files changed

+10452
-37
lines changed

‎.clang-format‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -440,8 +440,11 @@ ForEachMacros:
440440
- 'inet_lhash2_for_each_icsk'
441441
- 'inet_lhash2_for_each_icsk_continue'
442442
- 'inet_lhash2_for_each_icsk_rcu'
443+
- 'interval_tree_for_each_double_span'
444+
- 'interval_tree_for_each_span'
443445
- 'intlist__for_each_entry'
444446
- 'intlist__for_each_entry_safe'
447+
- 'iopt_for_each_contig_area'
445448
- 'kcore_copy__for_each_phdr'
446449
- 'key_for_each'
447450
- 'key_for_each_safe'

‎Documentation/userspace-api/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ place where this information is gathered.
2525
ebpf/index
2626
ioctl/index
2727
iommu
28+
iommufd
2829
media/index
2930
netlink/index
3031
sysfs-platform_profile

‎Documentation/userspace-api/ioctl/ioctl-number.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,7 @@ Code Seq# Include File Comments
105105
'8' all SNP8023 advanced NIC card
106106
<mailto:[email protected]>
107107
';' 64-7F linux/vfio.h
108+
';' 80-FF linux/iommufd.h
108109
'=' 00-3f uapi/linux/ptp_clock.h <mailto:[email protected]>
109110
'@' 00-0F linux/radeonfb.h conflict!
110111
'@' 00-0F drivers/video/aty/aty128fb.c conflict!

‎Documentation/userspace-api/iommufd.rst‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
.. SPDX-License-Identifier: GPL-2.0+
2+
3+
=======
4+
IOMMUFD
5+
=======
6+
7+
:Author: Jason Gunthorpe
8+
:Author: Kevin Tian
9+
10+
Overview
11+
========
12+
13+
IOMMUFD is the user API to control the IOMMU subsystem as it relates to managing
14+
IO page tables from userspace using file descriptors. It intends to be general
15+
and consumable by any driver that wants to expose DMA to userspace. These
16+
drivers are eventually expected to deprecate any internal IOMMU logic
17+
they may already/historically implement (e.g. vfio_iommu_type1.c).
18+
19+
At minimum iommufd provides universal support of managing I/O address spaces and
20+
I/O page tables for all IOMMUs, with room in the design to add non-generic
21+
features to cater to specific hardware functionality.
22+
23+
In this context the capital letter (IOMMUFD) refers to the subsystem while the
24+
small letter (iommufd) refers to the file descriptors created via /dev/iommu for
25+
use by userspace.
26+
27+
Key Concepts
28+
============
29+
30+
User Visible Objects
31+
--------------------
32+
33+
Following IOMMUFD objects are exposed to userspace:
34+
35+
- IOMMUFD_OBJ_IOAS, representing an I/O address space (IOAS), allowing map/unmap
36+
of user space memory into ranges of I/O Virtual Address (IOVA).
37+
38+
The IOAS is a functional replacement for the VFIO container, and like the VFIO
39+
container it copies an IOVA map to a list of iommu_domains held within it.
40+
41+
- IOMMUFD_OBJ_DEVICE, representing a device that is bound to iommufd by an
42+
external driver.
43+
44+
- IOMMUFD_OBJ_HW_PAGETABLE, representing an actual hardware I/O page table
45+
(i.e. a single struct iommu_domain) managed by the iommu driver.
46+
47+
The IOAS has a list of HW_PAGETABLES that share the same IOVA mapping and
48+
it will synchronize its mapping with each member HW_PAGETABLE.
49+
50+
All user-visible objects are destroyed via the IOMMU_DESTROY uAPI.
51+
52+
The diagram below shows relationship between user-visible objects and kernel
53+
datastructures (external to iommufd), with numbers referred to operations
54+
creating the objects and links::
55+
56+
_________________________________________________________
57+
| iommufd |
58+
| [1] |
59+
| _________________ |
60+
| | | |
61+
| | | |
62+
| | | |
63+
| | | |
64+
| | | |
65+
| | | |
66+
| | | [3] [2] |
67+
| | | ____________ __________ |
68+
| | IOAS |<--| |<------| | |
69+
| | | |HW_PAGETABLE| | DEVICE | |
70+
| | | |____________| |__________| |
71+
| | | | | |
72+
| | | | | |
73+
| | | | | |
74+
| | | | | |
75+
| | | | | |
76+
| |_________________| | | |
77+
| | | | |
78+
|_________|___________________|___________________|_______|
79+
| | |
80+
| _____v______ _______v_____
81+
| PFN storage | | | |
82+
|------------>|iommu_domain| |struct device|
83+
|____________| |_____________|
84+
85+
1. IOMMUFD_OBJ_IOAS is created via the IOMMU_IOAS_ALLOC uAPI. An iommufd can
86+
hold multiple IOAS objects. IOAS is the most generic object and does not
87+
expose interfaces that are specific to single IOMMU drivers. All operations
88+
on the IOAS must operate equally on each of the iommu_domains inside of it.
89+
90+
2. IOMMUFD_OBJ_DEVICE is created when an external driver calls the IOMMUFD kAPI
91+
to bind a device to an iommufd. The driver is expected to implement a set of
92+
ioctls to allow userspace to initiate the binding operation. Successful
93+
completion of this operation establishes the desired DMA ownership over the
94+
device. The driver must also set the driver_managed_dma flag and must not
95+
touch the device until this operation succeeds.
96+
97+
3. IOMMUFD_OBJ_HW_PAGETABLE is created when an external driver calls the IOMMUFD
98+
kAPI to attach a bound device to an IOAS. Similarly the external driver uAPI
99+
allows userspace to initiate the attaching operation. If a compatible
100+
pagetable already exists then it is reused for the attachment. Otherwise a
101+
new pagetable object and iommu_domain is created. Successful completion of
102+
this operation sets up the linkages among IOAS, device and iommu_domain. Once
103+
this completes the device could do DMA.
104+
105+
Every iommu_domain inside the IOAS is also represented to userspace as a
106+
HW_PAGETABLE object.
107+
108+
.. note::
109+
110+
Future IOMMUFD updates will provide an API to create and manipulate the
111+
HW_PAGETABLE directly.
112+
113+
A device can only bind to an iommufd due to DMA ownership claim and attach to at
114+
most one IOAS object (no support of PASID yet).
115+
116+
Kernel Datastructure
117+
--------------------
118+
119+
User visible objects are backed by following datastructures:
120+
121+
- iommufd_ioas for IOMMUFD_OBJ_IOAS.
122+
- iommufd_device for IOMMUFD_OBJ_DEVICE.
123+
- iommufd_hw_pagetable for IOMMUFD_OBJ_HW_PAGETABLE.
124+
125+
Several terminologies when looking at these datastructures:
126+
127+
- Automatic domain - refers to an iommu domain created automatically when
128+
attaching a device to an IOAS object. This is compatible to the semantics of
129+
VFIO type1.
130+
131+
- Manual domain - refers to an iommu domain designated by the user as the
132+
target pagetable to be attached to by a device. Though currently there are
133+
no uAPIs to directly create such domain, the datastructure and algorithms
134+
are ready for handling that use case.
135+
136+
- In-kernel user - refers to something like a VFIO mdev that is using the
137+
IOMMUFD access interface to access the IOAS. This starts by creating an
138+
iommufd_access object that is similar to the domain binding a physical device
139+
would do. The access object will then allow converting IOVA ranges into struct
140+
page * lists, or doing direct read/write to an IOVA.
141+
142+
iommufd_ioas serves as the metadata datastructure to manage how IOVA ranges are
143+
mapped to memory pages, composed of:
144+
145+
- struct io_pagetable holding the IOVA map
146+
- struct iopt_area's representing populated portions of IOVA
147+
- struct iopt_pages representing the storage of PFNs
148+
- struct iommu_domain representing the IO page table in the IOMMU
149+
- struct iopt_pages_access representing in-kernel users of PFNs
150+
- struct xarray pinned_pfns holding a list of pages pinned by in-kernel users
151+
152+
Each iopt_pages represents a logical linear array of full PFNs. The PFNs are
153+
ultimately derived from userspace VAs via an mm_struct. Once they have been
154+
pinned the PFNs are stored in IOPTEs of an iommu_domain or inside the pinned_pfns
155+
xarray if they have been pinned through an iommufd_access.
156+
157+
PFN have to be copied between all combinations of storage locations, depending
158+
on what domains are present and what kinds of in-kernel "software access" users
159+
exist. The mechanism ensures that a page is pinned only once.
160+
161+
An io_pagetable is composed of iopt_areas pointing at iopt_pages, along with a
162+
list of iommu_domains that mirror the IOVA to PFN map.
163+
164+
Multiple io_pagetable-s, through their iopt_area-s, can share a single
165+
iopt_pages which avoids multi-pinning and double accounting of page
166+
consumption.
167+
168+
iommufd_ioas is sharable between subsystems, e.g. VFIO and VDPA, as long as
169+
devices managed by different subsystems are bound to a same iommufd.
170+
171+
IOMMUFD User API
172+
================
173+
174+
.. kernel-doc:: include/uapi/linux/iommufd.h
175+
176+
IOMMUFD Kernel API
177+
==================
178+
179+
The IOMMUFD kAPI is device-centric with group-related tricks managed behind the
180+
scene. This allows the external drivers calling such kAPI to implement a simple
181+
device-centric uAPI for connecting its device to an iommufd, instead of
182+
explicitly imposing the group semantics in its uAPI as VFIO does.
183+
184+
.. kernel-doc:: drivers/iommu/iommufd/device.c
185+
:export:
186+
187+
.. kernel-doc:: drivers/iommu/iommufd/main.c
188+
:export:
189+
190+
VFIO and IOMMUFD
191+
----------------
192+
193+
Connecting a VFIO device to iommufd can be done in two ways.
194+
195+
First is a VFIO compatible way by directly implementing the /dev/vfio/vfio
196+
container IOCTLs by mapping them into io_pagetable operations. Doing so allows
197+
the use of iommufd in legacy VFIO applications by symlinking /dev/vfio/vfio to
198+
/dev/iommufd or extending VFIO to SET_CONTAINER using an iommufd instead of a
199+
container fd.
200+
201+
The second approach directly extends VFIO to support a new set of device-centric
202+
user API based on aforementioned IOMMUFD kernel API. It requires userspace
203+
change but better matches the IOMMUFD API semantics and easier to support new
204+
iommufd features when comparing it to the first approach.
205+
206+
Currently both approaches are still work-in-progress.
207+
208+
There are still a few gaps to be resolved to catch up with VFIO type1, as
209+
documented in iommufd_vfio_check_extension().
210+
211+
Future TODOs
212+
============
213+
214+
Currently IOMMUFD supports only kernel-managed I/O page table, similar to VFIO
215+
type1. New features on the radar include:
216+
217+
- Binding iommu_domain's to PASID/SSID
218+
- Userspace page tables, for ARM, x86 and S390
219+
- Kernel bypass'd invalidation of user page tables
220+
- Re-use of the KVM page table in the IOMMU
221+
- Dirty page tracking in the IOMMU
222+
- Runtime Increase/Decrease of IOPTE size
223+
- PRI support with faults resolved in userspace

‎MAINTAINERS‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10717,6 +10717,18 @@ F: drivers/iommu/dma-iommu.h
1071710717
F: drivers/iommu/iova.c
1071810718
F: include/linux/iova.h
1071910719

10720+
IOMMUFD
10721+
M: Jason Gunthorpe <[email protected]>
10722+
M: Kevin Tian <[email protected]>
10723+
L: [email protected]
10724+
S: Maintained
10725+
T: git git://git.kernel.org/pub/scm/linux/kernel/git/jgg/iommufd.git
10726+
F: Documentation/userspace-api/iommufd.rst
10727+
F: drivers/iommu/iommufd/
10728+
F: include/linux/iommufd.h
10729+
F: include/uapi/linux/iommufd.h
10730+
F: tools/testing/selftests/iommu/
10731+
1072010732
IOMMU SUBSYSTEM
1072110733
M: Joerg Roedel <[email protected]>
1072210734
M: Will Deacon <[email protected]>

‎drivers/iommu/Kconfig‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -188,6 +188,7 @@ config MSM_IOMMU
188188

189189
source "drivers/iommu/amd/Kconfig"
190190
source "drivers/iommu/intel/Kconfig"
191+
source "drivers/iommu/iommufd/Kconfig"
191192

192193
config IRQ_REMAP
193194
bool "Support for Interrupt Remapping"

‎drivers/iommu/Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# SPDX-License-Identifier: GPL-2.0
2-
obj-y += amd/ intel/ arm/
2+
obj-y += amd/ intel/ arm/ iommufd/
33
obj-$(CONFIG_IOMMU_API) += iommu.o
44
obj-$(CONFIG_IOMMU_API) += iommu-traces.o
55
obj-$(CONFIG_IOMMU_API) += iommu-sysfs.o

‎drivers/iommu/amd/iommu.c‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2278,6 +2278,8 @@ static bool amd_iommu_capable(struct device *dev, enum iommu_cap cap)
22782278
return false;
22792279
case IOMMU_CAP_PRE_BOOT_PROTECTION:
22802280
return amdr_ivrs_remap_support;
2281+
case IOMMU_CAP_ENFORCE_CACHE_COHERENCY:
2282+
return true;
22812283
default:
22822284
break;
22832285
}

‎drivers/iommu/intel/iommu.c‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4450,14 +4450,20 @@ static bool intel_iommu_enforce_cache_coherency(struct iommu_domain *domain)
44504450

44514451
static bool intel_iommu_capable(struct device *dev, enum iommu_cap cap)
44524452
{
4453-
if (cap == IOMMU_CAP_CACHE_COHERENCY)
4453+
struct device_domain_info *info = dev_iommu_priv_get(dev);
4454+
4455+
switch (cap) {
4456+
case IOMMU_CAP_CACHE_COHERENCY:
44544457
return true;
4455-
if (cap == IOMMU_CAP_INTR_REMAP)
4458+
case IOMMU_CAP_INTR_REMAP:
44564459
return irq_remapping_enabled == 1;
4457-
if (cap == IOMMU_CAP_PRE_BOOT_PROTECTION)
4460+
case IOMMU_CAP_PRE_BOOT_PROTECTION:
44584461
return dmar_platform_optin();
4459-
4460-
return false;
4462+
case IOMMU_CAP_ENFORCE_CACHE_COHERENCY:
4463+
return ecap_sc_support(info->iommu->ecap);
4464+
default:
4465+
return false;
4466+
}
44614467
}
44624468

44634469
static struct iommu_device *intel_iommu_probe_device(struct device *dev)

0 commit comments

Comments
 (0)