vfio/type1: Use mapping page mask for pfnmaps

jira LE-3557
Rebuild_History Non-Buildable kernel-5.14.0-570.26.1.el9_6
commit-author Alex Williamson <[email protected]>
commit 0fd0684
Empty-Commit: Cherry-Pick Conflicts during history rebuild.
Will be included in final tarball splat. Ref for failed cherry-pick at:
ciq/ciq_backports/kernel-5.14.0-570.26.1.el9_6/0fd06844.failed

vfio-pci supports huge_fault for PCI MMIO BARs and will insert pud and
pmd mappings for well aligned mappings.  follow_pfnmap_start() walks the
page table and therefore knows the page mask of the level where the
address is found and returns this through follow_pfnmap_args.addr_mask.
Subsequent pfns from this address until the end of the mapping page are
necessarily consecutive.  Use this information to retrieve a range of
pfnmap pfns in a single pass.

With optimal mappings and alignment on systems with 1GB pud and 4KB
page size, this reduces iterations for DMA mapping PCI BARs by a
factor of 256K.  In real world testing, the overhead of iterating
pfns for a VM DMA mapping a 32GB PCI BAR is reduced from ~1s to
sub-millisecond overhead.

	Reviewed-by: Peter Xu <[email protected]>
	Reviewed-by: Mitchell Augustin <[email protected]>
	Tested-by: Mitchell Augustin <[email protected]>
	Reviewed-by: Jason Gunthorpe <[email protected]>
Link: https://lore.kernel.org/r/[email protected]
	Signed-off-by: Alex Williamson <[email protected]>
(cherry picked from commit 0fd0684)
	Signed-off-by: Jonathan Maple <[email protected]>

# Conflicts:
#	drivers/vfio/vfio_iommu_type1.c

Author: PlaidCat

ciq/ciq_backports/kernel-5.14.0-570.26.1.el9_6/0fd06844.failed

@@ -0,0 +1,83 @@
+vfio/type1: Use mapping page mask for pfnmaps
+
+jira LE-3557
+Rebuild_History Non-Buildable kernel-5.14.0-570.26.1.el9_6
+commit-author Alex Williamson <[email protected]>
+commit 0fd06844de5d063cb384384e06a11ec7141a35d5
+Empty-Commit: Cherry-Pick Conflicts during history rebuild.
+Will be included in final tarball splat. Ref for failed cherry-pick at:
+ciq/ciq_backports/kernel-5.14.0-570.26.1.el9_6/0fd06844.failed
+
+vfio-pci supports huge_fault for PCI MMIO BARs and will insert pud and
+pmd mappings for well aligned mappings.  follow_pfnmap_start() walks the
+page table and therefore knows the page mask of the level where the
+address is found and returns this through follow_pfnmap_args.addr_mask.
+Subsequent pfns from this address until the end of the mapping page are
+necessarily consecutive.  Use this information to retrieve a range of
+pfnmap pfns in a single pass.
+
+With optimal mappings and alignment on systems with 1GB pud and 4KB
+page size, this reduces iterations for DMA mapping PCI BARs by a
+factor of 256K.  In real world testing, the overhead of iterating
+pfns for a VM DMA mapping a 32GB PCI BAR is reduced from ~1s to
+sub-millisecond overhead.
+
+	Reviewed-by: Peter Xu <[email protected]>
+	Reviewed-by: Mitchell Augustin <[email protected]>
+	Tested-by: Mitchell Augustin <[email protected]>
+	Reviewed-by: Jason Gunthorpe <[email protected]>
+Link: https://lore.kernel.org/r/[email protected]
+	Signed-off-by: Alex Williamson <[email protected]>
+(cherry picked from commit 0fd06844de5d063cb384384e06a11ec7141a35d5)
+	Signed-off-by: Jonathan Maple <[email protected]>
+
+# Conflicts:
+#	drivers/vfio/vfio_iommu_type1.c
+diff --cc drivers/vfio/vfio_iommu_type1.c
+index 410214696525,0ac56072af9f..000000000000
+--- a/drivers/vfio/vfio_iommu_type1.c
++++ b/drivers/vfio/vfio_iommu_type1.c
+@@@ -523,14 -520,12 +523,14 @@@ static void vfio_batch_fini(struct vfio
+  
+  static int follow_fault_pfn(struct vm_area_struct *vma, struct mm_struct *mm,
+  			    unsigned long vaddr, unsigned long *pfn,
+- 			    bool write_fault)
++ 			    unsigned long *addr_mask, bool write_fault)
+  {
+ -	struct follow_pfnmap_args args = { .vma = vma, .address = vaddr };
+ +	pte_t *ptep;
+ +	pte_t pte;
+ +	spinlock_t *ptl;
+  	int ret;
+  
+ -	ret = follow_pfnmap_start(&args);
+ +	ret = follow_pte(vma->vm_mm, vaddr, &ptep, &ptl);
+  	if (ret) {
+  		bool unlocked = false;
+  
+@@@ -549,14 -544,14 +549,23 @@@
+  			return ret;
+  	}
+  
+++<<<<<<< HEAD
+ +	pte = ptep_get(ptep);
+ +
+ +	if (write_fault && !pte_write(pte))
+ +		ret = -EFAULT;
+ +	else
+ +		*pfn = pte_pfn(pte);
+++=======
++ 	if (write_fault && !args.writable) {
++ 		ret = -EFAULT;
++ 	} else {
++ 		*pfn = args.pfn;
++ 		*addr_mask = args.addr_mask;
++ 	}
+++>>>>>>> 0fd06844de5d (vfio/type1: Use mapping page mask for pfnmaps)
+  
+ -	follow_pfnmap_end(&args);
+ +	pte_unmap_unlock(ptep, ptl);
+  	return ret;
+  }
+  
+* Unmerged path drivers/vfio/vfio_iommu_type1.c