mm: introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud

jira LE-3557
Rebuild_History Non-Buildable kernel-5.14.0-570.26.1.el9_6
commit-author Peter Xu <[email protected]>
commit 6857be5
Empty-Commit: Cherry-Pick Conflicts during history rebuild.
Will be included in final tarball splat. Ref for failed cherry-pick at:
ciq/ciq_backports/kernel-5.14.0-570.26.1.el9_6/6857be5f.failed

Patch series "mm: Support huge pfnmaps", v2.

Overview
========

This series implements huge pfnmaps support for mm in general.  Huge
pfnmap allows e.g.  VM_PFNMAP vmas to map in either PMD or PUD levels,
similar to what we do with dax / thp / hugetlb so far to benefit from TLB
hits.  Now we extend that idea to PFN mappings, e.g.  PCI MMIO bars where
it can grow as large as 8GB or even bigger.

Currently, only x86_64 (1G+2M) and arm64 (2M) are supported.  The last
patch (from Alex Williamson) will be the first user of huge pfnmap, so as
to enable vfio-pci driver to fault in huge pfn mappings.

Implementation
==============

In reality, it's relatively simple to add such support comparing to many
other types of mappings, because of PFNMAP's specialties when there's no
vmemmap backing it, so that most of the kernel routines on huge mappings
should simply already fail for them, like GUPs or old-school follow_page()
(which is recently rewritten to be folio_walk* APIs by David).

One trick here is that we're still unmature on PUDs in generic paths here
and there, as DAX is so far the only user.  This patchset will add the 2nd
user of it.  Hugetlb can be a 3rd user if the hugetlb unification work can
go on smoothly, but to be discussed later.

The other trick is how to allow gup-fast working for such huge mappings
even if there's no direct sign of knowing whether it's a normal page or
MMIO mapping.  This series chose to keep the pte_special solution, so that
it reuses similar idea on setting a special bit to pfnmap PMDs/PUDs so
that gup-fast will be able to identify them and fail properly.

Along the way, we'll also notice that the major pgtable pfn walker, aka,
follow_pte(), will need to retire soon due to the fact that it only works
with ptes.  A new set of simple API is introduced (follow_pfnmap* API) to
be able to do whatever follow_pte() can already do, plus that it can also
process huge pfnmaps now.  Half of this series is about that and
converting all existing pfnmap walkers to use the new API properly.
Hopefully the new API also looks better to avoid exposing e.g.  pgtable
lock details into the callers, so that it can be used in an even more
straightforward way.

Here, three more options will be introduced and involved in huge pfnmap:

  - ARCH_SUPPORTS_HUGE_PFNMAP

    Arch developers will need to select this option when huge pfnmap is
    supported in arch's Kconfig.  After this patchset applied, both x86_64
    and arm64 will start to enable it by default.

  - ARCH_SUPPORTS_PMD_PFNMAP / ARCH_SUPPORTS_PUD_PFNMAP

    These options are for driver developers to identify whether current
    arch / config supports huge pfnmaps, making decision on whether it can
    use the huge pfnmap APIs to inject them.  One can refer to the last
    vfio-pci patch from Alex on the use of them properly in a device
    driver.

So after the whole set applied, and if one would enable some dynamic debug
lines in vfio-pci core files, we should observe things like:

  vfio-pci 0000:00:06.0: vfio_pci_mmap_huge_fault(,order = 9) BAR 0 page offset 0x0: 0x100
  vfio-pci 0000:00:06.0: vfio_pci_mmap_huge_fault(,order = 9) BAR 0 page offset 0x200: 0x100
  vfio-pci 0000:00:06.0: vfio_pci_mmap_huge_fault(,order = 9) BAR 0 page offset 0x400: 0x100

In this specific case, it says that vfio-pci faults in PMDs properly for a
few BAR0 offsets.

Patch Layout
============

Patch 1:         Introduce the new options mentioned above for huge PFNMAPs
Patch 2:         A tiny cleanup
Patch 3-8:       Preparation patches for huge pfnmap (include introduce
                 special bit for pmd/pud)
Patch 9-16:      Introduce follow_pfnmap*() API, use it everywhere, and
                 then drop follow_pte() API
Patch 17:        Add huge pfnmap support for x86_64
Patch 18:        Add huge pfnmap support for arm64
Patch 19:        Add vfio-pci support for all kinds of huge pfnmaps (Alex)

TODO
====

More architectures / More page sizes
------------------------------------

Currently only x86_64 (2M+1G) and arm64 (2M) are supported.  There seems
to have plan to support arm64 1G later on top of this series [2].

Any arch will need to first support THP / THP_1G, then provide a special
bit in pmds/puds to support huge pfnmaps.

remap_pfn_range() support
-------------------------

Currently, remap_pfn_range() still only maps PTEs.  With the new option,
remap_pfn_range() can logically start to inject either PMDs or PUDs when
the alignment requirements match on the VAs.

When the support is there, it should be able to silently benefit all
drivers that is using remap_pfn_range() in its mmap() handler on better
TLB hit rate and overall faster MMIO accesses similar to processor on
hugepages.

More driver support
-------------------

VFIO is so far the only consumer for the huge pfnmaps after this series
applied.  Besides above remap_pfn_range() generic optimization, device
driver can also try to optimize its mmap() on a better VA alignment for
either PMD/PUD sizes.  This may, iiuc, normally require userspace changes,
as the driver doesn't normally decide the VA to map a bar.  But I don't
think I know all the drivers to know the full picture.

Credits all go to Alex on help testing the GPU/NIC use cases above.

[0] https://lore.kernel.org/r/[email protected]
[1] https://lore.kernel.org/r/[email protected]
[2] https://lore.kernel.org/r/[email protected]

This patch (of 19):

This patch introduces the option to introduce special pte bit into
pmd/puds.  Archs can start to define pmd_special / pud_special when
supported by selecting the new option.  Per-arch support will be added
later.

Before that, create fallbacks for these helpers so that they are always
available.

Link: https://lkml.kernel.org/r/[email protected]
Link: https://lkml.kernel.org/r/[email protected]
	Signed-off-by: Peter Xu <[email protected]>
	Cc: Alexander Gordeev <[email protected]>
	Cc: Alex Williamson <[email protected]>
	Cc: Aneesh Kumar K.V <[email protected]>
	Cc: Borislav Petkov <[email protected]>
	Cc: Catalin Marinas <[email protected]>
	Cc: Christian Borntraeger <[email protected]>
	Cc: Dave Hansen <[email protected]>
	Cc: David Hildenbrand <[email protected]>
	Cc: Gavin Shan <[email protected]>
	Cc: Gerald Schaefer <[email protected]>
	Cc: Heiko Carstens <[email protected]>
	Cc: Ingo Molnar <[email protected]>
	Cc: Jason Gunthorpe <[email protected]>
	Cc: Matthew Wilcox <[email protected]>
	Cc: Niklas Schnelle <[email protected]>
	Cc: Paolo Bonzini <[email protected]>
	Cc: Ryan Roberts <[email protected]>
	Cc: Sean Christopherson <[email protected]>
	Cc: Sven Schnelle <[email protected]>
	Cc: Thomas Gleixner <[email protected]>
	Cc: Vasily Gorbik <[email protected]>
	Cc: Will Deacon <[email protected]>
	Cc: Zi Yan <[email protected]>
	Signed-off-by: Andrew Morton <[email protected]>
(cherry picked from commit 6857be5)
	Signed-off-by: Jonathan Maple <[email protected]>

# Conflicts:
#	mm/Kconfig

Author: PlaidCat

ciq/ciq_backports/kernel-5.14.0-570.26.1.el9_6/6857be5f.failed

@@ -0,0 +1,246 @@
+mm: introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud
+
+jira LE-3557
+Rebuild_History Non-Buildable kernel-5.14.0-570.26.1.el9_6
+commit-author Peter Xu <[email protected]>
+commit 6857be5fecaebd9773ff27b6d29b6fff3b1abbce
+Empty-Commit: Cherry-Pick Conflicts during history rebuild.
+Will be included in final tarball splat. Ref for failed cherry-pick at:
+ciq/ciq_backports/kernel-5.14.0-570.26.1.el9_6/6857be5f.failed
+
+Patch series "mm: Support huge pfnmaps", v2.
+
+Overview
+========
+
+This series implements huge pfnmaps support for mm in general.  Huge
+pfnmap allows e.g.  VM_PFNMAP vmas to map in either PMD or PUD levels,
+similar to what we do with dax / thp / hugetlb so far to benefit from TLB
+hits.  Now we extend that idea to PFN mappings, e.g.  PCI MMIO bars where
+it can grow as large as 8GB or even bigger.
+
+Currently, only x86_64 (1G+2M) and arm64 (2M) are supported.  The last
+patch (from Alex Williamson) will be the first user of huge pfnmap, so as
+to enable vfio-pci driver to fault in huge pfn mappings.
+
+Implementation
+==============
+
+In reality, it's relatively simple to add such support comparing to many
+other types of mappings, because of PFNMAP's specialties when there's no
+vmemmap backing it, so that most of the kernel routines on huge mappings
+should simply already fail for them, like GUPs or old-school follow_page()
+(which is recently rewritten to be folio_walk* APIs by David).
+
+One trick here is that we're still unmature on PUDs in generic paths here
+and there, as DAX is so far the only user.  This patchset will add the 2nd
+user of it.  Hugetlb can be a 3rd user if the hugetlb unification work can
+go on smoothly, but to be discussed later.
+
+The other trick is how to allow gup-fast working for such huge mappings
+even if there's no direct sign of knowing whether it's a normal page or
+MMIO mapping.  This series chose to keep the pte_special solution, so that
+it reuses similar idea on setting a special bit to pfnmap PMDs/PUDs so
+that gup-fast will be able to identify them and fail properly.
+
+Along the way, we'll also notice that the major pgtable pfn walker, aka,
+follow_pte(), will need to retire soon due to the fact that it only works
+with ptes.  A new set of simple API is introduced (follow_pfnmap* API) to
+be able to do whatever follow_pte() can already do, plus that it can also
+process huge pfnmaps now.  Half of this series is about that and
+converting all existing pfnmap walkers to use the new API properly. 
+Hopefully the new API also looks better to avoid exposing e.g.  pgtable
+lock details into the callers, so that it can be used in an even more
+straightforward way.
+
+Here, three more options will be introduced and involved in huge pfnmap:
+
+  - ARCH_SUPPORTS_HUGE_PFNMAP
+
+    Arch developers will need to select this option when huge pfnmap is
+    supported in arch's Kconfig.  After this patchset applied, both x86_64
+    and arm64 will start to enable it by default.
+
+  - ARCH_SUPPORTS_PMD_PFNMAP / ARCH_SUPPORTS_PUD_PFNMAP
+
+    These options are for driver developers to identify whether current
+    arch / config supports huge pfnmaps, making decision on whether it can
+    use the huge pfnmap APIs to inject them.  One can refer to the last
+    vfio-pci patch from Alex on the use of them properly in a device
+    driver.
+
+So after the whole set applied, and if one would enable some dynamic debug
+lines in vfio-pci core files, we should observe things like:
+
+  vfio-pci 0000:00:06.0: vfio_pci_mmap_huge_fault(,order = 9) BAR 0 page offset 0x0: 0x100
+  vfio-pci 0000:00:06.0: vfio_pci_mmap_huge_fault(,order = 9) BAR 0 page offset 0x200: 0x100
+  vfio-pci 0000:00:06.0: vfio_pci_mmap_huge_fault(,order = 9) BAR 0 page offset 0x400: 0x100
+
+In this specific case, it says that vfio-pci faults in PMDs properly for a
+few BAR0 offsets.
+
+Patch Layout
+============
+
+Patch 1:         Introduce the new options mentioned above for huge PFNMAPs
+Patch 2:         A tiny cleanup
+Patch 3-8:       Preparation patches for huge pfnmap (include introduce
+                 special bit for pmd/pud)
+Patch 9-16:      Introduce follow_pfnmap*() API, use it everywhere, and
+                 then drop follow_pte() API
+Patch 17:        Add huge pfnmap support for x86_64
+Patch 18:        Add huge pfnmap support for arm64
+Patch 19:        Add vfio-pci support for all kinds of huge pfnmaps (Alex)
+
+TODO
+====
+
+More architectures / More page sizes
+------------------------------------
+
+Currently only x86_64 (2M+1G) and arm64 (2M) are supported.  There seems
+to have plan to support arm64 1G later on top of this series [2].
+
+Any arch will need to first support THP / THP_1G, then provide a special
+bit in pmds/puds to support huge pfnmaps.
+
+remap_pfn_range() support
+-------------------------
+
+Currently, remap_pfn_range() still only maps PTEs.  With the new option,
+remap_pfn_range() can logically start to inject either PMDs or PUDs when
+the alignment requirements match on the VAs.
+
+When the support is there, it should be able to silently benefit all
+drivers that is using remap_pfn_range() in its mmap() handler on better
+TLB hit rate and overall faster MMIO accesses similar to processor on
+hugepages.
+
+More driver support
+-------------------
+
+VFIO is so far the only consumer for the huge pfnmaps after this series
+applied.  Besides above remap_pfn_range() generic optimization, device
+driver can also try to optimize its mmap() on a better VA alignment for
+either PMD/PUD sizes.  This may, iiuc, normally require userspace changes,
+as the driver doesn't normally decide the VA to map a bar.  But I don't
+think I know all the drivers to know the full picture.
+
+Credits all go to Alex on help testing the GPU/NIC use cases above.
+
+[0] https://lore.kernel.org/r/[email protected]
+[1] https://lore.kernel.org/r/[email protected]
+[2] https://lore.kernel.org/r/[email protected]
+
+
+This patch (of 19):
+
+This patch introduces the option to introduce special pte bit into
+pmd/puds.  Archs can start to define pmd_special / pud_special when
+supported by selecting the new option.  Per-arch support will be added
+later.
+
+Before that, create fallbacks for these helpers so that they are always
+available.
+
+Link: https://lkml.kernel.org/r/[email protected]
+Link: https://lkml.kernel.org/r/[email protected]
+	Signed-off-by: Peter Xu <[email protected]>
+	Cc: Alexander Gordeev <[email protected]>
+	Cc: Alex Williamson <[email protected]>
+	Cc: Aneesh Kumar K.V <[email protected]>
+	Cc: Borislav Petkov <[email protected]>
+	Cc: Catalin Marinas <[email protected]>
+	Cc: Christian Borntraeger <[email protected]>
+	Cc: Dave Hansen <[email protected]>
+	Cc: David Hildenbrand <[email protected]>
+	Cc: Gavin Shan <[email protected]>
+	Cc: Gerald Schaefer <[email protected]>
+	Cc: Heiko Carstens <[email protected]>
+	Cc: Ingo Molnar <[email protected]>
+	Cc: Jason Gunthorpe <[email protected]>
+	Cc: Matthew Wilcox <[email protected]>
+	Cc: Niklas Schnelle <[email protected]>
+	Cc: Paolo Bonzini <[email protected]>
+	Cc: Ryan Roberts <[email protected]>
+	Cc: Sean Christopherson <[email protected]>
+	Cc: Sven Schnelle <[email protected]>
+	Cc: Thomas Gleixner <[email protected]>
+	Cc: Vasily Gorbik <[email protected]>
+	Cc: Will Deacon <[email protected]>
+	Cc: Zi Yan <[email protected]>
+	Signed-off-by: Andrew Morton <[email protected]>
+(cherry picked from commit 6857be5fecaebd9773ff27b6d29b6fff3b1abbce)
+	Signed-off-by: Jonathan Maple <[email protected]>
+
+# Conflicts:
+#	mm/Kconfig
+diff --cc mm/Kconfig
+index a91823e31f45,1aa282e35dc7..000000000000
+--- a/mm/Kconfig
++++ b/mm/Kconfig
+@@@ -898,6 -870,25 +898,28 @@@ config READ_ONLY_THP_FOR_F
+  endif # TRANSPARENT_HUGEPAGE
+  
+  #
+++<<<<<<< HEAD
+++=======
++ # The architecture supports pgtable leaves that is larger than PAGE_SIZE
++ #
++ config PGTABLE_HAS_HUGE_LEAVES
++ 	def_bool TRANSPARENT_HUGEPAGE || HUGETLB_PAGE
++ 
++ # TODO: Allow to be enabled without THP
++ config ARCH_SUPPORTS_HUGE_PFNMAP
++ 	def_bool n
++ 	depends on TRANSPARENT_HUGEPAGE
++ 
++ config ARCH_SUPPORTS_PMD_PFNMAP
++ 	def_bool y
++ 	depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE
++ 
++ config ARCH_SUPPORTS_PUD_PFNMAP
++ 	def_bool y
++ 	depends on ARCH_SUPPORTS_HUGE_PFNMAP && HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD
++ 
++ #
+++>>>>>>> 6857be5fecae (mm: introduce ARCH_SUPPORTS_HUGE_PFNMAP and special bits to pmd/pud)
+  # UP and nommu archs use km based percpu allocator
+  #
+  config NEED_PER_CPU_KM
+diff --git a/include/linux/mm.h b/include/linux/mm.h
+index 196c481ec160..7b6f347d05b9 100644
+--- a/include/linux/mm.h
++++ b/include/linux/mm.h
+@@ -2730,6 +2730,30 @@ static inline pte_t pte_mkspecial(pte_t pte)
+ }
+ #endif
+ 
++#ifndef CONFIG_ARCH_SUPPORTS_PMD_PFNMAP
++static inline bool pmd_special(pmd_t pmd)
++{
++	return false;
++}
++
++static inline pmd_t pmd_mkspecial(pmd_t pmd)
++{
++	return pmd;
++}
++#endif	/* CONFIG_ARCH_SUPPORTS_PMD_PFNMAP */
++
++#ifndef CONFIG_ARCH_SUPPORTS_PUD_PFNMAP
++static inline bool pud_special(pud_t pud)
++{
++	return false;
++}
++
++static inline pud_t pud_mkspecial(pud_t pud)
++{
++	return pud;
++}
++#endif	/* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */
++
+ #ifndef CONFIG_ARCH_HAS_PTE_DEVMAP
+ static inline int pte_devmap(pte_t pte)
+ {
+* Unmerged path mm/Kconfig