mm/vma: do not register private-anon mappings with khugepaged during mmap

commit 8c6ff7f
Author: Dev Jain <[email protected]>
Date:   Thu Mar 6 12:00:37 2025 +0530

    mm/vma: do not register private-anon mappings with khugepaged during mmap

    We already are registering private-anon VMAs with khugepaged during fault
    time, in do_huge_pmd_anonymous_page().  Commit "register suitable readonly
    file vmas for khugepaged" moved the khugepaged registration logic from
    shmem_mmap to the generic mmap path.

    The userspace-visible effect should be this: khugepaged will unnecessarily
    scan mm's which haven't yet faulted in.  Note that it won't actually
    collapse because all PTEs are none.

    Now that I think about it, the mm is going to have a file VMA anyways
    during fork+exec, so the mm already gets registered during mmap due to the
    non-anon case (I *think*), so at least one of either the mmap registration
    or fault-time registration is redundant.

    Make this logic specific for non-anon mappings.

    Link: https://lkml.kernel.org/r/[email protected]
    Fixes: 613bec0 ("mm: mmap: register suitable readonly file vmas for khugepaged")
    Signed-off-by: Dev Jain <[email protected]>
    Cc: "Aneesh Kumar K.V" <[email protected]>
    Cc: Anshuman Khandual <[email protected]>
    Cc: David Hildenbrand <[email protected]>
    Cc: Hugh Dickins <[email protected]>
    Cc: Jann Horn <[email protected]>
    Cc: Liam Howlett <[email protected]>
    Cc: Lorenzo Stoakes <[email protected]>
    Cc: Matthew Wilcow (Oracle) <[email protected]>
    Cc: Ryan Roberts <[email protected]>
    Cc: Vlastimil Babka <[email protected]>
    Cc: Yang Shi <[email protected]>
    Cc: Zi Yan <[email protected]>
    Signed-off-by: Andrew Morton <[email protected]>

JIRA: https://issues.redhat.com/browse/RHEL-77742
Signed-off-by: Nico Pache <[email protected]>

Author: PacheNico

mm/vma.c

@@ -2380,7 +2380,8 @@ static int __mmap_new_vma(struct mmap_state *map, struct vm_area_struct **vmap)
 	 * vma_merge_new_range() calls khugepaged_enter_vma() too, the below
 	 * call covers the non-merge case.
 	 */
-	khugepaged_enter_vma(vma, map->flags);
+	if (!vma_is_anonymous(vma))
+		khugepaged_enter_vma(vma, map->flags);
 	ksm_add_vma(vma);
 	*vmap = vma;
 	return 0;