mm: numa: serialise parallel get_user_page against THP migration

Base pages are unmapped and flushed from cache and TLB during normal
page migration and replaced with a migration entry that causes any
parallel NUMA hinting fault or gup to block until migration completes.

THP does not unmap pages due to a lack of support for migration entries
at a PMD level.  This allows races with get_user_pages and
get_user_pages_fast which commit 3f926ab945b6 ("mm: Close races between
THP migration and PMD numa clearing") made worse by introducing a
pmd_clear_flush().

This patch forces get_user_page (fast and normal) on a pmd_numa page to
go through the slow get_user_page path where it will serialise against
THP migration and properly account for the NUMA hinting fault.  On the
migration side the page table lock is taken for each PTE update.

Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Cc: Alex Thorlton <athorlton@sgi.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 16 insertions, 8 deletions

diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 33a5dc492810..51f069303ab9 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1243,6 +1243,10 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma,
         if ((flags & FOLL_DUMP) && is_huge_zero_pmd(*pmd))
                 return ERR_PTR(-EFAULT);
 
+        /* Full NUMA hinting faults to serialise migration in fault paths */
+        if ((flags & FOLL_NUMA) && pmd_numa(*pmd))
+                goto out;
+
         page = pmd_page(*pmd);
         VM_BUG_ON(!PageHead(page));
         if (flags & FOLL_TOUCH) {
@@ -1323,23 +1327,27 @@ int do_huge_pmd_numa_page(struct mm_struct *mm, struct vm_area_struct *vma,
                 /* If the page was locked, there are no parallel migrations */
                 if (page_locked)
                         goto clear_pmdnuma;
+        }
 
-                /*
-                 * Otherwise wait for potential migrations and retry. We do
-                 * relock and check_same as the page may no longer be mapped.
-                 * As the fault is being retried, do not account for it.
-                 */
+        /*
+         * If there are potential migrations, wait for completion and retry. We
+         * do not relock and check_same as the page may no longer be mapped.
+         * Furtermore, even if the page is currently misplaced, there is no
+         * guarantee it is still misplaced after the migration completes.
+         */
+        if (!page_locked) {
                 spin_unlock(ptl);
                 wait_on_page_locked(page);
                 page_nid = -1;
                 goto out;
         }
 
-        /* Page is misplaced, serialise migrations and parallel THP splits */
+        /*
+         * Page is misplaced. Page lock serialises migrations. Acquire anon_vma
+         * to serialises splits
+         */
         get_page(page);
         spin_unlock(ptl);
-        if (!page_locked)
-                lock_page(page);
         anon_vma = page_lock_anon_vma_read(page);
 
         /* Confirm the PMD did not change while page_table_lock was released */