mm/mmu_notifier: avoid call to invalidate_range() in range_end()

This is an optimization patch that only affect mmu_notifier users which
rely on the invalidate_range() callback.  This patch avoids calling that
callback twice in a row from inside __mmu_notifier_invalidate_range_end

Existing pattern (before this patch):
    mmu_notifier_invalidate_range_start()
        pte/pmd/pud_clear_flush_notify()
            mmu_notifier_invalidate_range()
    mmu_notifier_invalidate_range_end()
        mmu_notifier_invalidate_range()

New pattern (after this patch):
    mmu_notifier_invalidate_range_start()
        pte/pmd/pud_clear_flush_notify()
            mmu_notifier_invalidate_range()
    mmu_notifier_invalidate_range_only_end()

We call the invalidate_range callback after clearing the page table
under the page table lock and we skip the call to invalidate_range
inside the __mmu_notifier_invalidate_range_end() function.

Idea from Andrea Arcangeli

Link: http://lkml.kernel.org/r/20171017031003.7481-3-jglisse@redhat.com
Signed-off-by: Jérôme Glisse <jglisse@redhat.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Joerg Roedel <jroedel@suse.de>
Cc: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Cc: David Woodhouse <dwmw2@infradead.org>
Cc: Alistair Popple <alistair@popple.id.au>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Cc: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Cc: Nadav Amit <nadav.amit@gmail.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 12 insertions, 3 deletions

diff --git a/mm/migrate.c b/mm/migrate.c
index 1236449b4777..4d0be47a322a 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -2089,7 +2089,11 @@ int migrate_misplaced_transhuge_page(struct mm_struct *mm,
         set_page_owner_migrate_reason(new_page, MR_NUMA_MISPLACED);
 
         spin_unlock(ptl);
-        mmu_notifier_invalidate_range_end(mm, mmun_start, mmun_end);
+        /*
+         * No need to double call mmu_notifier->invalidate_range() callback as
+         * the above pmdp_huge_clear_flush_notify() did already call it.
+         */
+        mmu_notifier_invalidate_range_only_end(mm, mmun_start, mmun_end);
 
         /* Take an "isolate" reference and put new page on the LRU. */
         get_page(new_page);
@@ -2805,9 +2809,14 @@ static void migrate_vma_pages(struct migrate_vma *migrate)
                         migrate->src[i] &= ~MIGRATE_PFN_MIGRATE;
         }
 
+        /*
+         * No need to double call mmu_notifier->invalidate_range() callback as
+         * the above ptep_clear_flush_notify() inside migrate_vma_insert_page()
+         * did already call it.
+         */
         if (notified)
-                mmu_notifier_invalidate_range_end(mm, mmu_start,
-                                                  migrate->end);
+                mmu_notifier_invalidate_range_only_end(mm, mmu_start,
+                                                       migrate->end);
 }
 
 /*