mm, thp: avoid unlikely branches for split_huge_pmd

While doing MADV_DONTNEED on a large area of thp memory, I noticed we
encountered many unlikely() branches in profiles for each backing
hugepage.  This is because zap_pmd_range() would call split_huge_pmd(),
which rechecked the conditions that were already validated, but as part
of an unlikely() branch.

Avoid the unlikely() branch when in a context where pmd is known to be
good for __split_huge_pmd() directly.

Link: http://lkml.kernel.org/r/alpine.DEB.2.10.1610181600300.84525@chino.kir.corp.google.com
Signed-off-by: David Rientjes <rientjes@google.com>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 2 insertions, 2 deletions

diff --git a/mm/memory.c b/mm/memory.c
index 33f45edf8272..d86b7b4afd7d 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1240,7 +1240,7 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb,
                         if (next - addr != HPAGE_PMD_SIZE) {
                                 VM_BUG_ON_VMA(vma_is_anonymous(vma) &&
                                     !rwsem_is_locked(&tlb->mm->mmap_sem), vma);
-                                split_huge_pmd(vma, pmd, addr);
+                                __split_huge_pmd(vma, pmd, addr, false, NULL);
                         } else if (zap_huge_pmd(tlb, vma, pmd, addr))
                                 goto next;
                         /* fall through */
@@ -3454,7 +3454,7 @@ static int wp_huge_pmd(struct fault_env *fe, pmd_t orig_pmd)
 
         /* COW handled on pte level: split pmd */
         VM_BUG_ON_VMA(fe->vma->vm_flags & VM_SHARED, fe->vma);
-        split_huge_pmd(fe->vma, fe->pmd, fe->address);
+        __split_huge_pmd(fe->vma, fe->pmd, fe->address, false, NULL);
 
         return VM_FAULT_FALLBACK;
 }