[PATCH] mm: ptd_alloc take ptlock

Second step in pushing down the page_table_lock.  Remove the temporary
bridging hack from __pud_alloc, __pmd_alloc, __pte_alloc: expect callers not
to hold page_table_lock, whether it's on init_mm or a user mm; take
page_table_lock internally to check if a racing task already allocated.

Convert their callers from common code.  But avoid coming back to change them
again later: instead of moving the spin_lock(&mm->page_table_lock) down,
switch over to new macros pte_alloc_map_lock and pte_unmap_unlock, which
encapsulate the mapping+locking and unlocking+unmapping together, and in the
end may use alternatives to the mm page_table_lock itself.

These callers all hold mmap_sem (some exclusively, some not), so at no level
can a page table be whipped away from beneath them; and pte_alloc uses the
"atomic" pmd_present to test whether it needs to allocate.  It appears that on
all arches we can safely descend without page_table_lock.

Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 18 insertions, 0 deletions

diff --git a/include/linux/mm.h b/include/linux/mm.h
index 22c2d6922c0e..d4c3512e7db4 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -779,10 +779,28 @@ static inline pmd_t *pmd_alloc(struct mm_struct *mm, pud_t *pud, unsigned long a
 }
 #endif /* CONFIG_MMU && !__ARCH_HAS_4LEVEL_HACK */
 
+#define pte_offset_map_lock(mm, pmd, address, ptlp)     \
+({                                                      \
+        spinlock_t *__ptl = &(mm)->page_table_lock;     \
+        pte_t *__pte = pte_offset_map(pmd, address);    \
+        *(ptlp) = __ptl;                                \
+        spin_lock(__ptl);                               \
+        __pte;                                          \
+})
+
+#define pte_unmap_unlock(pte, ptl)      do {            \
+        spin_unlock(ptl);                               \
+        pte_unmap(pte);                                 \
+} while (0)
+
 #define pte_alloc_map(mm, pmd, address)                 \
         ((unlikely(!pmd_present(*(pmd))) && __pte_alloc(mm, pmd, address))? \
                 NULL: pte_offset_map(pmd, address))
 
+#define pte_alloc_map_lock(mm, pmd, address, ptlp)      \
+        ((unlikely(!pmd_present(*(pmd))) && __pte_alloc(mm, pmd, address))? \
+                NULL: pte_offset_map_lock(mm, pmd, address, ptlp))
+
 #define pte_alloc_kernel(pmd, address)                  \
         ((unlikely(!pmd_present(*(pmd))) && __pte_alloc_kernel(pmd, address))? \
                 NULL: pte_offset_kernel(pmd, address))