[PATCH] shared page table for hugetlb page

Following up with the work on shared page table done by Dave McCracken.  This
set of patch target shared page table for hugetlb memory only.

The shared page table is particular useful in the situation of large number of
independent processes sharing large shared memory segments.  In the normal
page case, the amount of memory saved from process' page table is quite
significant.  For hugetlb, the saving on page table memory is not the primary
objective (as hugetlb itself already cuts down page table overhead
significantly), instead, the purpose of using shared page table on hugetlb is
to allow faster TLB refill and smaller cache pollution upon TLB miss.

With PT sharing, pte entries are shared among hundreds of processes, the cache
consumption used by all the page table is smaller and in return, application
gets much higher cache hit ratio.  One other effect is that cache hit ratio
with hardware page walker hitting on pte in cache will be higher and this
helps to reduce tlb miss latency.  These two effects contribute to higher
application performance.

Signed-off-by: Ken Chen <kenneth.w.chen@intel.com>
Acked-by: Hugh Dickins <hugh@veritas.com>
Cc: Dave McCracken <dmccr@us.ibm.com>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: David Gibson <david@gibson.dropbear.id.au>
Cc: Adam Litke <agl@us.ibm.com>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: "David S. Miller" <davem@davemloft.net>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 7 insertions, 0 deletions

diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index f7355bf2f285..9244971b6791 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -386,6 +386,9 @@ void __unmap_hugepage_range(struct vm_area_struct *vma, unsigned long start,
                 if (!ptep)
                         continue;
 
+                if (huge_pmd_unshare(mm, &address, ptep))
+                        continue;
+
                 pte = huge_ptep_get_and_clear(mm, address, ptep);
                 if (pte_none(pte))
                         continue;
@@ -658,11 +661,14 @@ void hugetlb_change_protection(struct vm_area_struct *vma,
         BUG_ON(address >= end);
         flush_cache_range(vma, address, end);
 
+        spin_lock(&vma->vm_file->f_mapping->i_mmap_lock);
         spin_lock(&mm->page_table_lock);
         for (; address < end; address += HPAGE_SIZE) {
                 ptep = huge_pte_offset(mm, address);
                 if (!ptep)
                         continue;
+                if (huge_pmd_unshare(mm, &address, ptep))
+                        continue;
                 if (!pte_none(*ptep)) {
                         pte = huge_ptep_get_and_clear(mm, address, ptep);
                         pte = pte_mkhuge(pte_modify(pte, newprot));
@@ -671,6 +677,7 @@ void hugetlb_change_protection(struct vm_area_struct *vma,
                 }
         }
         spin_unlock(&mm->page_table_lock);
+        spin_unlock(&vma->vm_file->f_mapping->i_mmap_lock);
 
         flush_tlb_range(vma, start, end);
 }