[PATCH] mm: split page table lock

Christoph Lameter demonstrated very poor scalability on the SGI 512-way, with
a many-threaded application which concurrently initializes different parts of
a large anonymous area.

This patch corrects that, by using a separate spinlock per page table page, to
guard the page table entries in that page, instead of using the mm's single
page_table_lock.  (But even then, page_table_lock is still used to guard page
table allocation, and anon_vma allocation.)

In this implementation, the spinlock is tucked inside the struct page of the
page table page: with a BUILD_BUG_ON in case it overflows - which it would in
the case of 32-bit PA-RISC with spinlock debugging enabled.

Splitting the lock is not quite for free: another cacheline access.  Ideally,
I suppose we would use split ptlock only for multi-threaded processes on
multi-cpu machines; but deciding that dynamically would have its own costs.
So for now enable it by config, at some number of cpus - since the Kconfig
language doesn't support inequalities, let preprocessor compare that with
NR_CPUS.  But I don't think it's worth being user-configurable: for good
testing of both split and unsplit configs, split now at 4 cpus, and perhaps
change that to 8 later.

There is a benefit even for singly threaded processes: kswapd can be attacking
one part of the mm while another part is busy faulting.

Signed-off-by: Hugh Dickins <hugh@veritas.com>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 6 insertions, 6 deletions

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 510f0039b000..8970c0b74194 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -61,7 +61,7 @@ void swap_unplug_io_fn(struct backing_dev_info *unused_bdi, struct page *page)
         swp_entry_t entry;
 
         down_read(&swap_unplug_sem);
-        entry.val = page->private;
+        entry.val = page_private(page);
         if (PageSwapCache(page)) {
                 struct block_device *bdev = swap_info[swp_type(entry)].bdev;
                 struct backing_dev_info *bdi;
@@ -69,8 +69,8 @@ void swap_unplug_io_fn(struct backing_dev_info *unused_bdi, struct page *page)
                 /*
                  * If the page is removed from swapcache from under us (with a
                  * racy try_to_unuse/swapoff) we need an additional reference
-                 * count to avoid reading garbage from page->private above. If
-                 * the WARN_ON triggers during a swapoff it maybe the race
+                 * count to avoid reading garbage from page_private(page) above.
+                 * If the WARN_ON triggers during a swapoff it maybe the race
                  * condition and it's harmless. However if it triggers without
                  * swapoff it signals a problem.
                  */
@@ -294,7 +294,7 @@ static inline int page_swapcount(struct page *page)
         struct swap_info_struct *p;
         swp_entry_t entry;
 
-        entry.val = page->private;
+        entry.val = page_private(page);
         p = swap_info_get(entry);
         if (p) {
                 /* Subtract the 1 for the swap cache itself */
@@ -339,7 +339,7 @@ int remove_exclusive_swap_page(struct page *page)
         if (page_count(page) != 2) /* 2: us + cache */
                 return 0;
 
-        entry.val = page->private;
+        entry.val = page_private(page);
         p = swap_info_get(entry);
         if (!p)
                 return 0;
@@ -1042,7 +1042,7 @@ int page_queue_congested(struct page *page)
         BUG_ON(!PageLocked(page));      /* It pins the swap_info_struct */
 
         if (PageSwapCache(page)) {
-                swp_entry_t entry = { .val = page->private };
+                swp_entry_t entry = { .val = page_private(page) };
                 struct swap_info_struct *sis;
 
                 sis = get_swap_info_struct(swp_type(entry));