[PATCH] mm: simplify vmscan vs release refcounting

The VM has an interesting race where a page refcount can drop to zero, but it
is still on the LRU lists for a short time.  This was solved by testing a 0->1
refcount transition when picking up pages from the LRU, and dropping the
refcount in that case.

Instead, use atomic_add_unless to ensure we never pick up a 0 refcount page
from the LRU, thus a 0 refcount page will never have its refcount elevated
until it is allocated again.

Signed-off-by: Nick Piggin <npiggin@suse.de>
Signed-off-by: Andrew Morton <akpm@osdl.org>
Signed-off-by: Linus Torvalds <torvalds@osdl.org>

1 files changed, 11 insertions, 14 deletions

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 8e477b1a4838..e21bab4deda6 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1083,29 +1083,26 @@ static int isolate_lru_pages(int nr_to_scan, struct list_head *src,
         int scan = 0;
 
         while (scan++ < nr_to_scan && !list_empty(src)) {
+                struct list_head *target;
                 page = lru_to_page(src);
                 prefetchw_prev_lru_page(page, src, flags);
 
                 BUG_ON(!PageLRU(page));
 
                 list_del(&page->lru);
-                if (unlikely(get_page_testone(page))) {
+                target = src;
+                if (likely(get_page_unless_zero(page))) {
                         /*
-                         * It is being freed elsewhere
+                         * Be careful not to clear PageLRU until after we're
+                         * sure the page is not being freed elsewhere -- the
+                         * page release code relies on it.
                          */
-                        __put_page(page);
-                        list_add(&page->lru, src);
-                        continue;
-                }
+                        ClearPageLRU(page);
+                        target = dst;
+                        nr_taken++;
+                } /* else it is being freed elsewhere */
 
-                /*
-                 * Be careful not to clear PageLRU until after we're sure
-                 * the page is not being freed elsewhere -- the page release
-                 * code relies on it.
-                 */
-                ClearPageLRU(page);
-                list_add(&page->lru, dst);
-                nr_taken++;
+                list_add(&page->lru, target);
         }
 
         *scanned = scan;