mm: correct page->pfmemalloc to fix deactivate_slab regression

Commit cfd19c5a9ecf ("mm: only set page->pfmemalloc when
ALLOC_NO_WATERMARKS was used") tried to narrow down page->pfmemalloc
setting, but it missed some places the pfmemalloc should be set.

So, in __slab_alloc, the unalignment pfmemalloc and ALLOC_NO_WATERMARKS
cause incorrect deactivate_slab() on our core2 server:

    64.73%           fio  [kernel.kallsyms]     [k] _raw_spin_lock
                     |
                     --- _raw_spin_lock
                        |
                        |---0.34%-- deactivate_slab
                        |          __slab_alloc
                        |          kmem_cache_alloc
                        |          |

That causes our fio sync write performance to have a 40% regression.

Move the checking in get_page_from_freelist() which resolves this issue.

Signed-off-by: Alex Shi <alex.shi@intel.com>
Acked-by: Mel Gorman <mgorman@suse.de>
Cc: David Miller <davem@davemloft.net
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Tested-by: Eric Dumazet <eric.dumazet@gmail.com>
Tested-by: Sage Weil <sage@inktank.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 11 insertions, 10 deletions

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 009ac285fea7..07f19248acb5 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1928,6 +1928,17 @@ this_zone_full:
                 zlc_active = 0;
                 goto zonelist_scan;
         }
+
+        if (page)
+                /*
+                 * page->pfmemalloc is set when ALLOC_NO_WATERMARKS was
+                 * necessary to allocate the page. The expectation is
+                 * that the caller is taking steps that will free more
+                 * memory. The caller should avoid the page being used
+                 * for !PFMEMALLOC purposes.
+                 */
+                page->pfmemalloc = !!(alloc_flags & ALLOC_NO_WATERMARKS);
+
         return page;
 }
 
@@ -2389,14 +2400,6 @@ rebalance:
                                 zonelist, high_zoneidx, nodemask,
                                 preferred_zone, migratetype);
                 if (page) {
-                        /*
-                         * page->pfmemalloc is set when ALLOC_NO_WATERMARKS was
-                         * necessary to allocate the page. The expectation is
-                         * that the caller is taking steps that will free more
-                         * memory. The caller should avoid the page being used
-                         * for !PFMEMALLOC purposes.
-                         */
-                        page->pfmemalloc = true;
                         goto got_pg;
                 }
         }
@@ -2569,8 +2572,6 @@ retry_cpuset:
                 page = __alloc_pages_slowpath(gfp_mask, order,
                                 zonelist, high_zoneidx, nodemask,
                                 preferred_zone, migratetype);
-        else
-                page->pfmemalloc = false;
 
         trace_mm_page_alloc(page, order, gfp_mask, migratetype);