mm: move zone->pages_scanned into a vmstat counter

zone->pages_scanned is a write-intensive cache line during page reclaim
and it's also updated during page free.  Move the counter into vmstat to
take advantage of the per-cpu updates and do not update it in the free
paths unless necessary.

On a small UMA machine running tiobench the difference is marginal.  On
a 4-node machine the overhead is more noticable.  Note that automatic
NUMA balancing was disabled for this test as otherwise the system CPU
overhead is unpredictable.

          3.16.0-rc3  3.16.0-rc3  3.16.0-rc3
             vanillarearrange-v5   vmstat-v5
User          746.94      759.78      774.56
System      65336.22    58350.98    32847.27
Elapsed     27553.52    27282.02    27415.04

Note that the overhead reduction will vary depending on where exactly
pages are allocated and freed.

Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 4 insertions, 3 deletions

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 5fec1ba9951f..9c8222b499b4 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -174,7 +174,8 @@ static unsigned long zone_reclaimable_pages(struct zone *zone)
 
 bool zone_reclaimable(struct zone *zone)
 {
-        return zone->pages_scanned < zone_reclaimable_pages(zone) * 6;
+        return zone_page_state(zone, NR_PAGES_SCANNED) <
+                zone_reclaimable_pages(zone) * 6;
 }
 
 static unsigned long get_lru_size(struct lruvec *lruvec, enum lru_list lru)
@@ -1508,7 +1509,7 @@ shrink_inactive_list(unsigned long nr_to_scan, struct lruvec *lruvec,
         __mod_zone_page_state(zone, NR_ISOLATED_ANON + file, nr_taken);
 
         if (global_reclaim(sc)) {
-                zone->pages_scanned += nr_scanned;
+                __mod_zone_page_state(zone, NR_PAGES_SCANNED, nr_scanned);
                 if (current_is_kswapd())
                         __count_zone_vm_events(PGSCAN_KSWAPD, zone, nr_scanned);
                 else
@@ -1698,7 +1699,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
         nr_taken = isolate_lru_pages(nr_to_scan, lruvec, &l_hold,
                                      &nr_scanned, sc, isolate_mode, lru);
         if (global_reclaim(sc))
-                zone->pages_scanned += nr_scanned;
+                __mod_zone_page_state(zone, NR_PAGES_SCANNED, nr_scanned);
 
         reclaim_stat->recent_scanned[file] += nr_taken;