vmscan: do not unconditionally treat zones that fail zone_reclaim() as full

On NUMA machines, the administrator can configure zone_reclaim_mode that
is a more targetted form of direct reclaim.  On machines with large NUMA
distances for example, a zone_reclaim_mode defaults to 1 meaning that
clean unmapped pages will be reclaimed if the zone watermarks are not
being met.  The problem is that zone_reclaim() failing at all means the
zone gets marked full.

This can cause situations where a zone is usable, but is being skipped
because it has been considered full.  Take a situation where a large tmpfs
mount is occuping a large percentage of memory overall.  The pages do not
get cleaned or reclaimed by zone_reclaim(), but the zone gets marked full
and the zonelist cache considers them not worth trying in the future.

This patch makes zone_reclaim() return more fine-grained information about
what occured when zone_reclaim() failued.  The zone only gets marked full
if it really is unreclaimable.  If it's a case that the scan did not occur
or if enough pages were not reclaimed with the limited reclaim_mode, then
the zone is simply skipped.

There is a side-effect to this patch.  Currently, if zone_reclaim()
successfully reclaimed SWAP_CLUSTER_MAX, an allocation attempt would go
ahead.  With this patch applied, zone watermarks are rechecked after
zone_reclaim() does some work.

This bug was introduced by commit 9276b1bc96a132f4068fdee00983c532f43d3a26
("memory page_alloc zonelist caching speedup") way back in 2.6.19 when the
zonelist_cache was introduced.  It was not intended that zone_reclaim()
aggressively consider the zone to be full when it failed as full direct
reclaim can still be an option.  Due to the age of the bug, it should be
considered a -stable candidate.

Signed-off-by: Mel Gorman <mel@csn.ul.ie>
Reviewed-by: Wu Fengguang <fengguang.wu@intel.com>
Reviewed-by: Rik van Riel <riel@redhat.com>
Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Christoph Lameter <cl@linux-foundation.org>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 22 insertions, 4 deletions

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6407cbfccd77..2f457a756d46 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1462,15 +1462,33 @@ zonelist_scan:
                 BUILD_BUG_ON(ALLOC_NO_WATERMARKS < NR_WMARK);
                 if (!(alloc_flags & ALLOC_NO_WATERMARKS)) {
                         unsigned long mark;
+                        int ret;
+
                         mark = zone->watermark[alloc_flags & ALLOC_WMARK_MASK];
-                        if (!zone_watermark_ok(zone, order, mark,
-                                    classzone_idx, alloc_flags)) {
-                                if (!zone_reclaim_mode ||
-                                    !zone_reclaim(zone, gfp_mask, order))
+                        if (zone_watermark_ok(zone, order, mark,
+                                    classzone_idx, alloc_flags))
+                                goto try_this_zone;
+
+                        if (zone_reclaim_mode == 0)
+                                goto this_zone_full;
+
+                        ret = zone_reclaim(zone, gfp_mask, order);
+                        switch (ret) {
+                        case ZONE_RECLAIM_NOSCAN:
+                                /* did not scan */
+                                goto try_next_zone;
+                        case ZONE_RECLAIM_FULL:
+                                /* scanned but unreclaimable */
+                                goto this_zone_full;
+                        default:
+                                /* did we reclaim enough */
+                                if (!zone_watermark_ok(zone, order, mark,
+                                                classzone_idx, alloc_flags))
                                         goto this_zone_full;
                         }
                 }
 
+try_this_zone:
                 page = buffered_rmqueue(preferred_zone, zone, order,
                                                 gfp_mask, migratetype);
                 if (page)