mm: vmscan: do not OOM if aborting reclaim to start compaction

During direct reclaim it is possible that reclaim will be aborted so that
compaction can be attempted to satisfy a high-order allocation.  If this
decision is made before any pages are reclaimed, it is possible that 0 is
returned to the page allocator potentially triggering an OOM.  This has
not been observed but it is a possibility so this patch addresses it.

Signed-off-by: Mel Gorman <mgorman@suse.de>
Reviewed-by: Rik van Riel <riel@redhat.com>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Jan Kara <jack@suse.cz>
Cc: Andy Isaacson <adi@hexapodia.org>
Cc: Nai Xia <nai.xia@gmail.com>
Cc: Johannes Weiner <jweiner@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 7 insertions, 1 deletions

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 866ab27c52a5..cb68c53db4ec 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2263,6 +2263,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
         struct zoneref *z;
         struct zone *zone;
         unsigned long writeback_threshold;
+        bool should_abort_reclaim;
 
         get_mems_allowed();
         delayacct_freepages_start();
@@ -2274,7 +2275,8 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,
                 sc->nr_scanned = 0;
                 if (!priority)
                         disable_swap_token(sc->target_mem_cgroup);
-                if (shrink_zones(priority, zonelist, sc))
+                should_abort_reclaim = shrink_zones(priority, zonelist, sc);
+                if (should_abort_reclaim)
                         break;
 
                 /*
@@ -2342,6 +2344,10 @@ out:
         if (oom_killer_disabled)
                 return 0;
 
+        /* Aborting reclaim to try compaction? don't OOM, then */
+        if (should_abort_reclaim)
+                return 1;
+
         /* top priority shrink_zones still had more to do? don't OOM, then */
         if (global_reclaim(sc) && !all_unreclaimable(zonelist, sc))
                 return 1;