mm, THP, swap: delay splitting THP during swap out

Patch series "THP swap: Delay splitting THP during swapping out", v11.

This patchset is to optimize the performance of Transparent Huge Page
(THP) swap.

Recently, the performance of the storage devices improved so fast that
we cannot saturate the disk bandwidth with single logical CPU when do
page swap out even on a high-end server machine.  Because the
performance of the storage device improved faster than that of single
logical CPU.  And it seems that the trend will not change in the near
future.  On the other hand, the THP becomes more and more popular
because of increased memory size.  So it becomes necessary to optimize
THP swap performance.

The advantages of the THP swap support include:

 - Batch the swap operations for the THP to reduce lock
   acquiring/releasing, including allocating/freeing the swap space,
   adding/deleting to/from the swap cache, and writing/reading the swap
   space, etc. This will help improve the performance of the THP swap.

 - The THP swap space read/write will be 2M sequential IO. It is
   particularly helpful for the swap read, which are usually 4k random
   IO. This will improve the performance of the THP swap too.

 - It will help the memory fragmentation, especially when the THP is
   heavily used by the applications. The 2M continuous pages will be
   free up after THP swapping out.

 - It will improve the THP utilization on the system with the swap
   turned on. Because the speed for khugepaged to collapse the normal
   pages into the THP is quite slow. After the THP is split during the
   swapping out, it will take quite long time for the normal pages to
   collapse back into the THP after being swapped in. The high THP
   utilization helps the efficiency of the page based memory management
   too.

There are some concerns regarding THP swap in, mainly because possible
enlarged read/write IO size (for swap in/out) may put more overhead on
the storage device.  To deal with that, the THP swap in should be turned
on only when necessary.  For example, it can be selected via
"always/never/madvise" logic, to be turned on globally, turned off
globally, or turned on only for VMA with MADV_HUGEPAGE, etc.

This patchset is the first step for the THP swap support.  The plan is
to delay splitting THP step by step, finally avoid splitting THP during
the THP swapping out and swap out/in the THP as a whole.

As the first step, in this patchset, the splitting huge page is delayed
from almost the first step of swapping out to after allocating the swap
space for the THP and adding the THP into the swap cache.  This will
reduce lock acquiring/releasing for the locks used for the swap cache
management.

With the patchset, the swap out throughput improves 15.5% (from about
3.73GB/s to about 4.31GB/s) in the vm-scalability swap-w-seq test case
with 8 processes.  The test is done on a Xeon E5 v3 system.  The swap
device used is a RAM simulated PMEM (persistent memory) device.  To test
the sequential swapping out, the test case creates 8 processes, which
sequentially allocate and write to the anonymous pages until the RAM and
part of the swap device is used up.

This patch (of 5):

In this patch, splitting huge page is delayed from almost the first step
of swapping out to after allocating the swap space for the THP
(Transparent Huge Page) and adding the THP into the swap cache.  This
will batch the corresponding operation, thus improve THP swap out
throughput.

This is the first step for the THP swap optimization.  The plan is to
delay splitting the THP step by step and avoid splitting the THP
finally.

In this patch, one swap cluster is used to hold the contents of each THP
swapped out.  So, the size of the swap cluster is changed to that of the
THP (Transparent Huge Page) on x86_64 architecture (512).  For other
architectures which want such THP swap optimization,
ARCH_USES_THP_SWAP_CLUSTER needs to be selected in the Kconfig file for
the architecture.  In effect, this will enlarge swap cluster size by 2
times on x86_64.  Which may make it harder to find a free cluster when
the swap space becomes fragmented.  So that, this may reduce the
continuous swap space allocation and sequential write in theory.  The
performance test in 0day shows no regressions caused by this.

In the future of THP swap optimization, some information of the swapped
out THP (such as compound map count) will be recorded in the
swap_cluster_info data structure.

The mem cgroup swap accounting functions are enhanced to support charge
or uncharge a swap cluster backing a THP as a whole.

The swap cluster allocate/free functions are added to allocate/free a
swap cluster for a THP.  A fair simple algorithm is used for swap
cluster allocation, that is, only the first swap device in priority list
will be tried to allocate the swap cluster.  The function will fail if
the trying is not successful, and the caller will fallback to allocate a
single swap slot instead.  This works good enough for normal cases.  If
the difference of the number of the free swap clusters among multiple
swap devices is significant, it is possible that some THPs are split
earlier than necessary.  For example, this could be caused by big size
difference among multiple swap devices.

The swap cache functions is enhanced to support add/delete THP to/from
the swap cache as a set of (HPAGE_PMD_NR) sub-pages.  This may be
enhanced in the future with multi-order radix tree.  But because we will
split the THP soon during swapping out, that optimization doesn't make
much sense for this first step.

The THP splitting functions are enhanced to support to split THP in swap
cache during swapping out.  The page lock will be held during allocating
the swap cluster, adding the THP into the swap cache and splitting the
THP.  So in the code path other than swapping out, if the THP need to be
split, the PageSwapCache(THP) will be always false.

The swap cluster is only available for SSD, so the THP swap optimization
in this patchset has no effect for HDD.

[ying.huang@intel.com: fix two issues in THP optimize patch]
  Link: http://lkml.kernel.org/r/87k25ed8zo.fsf@yhuang-dev.intel.com
[hannes@cmpxchg.org: extensive cleanups and simplifications, reduce code size]
Link: http://lkml.kernel.org/r/20170515112522.32457-2-ying.huang@intel.com
Signed-off-by: "Huang, Ying" <ying.huang@intel.com>
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Andrew Morton <akpm@linux-foundation.org> [for config option]
Acked-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> [for changes in huge_memory.c and huge_mm.h]
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Ebru Akagunduz <ebru.akagunduz@gmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Shaohua Li <shli@kernel.org>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Rik van Riel <riel@redhat.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 190 insertions, 69 deletions

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 4f6cba1b6632..984f0dd94948 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -199,7 +199,11 @@ static void discard_swap_cluster(struct swap_info_struct *si,
         }
 }
 
+#ifdef CONFIG_THP_SWAP
+#define SWAPFILE_CLUSTER        HPAGE_PMD_NR
+#else
 #define SWAPFILE_CLUSTER        256
+#endif
 #define LATENCY_LIMIT           256
 
 static inline void cluster_set_flag(struct swap_cluster_info *info,
@@ -374,6 +378,14 @@ static void swap_cluster_schedule_discard(struct swap_info_struct *si,
         schedule_work(&si->discard_work);
 }
 
+static void __free_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+        struct swap_cluster_info *ci = si->cluster_info;
+
+        cluster_set_flag(ci + idx, CLUSTER_FLAG_FREE);
+        cluster_list_add_tail(&si->free_clusters, ci, idx);
+}
+
 /*
  * Doing discard actually. After a cluster discard is finished, the cluster
  * will be added to free cluster list. caller should hold si->lock.
@@ -394,10 +406,7 @@ static void swap_do_scheduled_discard(struct swap_info_struct *si)
 
                 spin_lock(&si->lock);
                 ci = lock_cluster(si, idx * SWAPFILE_CLUSTER);
-                cluster_set_flag(ci, CLUSTER_FLAG_FREE);
-                unlock_cluster(ci);
-                cluster_list_add_tail(&si->free_clusters, info, idx);
-                ci = lock_cluster(si, idx * SWAPFILE_CLUSTER);
+                __free_cluster(si, idx);
                 memset(si->swap_map + idx * SWAPFILE_CLUSTER,
                                 0, SWAPFILE_CLUSTER);
                 unlock_cluster(ci);
@@ -415,6 +424,34 @@ static void swap_discard_work(struct work_struct *work)
         spin_unlock(&si->lock);
 }
 
+static void alloc_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+        struct swap_cluster_info *ci = si->cluster_info;
+
+        VM_BUG_ON(cluster_list_first(&si->free_clusters) != idx);
+        cluster_list_del_first(&si->free_clusters, ci);
+        cluster_set_count_flag(ci + idx, 0, 0);
+}
+
+static void free_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+        struct swap_cluster_info *ci = si->cluster_info + idx;
+
+        VM_BUG_ON(cluster_count(ci) != 0);
+        /*
+         * If the swap is discardable, prepare discard the cluster
+         * instead of free it immediately. The cluster will be freed
+         * after discard.
+         */
+        if ((si->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) ==
+            (SWP_WRITEOK | SWP_PAGE_DISCARD)) {
+                swap_cluster_schedule_discard(si, idx);
+                return;
+        }
+
+        __free_cluster(si, idx);
+}
+
 /*
  * The cluster corresponding to page_nr will be used. The cluster will be
  * removed from free cluster list and its usage counter will be increased.
@@ -426,11 +463,8 @@ static void inc_cluster_info_page(struct swap_info_struct *p,
 
         if (!cluster_info)
                 return;
-        if (cluster_is_free(&cluster_info[idx])) {
-                VM_BUG_ON(cluster_list_first(&p->free_clusters) != idx);
-                cluster_list_del_first(&p->free_clusters, cluster_info);
-                cluster_set_count_flag(&cluster_info[idx], 0, 0);
-        }
+        if (cluster_is_free(&cluster_info[idx]))
+                alloc_cluster(p, idx);
 
         VM_BUG_ON(cluster_count(&cluster_info[idx]) >= SWAPFILE_CLUSTER);
         cluster_set_count(&cluster_info[idx],
@@ -454,21 +488,8 @@ static void dec_cluster_info_page(struct swap_info_struct *p,
         cluster_set_count(&cluster_info[idx],
                 cluster_count(&cluster_info[idx]) - 1);
 
-        if (cluster_count(&cluster_info[idx]) == 0) {
-                /*
-                 * If the swap is discardable, prepare discard the cluster
-                 * instead of free it immediately. The cluster will be freed
-                 * after discard.
-                 */
-                if ((p->flags & (SWP_WRITEOK | SWP_PAGE_DISCARD)) ==
-                                 (SWP_WRITEOK | SWP_PAGE_DISCARD)) {
-                        swap_cluster_schedule_discard(p, idx);
-                        return;
-                }
-
-                cluster_set_flag(&cluster_info[idx], CLUSTER_FLAG_FREE);
-                cluster_list_add_tail(&p->free_clusters, cluster_info, idx);
-        }
+        if (cluster_count(&cluster_info[idx]) == 0)
+                free_cluster(p, idx);
 }
 
 /*
@@ -558,6 +579,60 @@ new_cluster:
         return found_free;
 }
 
+static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset,
+                             unsigned int nr_entries)
+{
+        unsigned int end = offset + nr_entries - 1;
+
+        if (offset == si->lowest_bit)
+                si->lowest_bit += nr_entries;
+        if (end == si->highest_bit)
+                si->highest_bit -= nr_entries;
+        si->inuse_pages += nr_entries;
+        if (si->inuse_pages == si->pages) {
+                si->lowest_bit = si->max;
+                si->highest_bit = 0;
+                spin_lock(&swap_avail_lock);
+                plist_del(&si->avail_list, &swap_avail_head);
+                spin_unlock(&swap_avail_lock);
+        }
+}
+
+static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
+                            unsigned int nr_entries)
+{
+        unsigned long end = offset + nr_entries - 1;
+        void (*swap_slot_free_notify)(struct block_device *, unsigned long);
+
+        if (offset < si->lowest_bit)
+                si->lowest_bit = offset;
+        if (end > si->highest_bit) {
+                bool was_full = !si->highest_bit;
+
+                si->highest_bit = end;
+                if (was_full && (si->flags & SWP_WRITEOK)) {
+                        spin_lock(&swap_avail_lock);
+                        WARN_ON(!plist_node_empty(&si->avail_list));
+                        if (plist_node_empty(&si->avail_list))
+                                plist_add(&si->avail_list, &swap_avail_head);
+                        spin_unlock(&swap_avail_lock);
+                }
+        }
+        atomic_long_add(nr_entries, &nr_swap_pages);
+        si->inuse_pages -= nr_entries;
+        if (si->flags & SWP_BLKDEV)
+                swap_slot_free_notify =
+                        si->bdev->bd_disk->fops->swap_slot_free_notify;
+        else
+                swap_slot_free_notify = NULL;
+        while (offset <= end) {
+                frontswap_invalidate_page(si->type, offset);
+                if (swap_slot_free_notify)
+                        swap_slot_free_notify(si->bdev, offset);
+                offset++;
+        }
+}
+
 static int scan_swap_map_slots(struct swap_info_struct *si,
                                unsigned char usage, int nr,
                                swp_entry_t slots[])
@@ -676,18 +751,7 @@ checks:
         inc_cluster_info_page(si, si->cluster_info, offset);
         unlock_cluster(ci);
 
-        if (offset == si->lowest_bit)
-                si->lowest_bit++;
-        if (offset == si->highest_bit)
-                si->highest_bit--;
-        si->inuse_pages++;
-        if (si->inuse_pages == si->pages) {
-                si->lowest_bit = si->max;
-                si->highest_bit = 0;
-                spin_lock(&swap_avail_lock);
-                plist_del(&si->avail_list, &swap_avail_head);
-                spin_unlock(&swap_avail_lock);
-        }
+        swap_range_alloc(si, offset, 1);
         si->cluster_next = offset + 1;
         slots[n_ret++] = swp_entry(si->type, offset);
 
@@ -766,6 +830,52 @@ no_page:
         return n_ret;
 }
 
+#ifdef CONFIG_THP_SWAP
+static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+{
+        unsigned long idx;
+        struct swap_cluster_info *ci;
+        unsigned long offset, i;
+        unsigned char *map;
+
+        if (cluster_list_empty(&si->free_clusters))
+                return 0;
+
+        idx = cluster_list_first(&si->free_clusters);
+        offset = idx * SWAPFILE_CLUSTER;
+        ci = lock_cluster(si, offset);
+        alloc_cluster(si, idx);
+        cluster_set_count_flag(ci, SWAPFILE_CLUSTER, 0);
+
+        map = si->swap_map + offset;
+        for (i = 0; i < SWAPFILE_CLUSTER; i++)
+                map[i] = SWAP_HAS_CACHE;
+        unlock_cluster(ci);
+        swap_range_alloc(si, offset, SWAPFILE_CLUSTER);
+        *slot = swp_entry(si->type, offset);
+
+        return 1;
+}
+
+static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx)
+{
+        unsigned long offset = idx * SWAPFILE_CLUSTER;
+        struct swap_cluster_info *ci;
+
+        ci = lock_cluster(si, offset);
+        cluster_set_count_flag(ci, 0, 0);
+        free_cluster(si, idx);
+        unlock_cluster(ci);
+        swap_range_free(si, offset, SWAPFILE_CLUSTER);
+}
+#else
+static int swap_alloc_cluster(struct swap_info_struct *si, swp_entry_t *slot)
+{
+        VM_WARN_ON_ONCE(1);
+        return 0;
+}
+#endif /* CONFIG_THP_SWAP */
+
 static unsigned long scan_swap_map(struct swap_info_struct *si,
                                    unsigned char usage)
 {
@@ -781,13 +891,17 @@ static unsigned long scan_swap_map(struct swap_info_struct *si,
 
 }
 
-int get_swap_pages(int n_goal, swp_entry_t swp_entries[])
+int get_swap_pages(int n_goal, bool cluster, swp_entry_t swp_entries[])
 {
+        unsigned long nr_pages = cluster ? SWAPFILE_CLUSTER : 1;
         struct swap_info_struct *si, *next;
         long avail_pgs;
         int n_ret = 0;
 
-        avail_pgs = atomic_long_read(&nr_swap_pages);
+        /* Only single cluster request supported */
+        WARN_ON_ONCE(n_goal > 1 && cluster);
+
+        avail_pgs = atomic_long_read(&nr_swap_pages) / nr_pages;
         if (avail_pgs <= 0)
                 goto noswap;
 
@@ -797,7 +911,7 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[])
         if (n_goal > avail_pgs)
                 n_goal = avail_pgs;
 
-        atomic_long_sub(n_goal, &nr_swap_pages);
+        atomic_long_sub(n_goal * nr_pages, &nr_swap_pages);
 
         spin_lock(&swap_avail_lock);
 
@@ -823,10 +937,13 @@ start_over:
                         spin_unlock(&si->lock);
                         goto nextsi;
                 }
-                n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE,
-                                            n_goal, swp_entries);
+                if (cluster)
+                        n_ret = swap_alloc_cluster(si, swp_entries);
+                else
+                        n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE,
+                                                    n_goal, swp_entries);
                 spin_unlock(&si->lock);
-                if (n_ret)
+                if (n_ret || cluster)
                         goto check_out;
                 pr_debug("scan_swap_map of si %d failed to find offset\n",
                         si->type);
@@ -852,7 +969,8 @@ nextsi:
 
 check_out:
         if (n_ret < n_goal)
-                atomic_long_add((long) (n_goal-n_ret), &nr_swap_pages);
+                atomic_long_add((long)(n_goal - n_ret) * nr_pages,
+                                &nr_swap_pages);
 noswap:
         return n_ret;
 }
@@ -1008,32 +1126,8 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry)
         dec_cluster_info_page(p, p->cluster_info, offset);
         unlock_cluster(ci);
 
-        mem_cgroup_uncharge_swap(entry);
-        if (offset < p->lowest_bit)
-                p->lowest_bit = offset;
-        if (offset > p->highest_bit) {
-                bool was_full = !p->highest_bit;
-
-                p->highest_bit = offset;
-                if (was_full && (p->flags & SWP_WRITEOK)) {
-                        spin_lock(&swap_avail_lock);
-                        WARN_ON(!plist_node_empty(&p->avail_list));
-                        if (plist_node_empty(&p->avail_list))
-                                plist_add(&p->avail_list,
-                                          &swap_avail_head);
-                        spin_unlock(&swap_avail_lock);
-                }
-        }
-        atomic_long_inc(&nr_swap_pages);
-        p->inuse_pages--;
-        frontswap_invalidate_page(p->type, offset);
-        if (p->flags & SWP_BLKDEV) {
-                struct gendisk *disk = p->bdev->bd_disk;
-
-                if (disk->fops->swap_slot_free_notify)
-                        disk->fops->swap_slot_free_notify(p->bdev,
-                                                          offset);
-        }
+        mem_cgroup_uncharge_swap(entry, 1);
+        swap_range_free(p, offset, 1);
 }
 
 /*
@@ -1065,6 +1159,33 @@ void swapcache_free(swp_entry_t entry)
         }
 }
 
+#ifdef CONFIG_THP_SWAP
+void swapcache_free_cluster(swp_entry_t entry)
+{
+        unsigned long offset = swp_offset(entry);
+        unsigned long idx = offset / SWAPFILE_CLUSTER;
+        struct swap_cluster_info *ci;
+        struct swap_info_struct *si;
+        unsigned char *map;
+        unsigned int i;
+
+        si = swap_info_get(entry);
+        if (!si)
+                return;
+
+        ci = lock_cluster(si, offset);
+        map = si->swap_map + offset;
+        for (i = 0; i < SWAPFILE_CLUSTER; i++) {
+                VM_BUG_ON(map[i] != SWAP_HAS_CACHE);
+                map[i] = 0;
+        }
+        unlock_cluster(ci);
+        mem_cgroup_uncharge_swap(entry, SWAPFILE_CLUSTER);
+        swap_free_cluster(si, idx);
+        spin_unlock(&si->lock);
+}
+#endif /* CONFIG_THP_SWAP */
+
 void swapcache_free_entries(swp_entry_t *entries, int n)
 {
         struct swap_info_struct *p, *prev;