slub: move synchronize_sched out of slab_mutex on shrink

synchronize_sched() is a heavy operation and calling it per each cache
owned by a memory cgroup being destroyed may take quite some time.  What
is worse, it's currently called under the slab_mutex, stalling all works
doing cache creation/destruction.

Actually, there isn't much point in calling synchronize_sched() for each
cache - it's enough to call it just once - after setting cpu_partial for
all caches and before shrinking them.  This way, we can also move it out
of the slab_mutex, which we have to hold for iterating over the slab
cache list.

Link: https://bugzilla.kernel.org/show_bug.cgi?id=172991
Link: http://lkml.kernel.org/r/0a10d71ecae3db00fb4421bcd3f82bcc911f4be4.1475329751.git.vdavydov.dev@gmail.com
Signed-off-by: Vladimir Davydov <vdavydov.dev@gmail.com>
Reported-by: Doug Smythies <dsmythies@telus.net>
Acked-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: Christoph Lameter <cl@linux.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Pekka Enberg <penberg@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 25 insertions, 2 deletions

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 329b03843863..5d2f24fbafc5 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -573,6 +573,29 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
         get_online_cpus();
         get_online_mems();
 
+#ifdef CONFIG_SLUB
+        /*
+         * In case of SLUB, we need to disable empty slab caching to
+         * avoid pinning the offline memory cgroup by freeable kmem
+         * pages charged to it. SLAB doesn't need this, as it
+         * periodically purges unused slabs.
+         */
+        mutex_lock(&slab_mutex);
+        list_for_each_entry(s, &slab_caches, list) {
+                c = is_root_cache(s) ? cache_from_memcg_idx(s, idx) : NULL;
+                if (c) {
+                        c->cpu_partial = 0;
+                        c->min_partial = 0;
+                }
+        }
+        mutex_unlock(&slab_mutex);
+        /*
+         * kmem_cache->cpu_partial is checked locklessly (see
+         * put_cpu_partial()). Make sure the change is visible.
+         */
+        synchronize_sched();
+#endif
+
         mutex_lock(&slab_mutex);
         list_for_each_entry(s, &slab_caches, list) {
                 if (!is_root_cache(s))
@@ -584,7 +607,7 @@ void memcg_deactivate_kmem_caches(struct mem_cgroup *memcg)
                 if (!c)
                         continue;
 
-                __kmem_cache_shrink(c, true);
+                __kmem_cache_shrink(c);
                 arr->entries[idx] = NULL;
         }
         mutex_unlock(&slab_mutex);
@@ -755,7 +778,7 @@ int kmem_cache_shrink(struct kmem_cache *cachep)
         get_online_cpus();
         get_online_mems();
         kasan_cache_shrink(cachep);
-        ret = __kmem_cache_shrink(cachep, false);
+        ret = __kmem_cache_shrink(cachep);
         put_online_mems();
         put_online_cpus();
         return ret;