mm: optimize put_mems_allowed() usage

Since put_mems_allowed() is strictly optional, its a seqcount retry, we
don't need to evaluate the function if the allocation was in fact
successful, saving a smp_rmb some loads and comparisons on some relative
fast-paths.

Since the naming, get/put_mems_allowed() does suggest a mandatory
pairing, rename the interface, as suggested by Mel, to resemble the
seqcount interface.

This gives us: read_mems_allowed_begin() and read_mems_allowed_retry(),
where it is important to note that the return value of the latter call
is inverted from its previous incarnation.

Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 7 insertions, 9 deletions

diff --git a/mm/slub.c b/mm/slub.c
index 25f14ad8f817..7611f148ee81 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1684,7 +1684,7 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags,
                 return NULL;
 
         do {
-                cpuset_mems_cookie = get_mems_allowed();
+                cpuset_mems_cookie = read_mems_allowed_begin();
                 zonelist = node_zonelist(slab_node(), flags);
                 for_each_zone_zonelist(zone, z, zonelist, high_zoneidx) {
                         struct kmem_cache_node *n;
@@ -1696,19 +1696,17 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags,
                                 object = get_partial_node(s, n, c, flags);
                                 if (object) {
                                         /*
-                                         * Return the object even if
-                                         * put_mems_allowed indicated that
-                                         * the cpuset mems_allowed was
-                                         * updated in parallel. It's a
-                                         * harmless race between the alloc
-                                         * and the cpuset update.
+                                         * Don't check read_mems_allowed_retry()
+                                         * here - if mems_allowed was updated in
+                                         * parallel, that was a harmless race
+                                         * between allocation and the cpuset
+                                         * update
                                          */
-                                        put_mems_allowed(cpuset_mems_cookie);
                                         return object;
                                 }
                         }
                 }
-        } while (!put_mems_allowed(cpuset_mems_cookie));
+        } while (read_mems_allowed_retry(cpuset_mems_cookie));
 #endif
         return NULL;
 }