radix-tree: fix race in gang lookup

If the indirect_ptr bit is set on a slot, that indicates we need to redo
the lookup.  Introduce a new function radix_tree_iter_retry() which
forces the loop to retry the lookup by setting 'slot' to NULL and
turning the iterator back to point at the problematic entry.

This is a pretty rare problem to hit at the moment; the lookup has to
race with a grow of the radix tree from a height of 0.  The consequences
of hitting this race are that gang lookup could return a pointer to a
radix_tree_node instead of a pointer to whatever the user had inserted
in the tree.

Fixes: cebbd29e1c2f ("radix-tree: rewrite gang lookup using iterator")
Signed-off-by: Matthew Wilcox <willy@linux.intel.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Ohad Ben-Cohen <ohad@wizery.com>
Cc: Konstantin Khlebnikov <khlebnikov@openvz.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 10 insertions, 2 deletions

diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index fcf5d98574ce..6b79e9026e24 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -1019,9 +1019,13 @@ radix_tree_gang_lookup(struct radix_tree_root *root, void **results,
                 return 0;
 
         radix_tree_for_each_slot(slot, root, &iter, first_index) {
-                results[ret] = indirect_to_ptr(rcu_dereference_raw(*slot));
+                results[ret] = rcu_dereference_raw(*slot);
                 if (!results[ret])
                         continue;
+                if (radix_tree_is_indirect_ptr(results[ret])) {
+                        slot = radix_tree_iter_retry(&iter);
+                        continue;
+                }
                 if (++ret == max_items)
                         break;
         }
@@ -1098,9 +1102,13 @@ radix_tree_gang_lookup_tag(struct radix_tree_root *root, void **results,
                 return 0;
 
         radix_tree_for_each_tagged(slot, root, &iter, first_index, tag) {
-                results[ret] = indirect_to_ptr(rcu_dereference_raw(*slot));
+                results[ret] = rcu_dereference_raw(*slot);
                 if (!results[ret])
                         continue;
+                if (radix_tree_is_indirect_ptr(results[ret])) {
+                        slot = radix_tree_iter_retry(&iter);
+                        continue;
+                }
                 if (++ret == max_items)
                         break;
         }