radix-tree: fix RCU bug

Salman Qazi describes the following radix-tree bug:

In the following case, we get can get a deadlock:

0.  The radix tree contains two items, one has the index 0.
1.  The reader (in this case find_get_pages) takes the rcu_read_lock.
2.  The reader acquires slot(s) for item(s) including the index 0 item.
3.  The non-zero index item is deleted, and as a consequence the other item is
    moved to the root of the tree. The place where it used to be is queued for
    deletion after the readers finish.
3b. The zero item is deleted, removing it from the direct slot, it remains in
    the rcu-delayed indirect node.
4.  The reader looks at the index 0 slot, and finds that the page has 0 ref
    count
5.  The reader looks at it again, hoping that the item will either be freed or
    the ref count will increase. This never happens, as the slot it is looking
    at will never be updated. Also, this slot can never be reclaimed because
    the reader is holding rcu_read_lock and is in an infinite loop.

The fix is to re-use the same "indirect" pointer case that requires a slot
lookup retry into a general "retry the lookup" bit.

Signed-off-by: Nick Piggin <npiggin@kernel.dk>
Reported-by: Salman Qazi <sqazi@google.com>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 10 insertions, 16 deletions

diff --git a/mm/filemap.c b/mm/filemap.c
index 4ee2e998e937..ea89840fc65f 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -644,7 +644,9 @@ repeat:
         pagep = radix_tree_lookup_slot(&mapping->page_tree, offset);
         if (pagep) {
                 page = radix_tree_deref_slot(pagep);
-                if (unlikely(!page || page == RADIX_TREE_RETRY))
+                if (unlikely(!page))
+                        goto out;
+                if (radix_tree_deref_retry(page))
                         goto repeat;
 
                 if (!page_cache_get_speculative(page))
@@ -660,6 +662,7 @@ repeat:
                         goto repeat;
                 }
         }
+out:
         rcu_read_unlock();
 
         return page;
@@ -777,12 +780,11 @@ repeat:
                 page = radix_tree_deref_slot((void **)pages[i]);
                 if (unlikely(!page))
                         continue;
-                /*
-                 * this can only trigger if nr_found == 1, making livelock
-                 * a non issue.
-                 */
-                if (unlikely(page == RADIX_TREE_RETRY))
+                if (radix_tree_deref_retry(page)) {
+                        if (ret)
+                                start = pages[ret-1]->index;
                         goto restart;
+                }
 
                 if (!page_cache_get_speculative(page))
                         goto repeat;
@@ -830,11 +832,7 @@ repeat:
                 page = radix_tree_deref_slot((void **)pages[i]);
                 if (unlikely(!page))
                         continue;
-                /*
-                 * this can only trigger if nr_found == 1, making livelock
-                 * a non issue.
-                 */
-                if (unlikely(page == RADIX_TREE_RETRY))
+                if (radix_tree_deref_retry(page))
                         goto restart;
 
                 if (page->mapping == NULL || page->index != index)
@@ -887,11 +885,7 @@ repeat:
                 page = radix_tree_deref_slot((void **)pages[i]);
                 if (unlikely(!page))
                         continue;
-                /*
-                 * this can only trigger if nr_found == 1, making livelock
-                 * a non issue.
-                 */
-                if (unlikely(page == RADIX_TREE_RETRY))
+                if (radix_tree_deref_retry(page))
                         goto restart;
 
                 if (!page_cache_get_speculative(page))