SHM_UNLOCK: fix long unpreemptible section

scan_mapping_unevictable_pages() is used to make SysV SHM_LOCKed pages
evictable again once the shared memory is unlocked.  It does this with
pagevec_lookup()s across the whole object (which might occupy most of
memory), and takes 300ms to unlock 7GB here.  A cond_resched() every
PAGEVEC_SIZE pages would be good.

However, KOSAKI-san points out that this is called under shmem.c's
info->lock, and it's also under shm.c's shm_lock(), both spinlocks.
There is no strong reason for that: we need to take these pages off the
unevictable list soonish, but those locks are not required for it.

So move the call to scan_mapping_unevictable_pages() from shmem.c's
unlock handling up to shm.c's unlock handling.  Remove the recently
added barrier, not needed now we have spin_unlock() before the scan.

Use get_file(), with subsequent fput(), to make sure we have a reference
to mapping throughout scan_mapping_unevictable_pages(): that's something
that was previously guaranteed by the shm_lock().

Remove shmctl's lru_add_drain_all(): we don't fault in pages at SHM_LOCK
time, and we lazily discover them to be Unevictable later, so it serves
no purpose for SHM_LOCK; and serves no purpose for SHM_UNLOCK, since
pages still on pagevec are not marked Unevictable.

The original code avoided redundant rescans by checking VM_LOCKED flag
at its level: now avoid them by checking shp's SHM_LOCKED.

The original code called scan_mapping_unevictable_pages() on a locked
area at shm_destroy() time: perhaps we once had accounting cross-checks
which required that, but not now, so skip the overhead and just let
inode eviction deal with them.

Put check_move_unevictable_page() and scan_mapping_unevictable_pages()
under CONFIG_SHMEM (with stub for the TINY case when ramfs is used),
more as comment than to save space; comment them used for SHM_UNLOCK.

Signed-off-by: Hugh Dickins <hughd@google.com>
Reviewed-by: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Minchan Kim <minchan.kim@gmail.com>
Cc: Rik van Riel <riel@redhat.com>
Cc: Shaohua Li <shaohua.li@intel.com>
Cc: Eric Dumazet <eric.dumazet@gmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michel Lespinasse <walken@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 22 insertions, 15 deletions

diff --git a/ipc/shm.c b/ipc/shm.c
index 02ecf2c078fc..854ab58e5f6e 100644
--- a/ipc/shm.c
+++ b/ipc/shm.c
@@ -870,9 +870,7 @@ SYSCALL_DEFINE3(shmctl, int, shmid, int, cmd, struct shmid_ds __user *, buf)
         case SHM_LOCK:
         case SHM_UNLOCK:
         {
-                struct file *uninitialized_var(shm_file);
-
-                lru_add_drain_all();  /* drain pagevecs to lru lists */
+                struct file *shm_file;
 
                 shp = shm_lock_check(ns, shmid);
                 if (IS_ERR(shp)) {
@@ -895,22 +893,31 @@ SYSCALL_DEFINE3(shmctl, int, shmid, int, cmd, struct shmid_ds __user *, buf)
                 err = security_shm_shmctl(shp, cmd);
                 if (err)
                         goto out_unlock;
-                
-                if(cmd==SHM_LOCK) {
+
+                shm_file = shp->shm_file;
+                if (is_file_hugepages(shm_file))
+                        goto out_unlock;
+
+                if (cmd == SHM_LOCK) {
                         struct user_struct *user = current_user();
-                        if (!is_file_hugepages(shp->shm_file)) {
-                                err = shmem_lock(shp->shm_file, 1, user);
-                                if (!err && !(shp->shm_perm.mode & SHM_LOCKED)){
-                                        shp->shm_perm.mode |= SHM_LOCKED;
-                                        shp->mlock_user = user;
-                                }
+                        err = shmem_lock(shm_file, 1, user);
+                        if (!err && !(shp->shm_perm.mode & SHM_LOCKED)) {
+                                shp->shm_perm.mode |= SHM_LOCKED;
+                                shp->mlock_user = user;
                         }
-                } else if (!is_file_hugepages(shp->shm_file)) {
-                        shmem_lock(shp->shm_file, 0, shp->mlock_user);
-                        shp->shm_perm.mode &= ~SHM_LOCKED;
-                        shp->mlock_user = NULL;
+                        goto out_unlock;
                 }
+
+                /* SHM_UNLOCK */
+                if (!(shp->shm_perm.mode & SHM_LOCKED))
+                        goto out_unlock;
+                shmem_lock(shm_file, 0, shp->mlock_user);
+                shp->shm_perm.mode &= ~SHM_LOCKED;
+                shp->mlock_user = NULL;
+                get_file(shm_file);
                 shm_unlock(shp);
+                scan_mapping_unevictable_pages(shm_file->f_mapping);
+                fput(shm_file);
                 goto out;
         }
         case IPC_RMID: