memcg: add lock to synchronize page accounting and migration

Introduce a new bit spin lock, PCG_MOVE_LOCK, to synchronize the page
accounting and migration code.  This reworks the locking scheme of
_update_stat() and _move_account() by adding new lock bit PCG_MOVE_LOCK,
which is always taken under IRQ disable.

1. If pages are being migrated from a memcg, then updates to that
   memcg page statistics are protected by grabbing PCG_MOVE_LOCK using
   move_lock_page_cgroup().  In an upcoming commit, memcg dirty page
   accounting will be updating memcg page accounting (specifically: num
   writeback pages) from IRQ context (softirq).  Avoid a deadlocking
   nested spin lock attempt by disabling irq on the local processor when
   grabbing the PCG_MOVE_LOCK.

2. lock for update_page_stat is used only for avoiding race with
   move_account().  So, IRQ awareness of lock_page_cgroup() itself is not
   a problem.  The problem is between mem_cgroup_update_page_stat() and
   mem_cgroup_move_account_page().

Trade-off:
  * Changing lock_page_cgroup() to always disable IRQ (or
    local_bh) has some impacts on performance and I think
    it's bad to disable IRQ when it's not necessary.
  * adding a new lock makes move_account() slower.  Score is
    here.

Performance Impact: moving a 8G anon process.

Before:
	real    0m0.792s
	user    0m0.000s
	sys     0m0.780s

After:
	real    0m0.854s
	user    0m0.000s
	sys     0m0.842s

This score is bad but planned patches for optimization can reduce
this impact.

Signed-off-by: KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>
Signed-off-by: Greg Thelen <gthelen@google.com>
Reviewed-by: Minchan Kim <minchan.kim@gmail.com>
Acked-by: Daisuke Nishimura <nishimura@mxp.nes.nec.co.jp>
Cc: Andrea Righi <arighi@develer.com>
Cc: Balbir Singh <balbir@linux.vnet.ibm.com>
Cc: Wu Fengguang <fengguang.wu@intel.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

1 files changed, 7 insertions, 2 deletions

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 3d8a0c79dece..d888956a2cfc 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1606,6 +1606,7 @@ void mem_cgroup_update_page_stat(struct page *page,
         struct mem_cgroup *mem;
         struct page_cgroup *pc = lookup_page_cgroup(page);
         bool need_unlock = false;
+        unsigned long uninitialized_var(flags);
 
         if (unlikely(!pc))
                 return;
@@ -1617,7 +1618,7 @@ void mem_cgroup_update_page_stat(struct page *page,
         /* pc->mem_cgroup is unstable ? */
         if (unlikely(mem_cgroup_stealed(mem))) {
                 /* take a lock against to access pc->mem_cgroup */
-                lock_page_cgroup(pc);
+                move_lock_page_cgroup(pc, &flags);
                 need_unlock = true;
                 mem = pc->mem_cgroup;
                 if (!mem || !PageCgroupUsed(pc))
@@ -1640,7 +1641,7 @@ void mem_cgroup_update_page_stat(struct page *page,
 
 out:
         if (unlikely(need_unlock))
-                unlock_page_cgroup(pc);
+                move_unlock_page_cgroup(pc, &flags);
         rcu_read_unlock();
         return;
 }
@@ -2211,9 +2212,13 @@ static int mem_cgroup_move_account(struct page_cgroup *pc,
                 struct mem_cgroup *from, struct mem_cgroup *to, bool uncharge)
 {
         int ret = -EINVAL;
+        unsigned long flags;
+
         lock_page_cgroup(pc);
         if (PageCgroupUsed(pc) && pc->mem_cgroup == from) {
+                move_lock_page_cgroup(pc, &flags);
                 __mem_cgroup_move_account(pc, from, to, uncharge);
+                move_unlock_page_cgroup(pc, &flags);
                 ret = 0;
         }
         unlock_page_cgroup(pc);