lkml.org 
[lkml]   [2008]   [Dec]   [10]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC][RFT] memcg fix cgroup_mutex deadlock when cpusetreclaims memory
    * KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> [2008-12-10 20:32:03]:

    > Balbir Singh said:
    > > * KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> [2008-12-10
    > > 17:49:06]:
    > >
    > >> On Wed, 10 Dec 2008 10:49:47 +0530
    > >> Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
    > >>
    > >> > Hi,
    > >> >
    > >> > Here is a proposed fix for the memory controller cgroup_mutex deadlock
    > >> > reported. It is lightly tested and reviewed. I need help with review
    > >> > and test. Is the reported deadlock reproducible after this patch? A
    > >> > careful review of the cpuset impact will also be highly appreciated.
    > >> >
    > >> > From: Balbir Singh <balbir@linux.vnet.ibm.com>
    > >> >
    > >> > cpuset_migrate_mm() holds cgroup_mutex throughout the duration of
    > >> > do_migrate_pages(). The issue with that is that
    > >> >
    > >> > 1. It can lead to deadlock with memcg, as do_migrate_pages()
    > >> > enters reclaim
    > >> > 2. It can lead to long latencies, preventing users from creating/
    > >> > destroying other cgroups anywhere else
    > >> >
    > >> > The patch holds callback_mutex through the duration of
    > >> cpuset_migrate_mm() and
    > >> > gives up cgroup_mutex while doing so.
    > >> >
    > >> > Signed-off-by: Balbir Singh <balbir@linux.vnet.ibm.com>
    > >> > ---
    > >> >
    > >> > include/linux/cpuset.h | 13 ++++++++++++-
    > >> > kernel/cpuset.c | 23 ++++++++++++-----------
    > >> > 2 files changed, 24 insertions(+), 12 deletions(-)
    > >> >
    > >> > diff -puN kernel/cgroup.c~cpuset-remove-cgroup-mutex-from-update-path
    > >> kernel/cgroup.c
    > >> > diff -puN kernel/cpuset.c~cpuset-remove-cgroup-mutex-from-update-path
    > >> kernel/cpuset.c
    > >> > --- a/kernel/cpuset.c~cpuset-remove-cgroup-mutex-from-update-path
    > >> > +++ a/kernel/cpuset.c
    > >> > @@ -369,7 +369,7 @@ static void guarantee_online_mems(const
    > >> > * task has been modifying its cpuset.
    > >> > */
    > >> >
    > >> > -void cpuset_update_task_memory_state(void)
    > >> > +void __cpuset_update_task_memory_state(bool held)
    > >> > {
    > >> > int my_cpusets_mem_gen;
    > >> > struct task_struct *tsk = current;
    > >> > @@ -380,7 +380,8 @@ void cpuset_update_task_memory_state(voi
    > >> > rcu_read_unlock();
    > >> >
    > >> > if (my_cpusets_mem_gen != tsk->cpuset_mems_generation) {
    > >> > - mutex_lock(&callback_mutex);
    > >> > + if (!held)
    > >> > + mutex_lock(&callback_mutex);
    > >> > task_lock(tsk);
    > >> > cs = task_cs(tsk); /* Maybe changed when task not locked */
    > >> > guarantee_online_mems(cs, &tsk->mems_allowed);
    > >> > @@ -394,7 +395,8 @@ void cpuset_update_task_memory_state(voi
    > >> > else
    > >> > tsk->flags &= ~PF_SPREAD_SLAB;
    > >> > task_unlock(tsk);
    > >> > - mutex_unlock(&callback_mutex);
    > >> > + if (!held)
    > >> > + mutex_unlock(&callback_mutex);
    > >> > mpol_rebind_task(tsk, &tsk->mems_allowed);
    > >> > }
    > >> > }
    > >> > @@ -949,13 +951,15 @@ static int update_cpumask(struct cpuset
    > >> > * so that the migration code can allocate pages on these nodes.
    > >> > *
    > >> > * Call holding cgroup_mutex, so current's cpuset won't change
    > >> > - * during this call, as manage_mutex holds off any cpuset_attach()
    > >> > + * during this call, as callback_mutex holds off any
    > >> cpuset_attach()
    > >> > * calls. Therefore we don't need to take task_lock around the
    > >> > * call to guarantee_online_mems(), as we know no one is changing
    > >> > * our task's cpuset.
    > >> > *
    > >> > * Hold callback_mutex around the two modifications of our tasks
    > >> > - * mems_allowed to synchronize with cpuset_mems_allowed().
    > >> > + * mems_allowed to synchronize with cpuset_mems_allowed(). Give
    > >> > + * up cgroup_mutex to avoid deadlocking with other subsystems
    > >> > + * as we enter reclaim from do_migrate_pages().
    > >> > *
    > >> > * While the mm_struct we are migrating is typically from some
    > >> > * other task, the task_struct mems_allowed that we are hacking
    > >> > @@ -976,17 +980,14 @@ static void cpuset_migrate_mm(struct mm_
    > >> > {
    > >> > struct task_struct *tsk = current;
    > >> >
    > >> > - cpuset_update_task_memory_state();
    > >> > -
    > >> > + cgroup_unlock();
    > >> > mutex_lock(&callback_mutex);
    > >> > + cpuset_update_task_memory_state_locked();
    > >> > tsk->mems_allowed = *to;
    > >> > - mutex_unlock(&callback_mutex);
    > >> > -
    > >> > do_migrate_pages(mm, from, to, MPOL_MF_MOVE_ALL);
    > >> > -
    > >> > - mutex_lock(&callback_mutex);
    > >> > guarantee_online_mems(task_cs(tsk),&tsk->mems_allowed);
    > >> > mutex_unlock(&callback_mutex);
    > >> > + cgroup_lock();
    > >> > }
    > >> >
    > >>
    > >> Hmm...can't this happen ?
    > >>
    > >> Assume there is a task X and cgroup Z1 and Z2. Z1 and Z2 doesn't need to
    > >> be in
    > >> the same hierarchy.
    > >> ==
    > >> CPU A attach task X to cgroup Z1
    > >> cgroup_lock()
    > >> for_each_subsys_state()
    > >
    > > You mean for_each_subsys() right?
    > >
    > >> => attach(X,Z)
    > >> => migrate_mm()
    > >> => cgroup_unlock()
    > >> migration
    > >>
    > >> CPU B attach task X to cgroup Z2 at the same time
    > >> cgroup_lock()
    > >> replace css_set.
    > >> ==
    > >>
    > >> Works on CPU B can't break for_each_subsys_state() in CPU A ?
    > >>
    > >
    > > for_each_subsys is hierarchy aware, so if we try to add the same task
    > > to different hierachies, it should not be a problem right?
    > >
    > Ah, maybe. But what happens when Z1 and Z2 is the same hierarchy ?
    > Are there some locks ?
    >

    If they are in the same hierarchy, tsk->cgroups and tsk->cg_list is
    updated atomically and for_each_subsys should not be affected.

    Needs more thought and coffee though


    --
    Balbir


    \
     
     \ /
      Last update: 2008-12-10 14:27    [W:0.050 / U:0.020 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site