lkml.org 
[lkml]   [2019]   [Oct]   [8]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    SubjectRe: [PATCH v2] mm/page_isolation: fix a deadlock with printk()
    From
    Date
    On Tue, 2019-10-08 at 14:56 +0200, Christian Borntraeger wrote:
    > Adding Peter Oberparleiter.
    > Peter, can you have a look?
    >
    > On 08.10.19 10:27, Michal Hocko wrote:
    > > On Tue 08-10-19 09:43:57, Petr Mladek wrote:
    > > > On Mon 2019-10-07 16:49:37, Michal Hocko wrote:
    > > > > [Cc s390 maintainers - the lockdep is http://lkml.kernel.org/r/1570228005-24979-1-git-send-email-cai@lca.pw
    > > > > Petr has explained it is a false positive
    > > > > http://lkml.kernel.org/r/20191007143002.l37bt2lzqtnqjqxu@pathway.suse.cz]
    > > > > On Mon 07-10-19 16:30:02, Petr Mladek wrote:
    > > > > [...]
    > > > > > I believe that it cannot really happen because:
    > > > > >
    > > > > > static int __init
    > > > > > sclp_console_init(void)
    > > > > > {
    > > > > > [...]
    > > > > > rc = sclp_rw_init();
    > > > > > [...]
    > > > > > register_console(&sclp_console);
    > > > > > return 0;
    > > > > > }
    > > > > >
    > > > > > sclp_rw_init() is called before register_console(). And
    > > > > > console_unlock() will never call sclp_console_write() before
    > > > > > the console is registered.
    > > > > >
    > > > > > AFAIK, lockdep only compares existing chain of locks. It does
    > > > > > not know about console registration that would make some
    > > > > > code paths mutually exclusive.
    > > > > >
    > > > > > I believe that it is a false positive. I do not know how to
    > > > > > avoid this lockdep report. I hope that it will disappear
    > > > > > by deferring all printk() calls rather soon.
    > > > >
    > > > > Thanks a lot for looking into this Petr. I have also checked the code
    > > > > and I really fail to see why the allocation has to be done under the
    > > > > lock in the first place. sclp_read_sccb and sclp_init_sccb are global
    > > > > variables but I strongly suspect that they need a synchronization during
    > > > > early init, callbacks are registered only later IIUC:
    > > >
    > > > Good idea. It would work when the init function is called only once.
    > > > But see below.
    > > >
    > > > > diff --git a/drivers/s390/char/sclp.c b/drivers/s390/char/sclp.c
    > > > > index d2ab3f07c008..4b1c033e3255 100644
    > > > > --- a/drivers/s390/char/sclp.c
    > > > > +++ b/drivers/s390/char/sclp.c
    > > > > @@ -1169,13 +1169,13 @@ sclp_init(void)
    > > > > unsigned long flags;
    > > > > int rc = 0;
    > > > >
    > > > > + sclp_read_sccb = (void *) __get_free_page(GFP_ATOMIC | GFP_DMA);
    > > > > + sclp_init_sccb = (void *) __get_free_page(GFP_ATOMIC | GFP_DMA);
    > > > > spin_lock_irqsave(&sclp_lock, flags);
    > > > > /* Check for previous or running initialization */
    > > > > if (sclp_init_state != sclp_init_state_uninitialized)
    > > > > goto fail_unlock;
    > > >
    > > > It seems that sclp_init() could be called several times in parallel.
    > > > I see it called from sclp_register() and sclp_initcall().
    > >
    > > Interesting. Something for s390 people to answer I guess.
    > > Anyway, this should be quite trivial to workaround by a cmpxch or alike.
    > >

    The above fix is simply insufficient,

    00: [    3.654307] WARNING: possible circular locking dependency detected       
    00: [    3.654309] 5.4.0-rc1-next-20191004+ #4 Not tainted                      
    00: [    3.654311] ------------------------------------------------------       
    00: [    3.654313] swapper/0/1 is trying to acquire lock:                       
    00: [    3.654314] 00000000553f3fb8 (sclp_lock){-.-.}, at: sclp_add_request+0x34
    00: /0x308                                                                      
    00: [    3.654320]                                                              
    00: [    3.654322] but task is already holding lock:                            
    00: [    3.654323] 00000000550c9fc0 (console_owner){....}, at: console_unlock+0x
    00: 328/0xa30                                                                   
    00: [    3.654329]                                                              
    00: [    3.654331] which lock already depends on the new lock.                  
    00: [    3.654332]                                                              
    00: [    3.654333]                                                              
    00: [    3.654335] the existing dependency chain (in reverse order) is:         
    00: [    3.654336]                                                              
    00: [    3.654337] -> #3 (console_owner){....}:                                 
    00: [    3.654343]        lock_acquire+0x21a/0x468                              
    00: [    3.654345]        console_unlock+0x3a6/0xa30                            
    00: [    3.654346]        vprintk_emit+0x184/0x3c8                              
    00: [    3.654348]        vprintk_default+0x44/0x50                             
    00: [    3.654349]        printk+0xa8/0xc0                                      
    00: [    3.654351]        get_random_u64+0x40/0x108                             
    00: [    3.654360]        add_to_free_area_random+0x188/0x1c0                   
    00: [    3.654364]        free_one_page+0x72/0x128                              
    00: [    3.654366]        __free_pages_ok+0x51c/0xca0                           
    00: [    3.654368]        memblock_free_all+0x30a/0x3b0                         
    00: [    3.654370]        mem_init+0x84/0x200                                   
    00: [    3.654371]        start_kernel+0x384/0x6a0                              
    00: [    3.654373]        startup_continue+0x70/0xd0                            
    00: [    3.654374]                                                              
    00: [    3.654375] -> #2 (&(&zone->lock)->rlock){....}:                         
    00: [    3.654382]        lock_acquire+0x21a/0x468                              
    00: [    3.654383]        _raw_spin_lock+0x54/0x68                              
    00: [    3.654385]        get_page_from_freelist+0x8b6/0x2d28                   
    00: [    3.654387]        __alloc_pages_nodemask+0x246/0x658                    
    00: [    3.654389]        alloc_slab_page+0x43c/0x858                           
    00: [    3.654390]        allocate_slab+0x90/0x6f0                              
    00: [    3.654392]        new_slab+0x88/0x90                                    
    00: [    3.654393]        ___slab_alloc+0x600/0x988                             
    00: [    3.654395]        __slab_alloc+0x5a/0x90                                
    00: [    3.654396]        kmem_cache_alloc+0x340/0x4c0                          
    00: [    3.654398]        fill_pool+0x27a/0x498                                 
    00: [    3.654400]        __debug_object_init+0xa8/0x858                        
    00: [    3.654401]        debug_object_activate+0x208/0x370                     
    00: [    3.654403]        __call_rcu+0xb6/0x4a8                                 
    00: [    3.654405]        unregister_external_irq+0x13a/0x140                   
    00: [    3.654406]        sclp_init+0x5e0/0x688                                 
    00: [    3.654408]        sclp_register+0x2e/0x248                              
    00: [    3.654409]        sclp_rw_init+0x4a/0x70                                
    00: [    3.654411]        sclp_console_init+0x4a/0x1b8                          
    00: [    3.654413]        console_init+0x2c8/0x410                              
    00: [    3.654414]        start_kernel+0x530/0x6a0                              
    00: [    3.654416]        startup_continue+0x70/0xd0                            
    00: [    3.654417]                                                              
    00: [    3.654418] -> #1 (ext_int_hash_lock){....}:                             
    00: [    3.654424]        lock_acquire+0x21a/0x468                              
    00: [    3.654426]        _raw_spin_lock_irqsave+0xcc/0xe8                      
    00: [    3.654427]        register_external_irq+0xb6/0x138                      
    00: [    3.654429]        sclp_init+0x212/0x688                                 
    00: [    3.654430]        sclp_register+0x2e/0x248                              
    00: [    3.654432]        sclp_rw_init+0x4a/0x70                                
    00: [    3.654434]        sclp_console_init+0x4a/0x1b8                          
    00: [    3.654435]        console_init+0x2c8/0x410                              
    00: [    3.654437]        start_kernel+0x530/0x6a0                              
    00: [    3.654438]        startup_continue+0x70/0xd0                            
    00: [    3.654439]                                                              
    00: [    3.654440] -> #0 (sclp_lock){-.-.}:                                     
    00: [    3.654446]        check_noncircular+0x338/0x3e0                         
    00: [    3.654448]        __lock_acquire+0x1e66/0x2d88                          
    00: [    3.654450]        lock_acquire+0x21a/0x468                              
    00: [    3.654451]        _raw_spin_lock_irqsave+0xcc/0xe8                      
    00: [    3.654453]        sclp_add_request+0x34/0x308                           
    00: [    3.654455]        sclp_conbuf_emit+0x100/0x138                          
    00: [    3.654456]        sclp_console_write+0x96/0x3b8                         
    00: [    3.654458]        console_unlock+0x6dc/0xa30                            
    00: [    3.654460]        vprintk_emit+0x184/0x3c8                              
    00: [    3.654461]        vprintk_default+0x44/0x50                             
    00: [    3.654463]        printk+0xa8/0xc0                                      
    00: [    3.654464]        iommu_debugfs_setup+0xf2/0x108                        
    00: [    3.654466]        iommu_init+0x6c/0x78                                  
    00: [    3.654467]        do_one_initcall+0x162/0x680                           
    00: [    3.654469]        kernel_init_freeable+0x4e8/0x5a8                      
    00: [    3.654471]        kernel_init+0x2a/0x188                                
    00: [    3.654472]        ret_from_fork+0x30/0x34                               
    00: [    3.654474]        kernel_thread_starter+0x0/0xc                         
    00: [    3.654475]                                                              
    00: [    3.654476] other info that might help us debug this:                    
    00: [    3.654477]                                                              
    00: [    3.654479] Chain exists of:                                             
    00: [    3.654480]   sclp_lock --> &(&zone->lock)->rlock --> console_owner      
    00: [    3.654488]                                                              
    00: [    3.654489]  Possible unsafe locking scenario:                           
    00: [    3.654490]                                                              
    00: [    3.654492]        CPU0                    CPU1                          
    00: [    3.654493]        ----                    ----                          
    00: [    3.654495]   lock(console_owner);                                       
    00: [    3.654498]                                lock(&(&zone->lock)->rlock);  
    00: [    3.654503]                                lock(console_owner);          
    00: [    3.654506]   lock(sclp_lock);                                           
    00: [    3.654509]                                                              
    00: [    3.654511]  *** DEADLOCK ***                                            
    00: [    3.654512]                                                              
    00: [    3.654513] 2 locks held by swapper/0/1:                                 
    00: [    3.654514]  #0: 00000000550ca200 (console_lock){+.+.}, at: vprintk_emit+
    00: 0x178/0x3c8                                                                 
    00: [    3.654521]  #1: 00000000550c9fc0 (console_owner){....}, at: console_unlo
    00: ck+0x328/0xa30                                                              
    00: [    3.654529]                                                              
    00: [    3.654530] stack backtrace:                                             
    00: [    3.654532] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-rc1-next-2019
    00: 1004+ #4                                                                    
    00: [    3.654534] Hardware name: IBM 2964 N96 400 (z/VM 6.4.0)                 
    00: [    3.654535] Call Trace:                                                  
    00: [    3.654537] ([<000000005431e218>] show_stack+0x110/0x1b0)                
    00: [    3.654539]  [<0000000054bdd926>] dump_stack+0x126/0x178                 
    00: [    3.654541]  [<0000000054414b08>] check_noncircular+0x338/0x3e0          
    00: [    3.654543]  [<000000005441aaf6>] __lock_acquire+0x1e66/0x2d88           
    00: [    3.654545]  [<0000000054417e12>] lock_acquire+0x21a/0x468               
    00: [    3.654547]  [<0000000054c18cc4>] _raw_spin_lock_irqsave+0xcc/0xe8       
    00: [    3.654549]  [<00000000549f2e14>] sclp_add_request+0x34/0x308            
    00: [    3.654551]  [<00000000549fa7f0>] sclp_conbuf_emit+0x100/0x138           
    00: [    3.654553]  [<00000000549fa90e>] sclp_console_write+0x96/0x3b8          
    00: [    3.654554]  [<000000005442b634>] console_unlock+0x6dc/0xa30             
    00: [    3.654556]  [<000000005442de2c>] vprintk_emit+0x184/0x3c8               
    00: [    3.654558]  [<000000005442e0b4>] vprintk_default+0x44/0x50              
    00: [    3.654560]  [<000000005442eb60>] printk+0xa8/0xc0                       
    00: [    3.654562]  [<000000005494ca3a>] iommu_debugfs_setup+0xf2/0x108         
    00: [    3.654564]  [<000000005557c0ec>] iommu_init+0x6c/0x78                   
    00: [    3.654566]  [<0000000054300fda>] do_one_initcall+0x162/0x680            
    00: [    3.654568]  [<000000005553b9f0>] kernel_init_freeable+0x4e8/0x5a8       
    00: [    3.654569]  [<0000000054c04172>] kernel_init+0x2a/0x188                 
    00: [    3.654571]  [<0000000054c19fbc>] ret_from_fork+0x30/0x34                
    00: [    3.654573]  [<0000000054c19fc0>] kernel_thread_starter+0x0/0xc 

    \
     
     \ /
      Last update: 2019-10-08 18:09    [W:4.113 / U:0.332 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site