Messages in this thread | | | Date | Thu, 10 Oct 2013 08:52:33 +0200 | Subject | Re: [uml-devel] BUG: soft lockup for a user mode linux image | From | Geert Uytterhoeven <> |
| |
On Thu, Oct 10, 2013 at 4:46 AM, Fengguang Wu <fengguang.wu@intel.com> wrote: > On Wed, Oct 09, 2013 at 11:47:33PM +0200, Jan Kara wrote: >> On Wed 09-10-13 20:43:50, Richard Weinberger wrote: >> > Am 09.10.2013 19:26, schrieb Toralf Förster: >> > > On 10/08/2013 10:07 PM, Geert Uytterhoeven wrote: >> > >> On Sun, Oct 6, 2013 at 11:01 PM, Toralf Förster <toralf.foerster@gmx.de> wrote: >> > >>>> Hmm, now pages_dirtied is zero, according to the backtrace, but the BUG_ON() >> > >>>> asserts its strict positive?!? >> > >>>> >> > >>>> Can you please try the following instead of the BUG_ON(): >> > >>>> >> > >>>> if (pause < 0) { >> > >>>> printk("pages_dirtied = %lu\n", pages_dirtied); >> > >>>> printk("task_ratelimit = %lu\n", task_ratelimit); >> > >>>> printk("pause = %ld\n", pause);
>> > >>> I tried it in different ways already - I'm completely unsuccessful in getting any printk output. >> > >>> As soon as the issue happens I do have a >> > >>> >> > >>> BUG: soft lockup - CPU#0 stuck for 22s! [trinity-child0:1521] >> > >>> >> > >>> at stderr of the UML and then no further input is accepted. With uml_mconsole I'm however able >> > >>> to run very basic commands like a crash dump, sysrq ond so on. >> > >> >> > >> You may get an idea of the magnitude of pages_dirtied by using a chain of >> > >> BUG_ON()s, like: >> > >> >> > >> BUG_ON(pages_dirtied > 2000000000); >> > >> BUG_ON(pages_dirtied > 1000000000); >> > >> BUG_ON(pages_dirtied > 100000000); >> > >> BUG_ON(pages_dirtied > 10000000); >> > >> BUG_ON(pages_dirtied > 1000000); >> > >> >> > >> Probably 1 million is already too much for normal operation? >> > >> >> > > period = HZ * pages_dirtied / task_ratelimit; >> > > BUG_ON(pages_dirtied > 2000000000); >> > > BUG_ON(pages_dirtied > 1000000000); <-------------- this is line 1467 >> > >> > Summary for mm people: >> > >> > Toralf runs trinty on UML/i386. >> > After some time pages_dirtied becomes very large. >> > More than 1000000000 pages in this case. >> Huh, this is really strange. pages_dirtied is passed into >> balance_dirty_pages() from current->nr_dirtied. So I wonder how a value >> over 10^9 can get there. > > I noticed aio_setup_ring() in the call trace and find it recently > added a SetPageDirty() call in a loop by commit 36bc08cc01 ("fs/aio: > Add support to aio ring pages migration"). So added CC to its authors. > >> After all that is over 4TB so I somewhat doubt the >> task was ever able to dirty that much during its lifetime (but correct me >> if I'm wrong here, with UML and memory backed disks it is not totally >> impossible)... I went through the logic of handling ->nr_dirtied but >> I didn't find any obvious problem there. Hum, maybe one thing - what >> 'task_ratelimit' values do you see in balance_dirty_pages? If that one was >> huge, we could possibly accumulate huge current->nr_dirtied. >> >> > Thus, period = HZ * pages_dirtied / task_ratelimit overflows >> > and period/pause becomes extremely large.
period/pause are signed long, so they become negative instead of extremely large when overflowing.
>> > It looks like io_schedule_timeout() get's called with a very large timeout. >> > I don't know why "if (unlikely(pause > max_pause)) {" does not help.
Because pause is now negative.
Gr{oetje,eeting}s,
Geert
-- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |