Messages in this thread | | | From | Steven Whitehouse <> | Subject | Re: Kernel deadlock using nbd over acenic driver | Date | Wed, 15 May 2002 20:43:49 +0100 (BST) |
| |
Hi,
thanks for the info. I'm starting to form some ideas of what the problem with nbd might be. Here is my initial idea of what might be going on:
1. Something happens which starts lots of I/O (e.g. the ext3/xfs journal flush that Xiangping says usually triggers the problem) 2. One of the threads doing the writes blocks running the device I/O queue and causing nbd_send_req(), nbd_xmit() to block in the sendmsg() call (trying to allocate memory GFP_NOIO). I think we only need to have each memory zones free pages just below pages_min at the right time to trigger this. 3. Since bdflush will most likely be running it waits for the dirty blocks its submitted to finish being written back to the nbd device to finish.
So something to try is this, in nbd_send_req() add the lines:
if (current->flags & PF_MEMALLOC == 0) { current->flags |= PF_MEMALLOC; we_set_memalloc = 1; }
before the first nbd_xmit() call and
if (we_set_memalloc) current->flags &= ~PF_MEMALLOC;
at the end just before the return; rememebring to declare the variable:
int we_set_memalloc = 0;
at the top of the function. We know that since the box stays responsive to pings that there must be some free memory, so I suspect some kind of "priority inversion" is at work here.
Another interesting idea... if we changed the icmp receive function so that it leaked all echo request packets rather than recycling them we could measure the free memory after the box has hung by seeing how many times we can ping it before it stops replying. An interesting way to measure free memory, but probably quite effective :-)
It looks like adding the line:
if (icmph->type == ICMP_ECHO) return;
just after (icmp_pointers[icmph->type].handler)(skb); in icmp.c:icmp_rcv() should do the trick in this case.
Those are my current thoughts. Let me know if you are sure that some of what I've said is right/wrong, otherwise I'll have a go tomorrow at trying to prove some of it on my test system here,
Steve.
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |