Messages in this thread | | | Date | Mon, 24 Aug 2020 11:48:41 -0400 | From | Brian Foster <> | Subject | Re: [PATCH] iomap: Fix the write_count in iomap_add_to_ioend(). |
| |
On Mon, Aug 24, 2020 at 04:04:17PM +0100, Christoph Hellwig wrote: > On Mon, Aug 24, 2020 at 10:28:23AM -0400, Brian Foster wrote: > > Do I understand the current code (__bio_try_merge_page() -> > > page_is_mergeable()) correctly in that we're checking for physical page > > contiguity and not necessarily requiring a new bio_vec per physical > > page? > > > Yes. >
Ok. I also realize now that this occurs on a kernel without commit 07173c3ec276 ("block: enable multipage bvecs"). That is probably a contributing factor, but it's not clear to me whether it's feasible to backport whatever supporting infrastructure is required for that mechanism to work (I suspect not).
> > With regard to Dave's earlier point around seeing excessively sized bio > > chains.. If I set up a large memory box with high dirty mem ratios and > > do contiguous buffered overwrites over a 32GB range followed by fsync, I > > can see upwards of 1GB per bio and thus chains on the order of 32+ bios > > for the entire write. If I play games with how the buffered overwrite is > > submitted (i.e., in reverse) however, then I can occasionally reproduce > > a ~32GB chain of ~32k bios, which I think is what leads to problems in > > I/O completion on some systems. Granted, I don't reproduce soft lockup > > issues on my system with that behavior, so perhaps there's more to that > > particular issue. > > > > Regardless, it seems reasonable to me to at least have a conservative > > limit on the length of an ioend bio chain. Would anybody object to > > iomap_ioend growing a chain counter and perhaps forcing into a new ioend > > if we chain something like more than 1k bios at once? > > So what exactly is the problem of processing a long chain in the > workqueue vs multiple small chains? Maybe we need a cond_resched() > here and there, but I don't see how we'd substantially change behavior. >
The immediate problem is a watchdog lockup detection in bio completion:
NMI watchdog: Watchdog detected hard LOCKUP on cpu 25
This effectively lands at the following segment of iomap_finish_ioend():
... /* walk each page on bio, ending page IO on them */ bio_for_each_segment_all(bv, bio, iter_all) iomap_finish_page_writeback(inode, bv->bv_page, error);
I suppose we could add a cond_resched(), but is that safe directly inside of a ->bi_end_io() handler? Another option could be to dump large chains into the completion workqueue, but we may still need to track the length to do that. Thoughts?
Brian
| |