lkml.org 
[lkml]   [2021]   [Oct]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [RFC PATCH] mtd: rawnand: use mutex to protect access while in suspend
On Thu, 7 Oct 2021 13:43:51 +0200
Sean Nyekjaer <sean@geanix.com> wrote:

> On Tue, Oct 05, 2021 at 10:58:36AM +0200, Boris Brezillon wrote:
> > On Tue, 5 Oct 2021 10:49:38 +0200
> > Sean Nyekjaer <sean@geanix.com> wrote:
> >
> > > On Tue, Oct 05, 2021 at 10:23:00AM +0200, Boris Brezillon wrote:
> > > > On Tue, 5 Oct 2021 09:09:30 +0200
> > > > Sean Nyekjaer <sean@geanix.com> wrote:
> > >
> > > [ ... ]
> > >
> > > > >
> > > > > Have you seen the reproducer script?
> > > >
> > > > How would I know about this script or your previous attempt (mentioned
> > > > at the end of this email) given I was not Cc-ed on the previous
> > > > discussion, and nothing mentions it in this RFC...
> > > >
> > >
> > > That's why I shared it here ;)
> > > Initially I thought this was a bug introduced by exec_op.
> > >
> > > > > ---
> > > > > root@iwg26-v1:/data/root# cat /data/crash.sh
> > > > > #!/bin/sh -x
> > > > >
> > > > > echo enabled > /sys/devices/platform/soc/2100000.bus/21f4000.serial/tty/ttymxc4/power/wakeup
> > > > >
> > > > > rm /data/test50M
> > > > > dd if=/dev/urandom of=/tmp/test50M bs=1M count=50
> > > > > cp /tmp/test50M /data/ &
> > > > > sleep 1
> > > > > echo mem > /sys/power/state
> > > > > ---
> > > > >
> > > > > As seen in the log above disk is synced before suspend.
> > > > > cp is continuing to copy data to ubifs.
> > > > > And then user space processes are frozen.
> > > > > At this point the kernel thread would have unwritten data.
> > > > >
> > > > > We tried to solve this with:
> > > > > https://lkml.org/lkml/2021/9/1/280
> > > >
> > > > I see. It's still unclear to me when the write happens. Is it in the
> > > > suspend path (before the system is actually suspended), or in the
> > > > resume path (when the system is being resumed).
> > > >
> > > > Anyway, let's admit writing to a storage device while it's suspended is
> > > > a valid use case and requires the storage layer to put this request on
> > > > old. This wait should not, IMHO, be handled at the NAND level, but at
> > > > the MTD level (using a waitqueue, and an atomic to make
> > > > suspended/resumed transitions safe). And abusing a mutex to implement
> > > > that is certainly not a good idea.
> > >
> > > I did't say this was the right solution ;) I actually asked in the RFC:
> > > "Should we introduce a new mutex? Or maybe a spin_lock?"
> > >
> > > What are you proposing, a waitqueue in mtd_info? That gets checked in
> > > mtd_write()/mtd_read()?
> >
> > Yes, and replacing the suspended state by an atomic, and providing a
> > helper to wait on the device readiness. Helper you will call in every
> > path involving a communication with the HW, not just mtd_read/write()
> > (you're missing erase at least, and I fear there are other hooks that
> > might lead to commands being issued to the device). But before we get
> > there, I think it's important to understand what the kernel expects.
> > IOW, if and when threads can do a request on a suspended device, and
> > when it's acceptable to wait (vs returning -EBUSY), otherwise I fear
> > we'll end up with deadlocks in the suspend/resume path.
>
> I have a proposal [0] and yes I have ended up in many deadlocks during
> testing. The hardest part is the locking when going into suspend.
> I'm not sure the wait_queue is initialized the right place :)
> And I'm kinda abusing the nand_get_device() for this...
>
> Who do you think we should add to the discussion?
>
> /Sean
>
> [0]:
> diff --git a/drivers/mtd/nand/raw/nand_base.c b/drivers/mtd/nand/raw/nand_base.c
> index 3d6c6e880520..735dfff18143 100644
> --- a/drivers/mtd/nand/raw/nand_base.c
> +++ b/drivers/mtd/nand/raw/nand_base.c

As I said previously, I think this should be handled MTD level
(drivers/mtd/mtdcore.c) not in the raw NAND framework.

> @@ -337,11 +337,10 @@ static int nand_isbad_bbm(struct nand_chip *chip, loff_t ofs)
> */
> static int nand_get_device(struct nand_chip *chip)
> {
> + struct mtd_info *mtd = nand_to_mtd(chip);
> +
> + wait_event(mtd->wait_queue, atomic_read(&chip->suspended) == 0);
> mutex_lock(&chip->lock);
> - if (chip->suspended) {
> - mutex_unlock(&chip->lock);
> - return -EBUSY;
> - }

There's a race here: the device might enter suspend again before you're
able to acquire the lock.

> mutex_lock(&chip->controller->lock);
>
> return 0;
> @@ -4562,11 +4561,15 @@ static int nand_suspend(struct mtd_info *mtd)
> struct nand_chip *chip = mtd_to_nand(mtd);
> int ret = 0;
>
> + atomic_inc(&chip->suspended);
> mutex_lock(&chip->lock);

And it's racy here as well: you mark the device as suspended before you
even acquired the lock.

> if (chip->ops.suspend)
> ret = chip->ops.suspend(chip);
> - if (!ret)
> - chip->suspended = 1;
> + if (ret) {
> + /* Wake things up again if suspend fails */
> + atomic_dec(&chip->suspended);
> + wake_up(&mtd->wait_queue);
> + }
> mutex_unlock(&chip->lock);
>
> return ret;
> @@ -4581,10 +4584,12 @@ static void nand_resume(struct mtd_info *mtd)
> struct nand_chip *chip = mtd_to_nand(mtd);
>
> mutex_lock(&chip->lock);
> - if (chip->suspended) {
> + if (atomic_read(&chip->suspended)) {
> if (chip->ops.resume)
> chip->ops.resume(chip);
> - chip->suspended = 0;
> +
> + atomic_dec(&chip->suspended);
> + wake_up(&mtd->wait_queue);
> } else {
> pr_err("%s called for a chip which is not in suspended state\n",
> __func__);
> @@ -5099,6 +5104,9 @@ static int nand_detect(struct nand_chip *chip, struct nand_flash_dev *type)
> pr_info("%d MiB, %s, erase size: %d KiB, page size: %d, OOB size: %d\n",
> (int)(targetsize >> 20), nand_is_slc(chip) ? "SLC" : "MLC",
> mtd->erasesize >> 10, mtd->writesize, mtd->oobsize);
> +
> + init_waitqueue_head(&mtd->wait_queue);
> +

It's an MTD field. It should be initialized somewhere in mtdcore.c.

> return 0;
>
> free_detect_allocation:
> @@ -6264,6 +6272,8 @@ static int nand_scan_tail(struct nand_chip *chip)
> if (chip->options & NAND_SKIP_BBTSCAN)
> return 0;
>
> + atomic_set(&chip->suspended, 0);
> +
> /* Build bad block table */
> ret = nand_create_bbt(chip);
> if (ret)
> diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
> index 88227044fc86..f7dcbc336170 100644
> --- a/include/linux/mtd/mtd.h
> +++ b/include/linux/mtd/mtd.h
> @@ -360,6 +360,8 @@ struct mtd_info {
> int (*_get_device) (struct mtd_info *mtd);
> void (*_put_device) (struct mtd_info *mtd);
>
> + wait_queue_head_t wait_queue;
> +

wait_queue doesn't really describe what this waitqueue is used for
(maybe resume_wq), and the suspended state should be here as well
(actually, there's one already).

Actually, what we need is a way to prevent the device from being
suspended while accesses are still in progress, and new accesses from
being queued if a suspend is pending. So, I think you need a readwrite
lock here:

* take the lock in read mode for all IO accesses, check the
mtd->suspended value
- if true, release the lock, and wait (retry on wakeup)
- if false, just do the IO

* take the lock in write mode when you want to suspend/resume the
device and update the suspended field. Call wake_up_all() in the
resume path

\
 
 \ /
  Last update: 2021-10-07 14:20    [W:0.065 / U:0.036 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site