Messages in this thread | | | From | Valentin Schneider <> | Subject | [RT BUG] Stall caused by eventpoll, rwlocks and CFS bandwidth controller | Date | Thu, 12 Oct 2023 17:07:02 +0200 |
| |
Hi folks,
We've had reports of stalls happening on our v6.0-ish frankenkernels, and while we haven't been able to come out with a reproducer (yet), I don't see anything upstream that would prevent them from happening.
The setup involves eventpoll, CFS bandwidth controller and timer expiry, and the sequence looks as follows (time-ordered):
p_read (on CPUn, CFS with bandwidth controller active) ======
ep_poll_callback() read_lock_irqsave() ... try_to_wake_up() <- enqueue causes an update_curr() + sets need_resched due to having no more runtime preempt_enable() preempt_schedule() <- switch out due to p_read being now throttled
p_write =======
ep_poll() write_lock_irq() <- blocks due to having active readers (p_read)
ktimers/n =========
timerfd_tmrproc() `\ ep_poll_callback() `\ read_lock_irqsave() <- blocks due to having active writer (p_write)
From this point we have a circular dependency:
p_read -> ktimers/n (to replenish runtime of p_read) ktimers/n -> p_write (to let ktimers/n acquire the readlock) p_write -> p_read (to let p_write acquire the writelock)
IIUC reverting 286deb7ec03d ("locking/rwbase: Mitigate indefinite writer starvation") should unblock this as the ktimers/n thread wouldn't block, but then we're back to having the indefinite starvation so I wouldn't necessarily call this a win.
Two options I'm seeing: - Prevent p_read from being preempted when it's doing the wakeups under the readlock (icky) - Prevent ktimers / ksoftirqd (*) from running the wakeups that have ep_poll_callback() as a wait_queue_entry callback. Punting that to e.g. a kworker /should/ do.
(*) It's not just timerfd, I've also seen it via net::sock_def_readable - it should be anything that's pollable.
I'm still scratching my head on this, so any suggestions/comments welcome!
Cheers, Valentin
| |