Messages in this thread Patch in this message | | | Date | Sun, 7 Apr 2024 15:09:14 +0200 | From | Oleg Nesterov <> | Subject | Re: Nohz_full on boot CPU is broken (was: Re: [PATCH v2 1/1] wq: Avoid using isolated cpus' timers on queue_delayed_work) |
| |
On 04/05, Frederic Weisbecker wrote: > > +Cc Nick > > Le Fri, Apr 05, 2024 at 04:04:49PM +0200, Oleg Nesterov a écrit : > > On 04/03, Oleg Nesterov wrote: > > > > > > > > OTOH, Documentation/timers/no_hz.rst says > > > > > > > > > > Therefore, the > > > > > boot CPU is prohibited from entering adaptive-ticks mode. Specifying a > > > > > "nohz_full=" mask that includes the boot CPU will result in a boot-time > > > > > error message, and the boot CPU will be removed from the mask. > > > > > > > > > > and this doesn't match the reality. > > > > > > > > Don't some archs allow the boot CPU to go down too tho? If so, this doesn't > > > > really solve the problem, right? > > > > > > I do not know. But I thought about this too. > > > > > > In the context of this discussion we do not care if the boot CPU goes down. > > > But we need at least one housekeeping CPU after cpu_down(). The comment in > > > cpu_down_maps_locked() says > > > > > > Also keep at least one housekeeping cpu onlined > > > > > > but it checks HK_TYPE_DOMAIN, and I do not know (and it is too late for me > > > to try to read the code ;) if housekeeping.cpumasks[HK_TYPE_TIMER] can get > > > empty or not. > > > > This nearly killed me, but I managed to convince myself we shouldn't worry > > about cpu_down(). > > > > HK_FLAG_TIMER implies HK_FLAG_TICK. > > > > HK_FLAG_TICK implies tick_nohz_full_setup() which sets > > tick_nohz_full_mask = non_housekeeping_mask. > > > > When tick_setup_device() is called on a housekeeping CPU it does > > > > else if (tick_do_timer_boot_cpu != -1 && > > !tick_nohz_full_cpu(cpu)) { > > tick_take_do_timer_from_boot(); > > tick_do_timer_boot_cpu = -1; > > > > > > and this sets tick_do_timer_cpu = first-housekeeping-cpu. > > > > cpu_down(tick_do_timer_cpu) will fail, tick_nohz_cpu_down() will nack it. > > > > So cpu_down() can't make housekeeping.cpumasks[HK_FLAG_TIMER] empty and I > > still think that the change below is the right approach. > > > > But probably WARN_ON() in housekeeping_any_cpu() makes sense anyway. > > > > What do you think? > > Good analysis on this nasty housekeeping VS tick code. I promised so many > times to cleanup this mess but things keep piling up. > > It is indeed possible for the boot CPU to be a nohz_full CPU and as > you can see, it's only half-working. This is so ever since: > > 08ae95f4fd3b (nohz_full: Allow the boot CPU to be nohz_full)
Thanks... So this is intentional. I was confused by
Therefore, the boot CPU is prohibited from entering adaptive-ticks mode. Specifying a "nohz_full=" mask that includes the boot CPU will result in a boot-time error message, and the boot CPU will be removed from the mask.
from Documentation/timers/no_hz.rst
> I would love > to revert that now but I don't know if anyone uses this and have it working > by chance somewhere... Should we continue to support a broken feature? Can we > break user ABI if it's already half-broken?
Well, the changelog says
nohz_full has been trialed at a large supercomputer site and found to significantly reduce jitter. In order to deploy it in production, they need CPU0 to be nohz_full
so I guess this feature has users.
But after the commit aae17ebb53cd3da ("workqueue: Avoid using isolated cpus' timers on queue_delayed_work") the kernel will crash at boot time if the boot CPU is nohz_full.
So we need a workaround at least. I am starting to think I will send a trivial patch which changes __queue_delayed_work() to validate the cpu returned by housekeeping_any_cpu(HK_TYPE_TIMER).
But perhaps something like below makes more sense as a (stupid) workaround?
Oleg.
--- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -46,7 +46,15 @@ int housekeeping_any_cpu(enum hk_type type) if (cpu < nr_cpu_ids) return cpu; - return cpumask_any_and(housekeeping.cpumasks[type], cpu_online_mask); + cpu = cpumask_any_and(housekeeping.cpumasks[type], cpu_online_mask); + if (cpu < nr_cpu_ids) + return cpu; + + cpu = READ_ONCE(tick_do_timer_boot_cpu); + if (cpu >= 0) + return cpu; + + WARN_ON_ONCE(1); } } return smp_processor_id();
| |