Messages in this thread | ![/](/images/icornerl.gif) | | Date | Fri, 5 Apr 2024 23:52:03 +0200 | From | Frederic Weisbecker <> | Subject | Nohz_full on boot CPU is broken (was: Re: [PATCH v2 1/1] wq: Avoid using isolated cpus' timers on queue_delayed_work) |
| |
+Cc Nick
Le Fri, Apr 05, 2024 at 04:04:49PM +0200, Oleg Nesterov a écrit : > On 04/03, Oleg Nesterov wrote: > > > > > > OTOH, Documentation/timers/no_hz.rst says > > > > > > > > Therefore, the > > > > boot CPU is prohibited from entering adaptive-ticks mode. Specifying a > > > > "nohz_full=" mask that includes the boot CPU will result in a boot-time > > > > error message, and the boot CPU will be removed from the mask. > > > > > > > > and this doesn't match the reality. > > > > > > Don't some archs allow the boot CPU to go down too tho? If so, this doesn't > > > really solve the problem, right? > > > > I do not know. But I thought about this too. > > > > In the context of this discussion we do not care if the boot CPU goes down. > > But we need at least one housekeeping CPU after cpu_down(). The comment in > > cpu_down_maps_locked() says > > > > Also keep at least one housekeeping cpu onlined > > > > but it checks HK_TYPE_DOMAIN, and I do not know (and it is too late for me > > to try to read the code ;) if housekeeping.cpumasks[HK_TYPE_TIMER] can get > > empty or not. > > This nearly killed me, but I managed to convince myself we shouldn't worry > about cpu_down(). > > HK_FLAG_TIMER implies HK_FLAG_TICK. > > HK_FLAG_TICK implies tick_nohz_full_setup() which sets > tick_nohz_full_mask = non_housekeeping_mask. > > When tick_setup_device() is called on a housekeeping CPU it does > > else if (tick_do_timer_boot_cpu != -1 && > !tick_nohz_full_cpu(cpu)) { > tick_take_do_timer_from_boot(); > tick_do_timer_boot_cpu = -1; > > > and this sets tick_do_timer_cpu = first-housekeeping-cpu. > > cpu_down(tick_do_timer_cpu) will fail, tick_nohz_cpu_down() will nack it. > > So cpu_down() can't make housekeeping.cpumasks[HK_FLAG_TIMER] empty and I > still think that the change below is the right approach. > > But probably WARN_ON() in housekeeping_any_cpu() makes sense anyway. > > What do you think?
Good analysis on this nasty housekeeping VS tick code. I promised so many times to cleanup this mess but things keep piling up.
It is indeed possible for the boot CPU to be a nohz_full CPU and as you can see, it's only half-working. This is so ever since:
08ae95f4fd3b (nohz_full: Allow the boot CPU to be nohz_full)
I wish I had nacked it before it got merged, especially as the changelog mentions that the user could have solved this with modifying its setup... I would love to revert that now but I don't know if anyone uses this and have it working by chance somewhere... Should we continue to support a broken feature? Can we break user ABI if it's already half-broken?
Anyway so during boot it's possible to have an empty housekeeping_mask(HK_TYPE_TIMER) & cpu_online_mask. After boot though (provided any CPU from the housekeeping_mask(HK_TYPE_TIMER) has actually booted, which isn't even guaranteed if maxcpus= is passed...) the first online housekeeping can't go down like you spotted.
Thanks.
| ![\](/images/icornerr.gif) |