Messages in this thread | ![/](/images/icornerl.gif) | | Date | Sat, 25 Mar 2023 20:59:50 +0800 | Subject | Re: [PATCH] iommu/vt-d: Fix a IOMMU perfmon warning when CPU hotplug | From | Baolu Lu <> |
| |
On 2023/3/24 23:11, kan.liang@linux.intel.com wrote: > From: Kan Liang<kan.liang@linux.intel.com> > > A warning can be triggered when hotplug CPU 0. > $ echo 0 > /sys/devices/system/cpu/cpu0/online > > [11958.737635] ------------[ cut here ]------------ > [11958.742882] Voluntary context switch within RCU read-side critical > section! > [11958.742891] WARNING: CPU: 0 PID: 19 at kernel/rcu/tree_plugin.h:318 > rcu_note_context_switch+0x4f4/0x580 > [11958.860095] RIP: 0010:rcu_note_context_switch+0x4f4/0x580 > [11958.960360] Call Trace: > [11958.963161] <TASK> > [11958.965565] ? perf_event_update_userpage+0x104/0x150 > [11958.971293] __schedule+0x8d/0x960 > [11958.975165] ? perf_event_set_state.part.82+0x11/0x50 > [11958.980891] schedule+0x44/0xb0 > [11958.984464] schedule_timeout+0x226/0x310 > [11958.989017] ? __perf_event_disable+0x64/0x1a0 > [11958.994054] ? _raw_spin_unlock+0x14/0x30 > [11958.998605] wait_for_completion+0x94/0x130 > [11959.003352] __wait_rcu_gp+0x108/0x130 > [11959.007616] synchronize_rcu+0x67/0x70 > [11959.011876] ? invoke_rcu_core+0xb0/0xb0 > [11959.016333] ? __bpf_trace_rcu_stall_warning+0x10/0x10 > [11959.022147] perf_pmu_migrate_context+0x121/0x370 > [11959.027478] iommu_pmu_cpu_offline+0x6a/0xa0 > [11959.032325] ? iommu_pmu_del+0x1e0/0x1e0 > [11959.036782] cpuhp_invoke_callback+0x129/0x510 > [11959.041825] cpuhp_thread_fun+0x94/0x150 > [11959.046283] smpboot_thread_fn+0x183/0x220 > [11959.050933] ? sort_range+0x20/0x20 > [11959.054902] kthread+0xe6/0x110 > [11959.058479] ? kthread_complete_and_exit+0x20/0x20 > [11959.063911] ret_from_fork+0x1f/0x30 > [11959.067982] </TASK> > [11959.070489] ---[ end trace 0000000000000000 ]--- > > The synchronize_rcu() will be invoked in the perf_pmu_migrate_context(), > when migrating a PMU to a new CPU. However, the current for_each_iommu() > is within RCU read-side critical section. > > Use the dmar_global_lock to replace the RCU read lock when going through > the drhd list.
This introduces the following lockdep warning.
The iommu_pmu_register() path holds the dmar_global_lock and applies for cpuhp_state lock; On the contrary, the iommu_pmu_cpu_offline() path holds the cpuhp_state and applies for the dmar_global_lock.
Can you please have a look into it?
[12861.883945] ====================================================== [12861.884750] WARNING: possible circular locking dependency detected [12861.885556] 6.3.0-rc3-00033-g5ff8930d087a #491 Not tainted [12861.886272] ------------------------------------------------------ [12861.887077] cpuhp/0/19 is trying to acquire lock: [12861.887703] ffffffff91c1d0d0 (dmar_global_lock){++++}-{3:3}, at: iommu_pmu_cpu_offline+0x62/0xd0 [12861.888839] but task is already holding lock: [12861.889605] ffffffff8e26ed40 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_thread_fun+0x4b/0x210 [12861.890701] which lock already depends on the new lock.
[12861.891747] the existing dependency chain (in reverse order) is: [12861.892713] -> #2 (cpuhp_state-down){+.+.}-{0:0}: [12861.893525] lock_acquire+0xc2/0x2a0 [12861.894074] cpuhp_kick_ap_work+0x5b/0x1d0 [12861.894692] _cpu_down+0xc3/0x300 [12861.895210] cpu_down+0x2e/0x50 [12861.895708] device_offline+0x8c/0xc0 [12861.896266] online_store+0x4e/0xa0 [12861.896804] kernfs_fop_write_iter+0x12a/0x1d0 [12861.897462] vfs_write+0x313/0x4c0 [12861.897989] ksys_write+0x60/0xe0 [12861.898507] do_syscall_64+0x43/0x90 [12861.899055] entry_SYSCALL_64_after_hwframe+0x72/0xdc [12861.899783] -> #1 (cpu_hotplug_lock){++++}-{0:0}: [12861.900595] lock_acquire+0xc2/0x2a0 [12861.901143] __cpuhp_setup_state+0x5b/0x1f0 [12861.901772] iommu_pmu_register+0x1a2/0x200 [12861.902400] intel_iommu_init+0x5de/0x880 [12861.903009] pci_iommu_init+0x12/0x40 [12861.903567] do_one_initcall+0x65/0x290 [12861.904155] kernel_init_freeable+0x3cf/0x560 [12861.904804] kernel_init+0x1a/0x140 [12861.905342] ret_from_fork+0x29/0x50 [12861.905890] -> #0 (dmar_global_lock){++++}-{3:3}: [12861.906702] check_prevs_add+0x160/0xee0 [12861.907300] __lock_acquire+0x143b/0x18c0 [12861.907909] lock_acquire+0xc2/0x2a0 [12861.908457] down_write+0x3f/0xc0 [12861.908976] iommu_pmu_cpu_offline+0x62/0xd0 [12861.909614] cpuhp_invoke_callback+0x175/0x670 [12861.910273] cpuhp_thread_fun+0x19d/0x210 [12861.910882] smpboot_thread_fn+0x11d/0x240 [12861.911501] kthread+0xeb/0x120 [12861.911998] ret_from_fork+0x29/0x50 [12861.912546] other info that might help us debug this:
[12861.913572] Chain exists of: dmar_global_lock --> cpu_hotplug_lock --> cpuhp_state-down
[12861.914977] Possible unsafe locking scenario:
[12861.915753] CPU0 CPU1 [12861.916358] ---- ---- [12861.916963] lock(cpuhp_state-down); [12861.917460] lock(cpu_hotplug_lock); [12861.918269] lock(cpuhp_state-down); [12861.919077] lock(dmar_global_lock); [12861.919575] *** DEADLOCK ***
[12861.920350] 2 locks held by cpuhp/0/19: [12861.920865] #0: ffffffff8e26ebd0 (cpu_hotplug_lock){++++}-{0:0}, at: cpuhp_thread_fun+0x47/0x210 [12861.922014] #1: ffffffff8e26ed40 (cpuhp_state-down){+.+.}-{0:0}, at: cpuhp_thread_fun+0x4b/0x210 [12861.923162] stack backtrace: [12861.923747] CPU: 0 PID: 19 Comm: cpuhp/0 Not tainted 6.3.0-rc3-00033-g5ff8930d087a #491 [12861.924775] Hardware name: Intel Corporation BIRCHSTREAM/BIRCHSTREAM, BIOS BHSDREL1.86B.0018.D47.2211031932 11/03/2022 [12861.926133] Call Trace: [12861.926478] <TASK> [12861.926783] dump_stack_lvl+0x4a/0x80 [12861.927282] check_noncircular+0x102/0x120 [12861.927833] ? alloc_list_entry+0x13/0x130 [12861.928384] check_prevs_add+0x160/0xee0 [12861.928918] __lock_acquire+0x143b/0x18c0 [12861.929461] lock_acquire+0xc2/0x2a0 [12861.929950] ? iommu_pmu_cpu_offline+0x62/0xd0 [12861.930550] ? lock_is_held_type+0x9d/0x110 [12861.931113] down_write+0x3f/0xc0 [12861.931572] ? iommu_pmu_cpu_offline+0x62/0xd0 [12861.932171] iommu_pmu_cpu_offline+0x62/0xd0 [12861.932749] ? __pfx_iommu_pmu_cpu_offline+0x10/0x10 [12861.933408] cpuhp_invoke_callback+0x175/0x670 [12861.934007] ? lock_release+0xc3/0x250 [12861.934519] cpuhp_thread_fun+0x19d/0x210 [12861.935059] ? smpboot_thread_fn+0x2f/0x240 [12861.935618] ? smpboot_thread_fn+0x69/0x240 [12861.936177] ? smpboot_thread_fn+0x125/0x240 [12861.936757] smpboot_thread_fn+0x11d/0x240 [12861.937308] ? __pfx_smpboot_thread_fn+0x10/0x10 [12861.937927] kthread+0xeb/0x120 [12861.938365] ? __pfx_kthread+0x10/0x10 [12861.938874] ret_from_fork+0x29/0x50 [12861.939370] </TASK>
Best regards, baolu
| ![\](/images/icornerr.gif) |