[lkml]   [2024]   [Apr]   [19]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
Messages in this thread
    SubjectRe: [PATCH v7 00/16] ACPI/arm64: add support for virtual cpu hotplug

    > On 18 Apr 2024, at 13:53, Jonathan Cameron <> wrote:
    > Whilst it is a bit quick after v6, a couple of critical issues
    > were pointed out by Russell, Salil and Rafael + one build issue
    > had been missed, so it seems sensible to make sure those conducting
    > testing or further review have access to a fixed version.
    > v7:
    > - Fix misplaced config guard that broke bisection.
    > - Greatly simplify the condition on which we call
    > acpi_processor_hotadd_init().
    > - Improve teardown ordering.

    Hi Jonathan,

    I've tested v7 on an arm64 machine running QEMU from virt-cpuhp-armv8/rfc-v2, with KVM.

    - boot
    - hotplug up to 'maxcpus'
    - hotunplug down to the number of boot cpus
    - hotplug vcpus and migrate with vcpus offline
    - hotplug vcpus and migrate with vcpus online
    - hotplug vcpus then unplug vcpus then migrate
    - successive live migrations

    Feel free to add:
    Tested-by: Miguel Luis <>

    Thank you

    > Fundamental change v6+: At the level of common ACPI infrastructure, use
    > the existing hotplug path for arm64 even though what needs to be
    > done at the architecture specific level is quite different.
    > An explicit check in arch_register_cpu() for arm64 prevents
    > this code doing anything if Physical CPU Hotplug is signalled.
    > This should resolve any concerns about treating virtual CPU
    > hotplug as if it were physical and potential unwanted side effects
    > if physical CPU hotplug is added to the ARM architecture in the
    > future.
    > v6: Thanks to Rafael for extensive help with the approach + reviews.
    > Specific changes:
    > - Do not differentiate wrt code flow between traditional CPU HP
    > and the new ARM flow. The conditions on performing hotplug actions
    > do need to be adjusted though to incorporate the slightly different
    > state transition
    > to existing !PRESENT + !ENABLED -> PRESENT + ENABLED
    > - Enable ACPI_HOTPLUG_CPU on arm64 and drop the earlier patches that
    > took various code out of the protection of that. Now the paths
    > - New patch to drop unnecessary _STA check in hotplug code. This
    > code cannot be entered unless ENABLED + PRESENT are set.
    > - New patch to unify the flow of already onlined (at time of driver
    > load) and hotplugged CPUs in acpi/processor_driver.c.
    > This change is necessary because we can't easily distinguish the
    > 2 cases of deferred vs hotplug calls of register_cpu() on arm64.
    > It is also a nice simplification.
    > - Use flags rather than a structure for the extra parameter to
    > acpi_scan_check_and_detach() - Thank to Shameer for offline feedback.
    > Updated version of James' original introduction.
    > This series adds what looks like cpuhotplug support to arm64 for use in
    > virtual machines. It does this by moving the cpu_register() calls for
    > architectures that support ACPI into an arch specific call made from
    > the ACPI processor driver.
    > The kubernetes folk really want to be able to add CPUs to an existing VM,
    > in exactly the same way they do on x86. The use-case is pre-booting guests
    > with one CPU, then adding the number that were actually needed when the
    > workload is provisioned.
    > Wait? Doesn't arm64 support cpuhotplug already!?
    > In the arm world, cpuhotplug gets used to mean removing the power from a CPU.
    > The CPU is offline, and remains present. For x86, and ACPI, cpuhotplug
    > has the additional step of physically removing the CPU, so that it isn't
    > present anymore.
    > Arm64 doesn't support this, and can't support it: CPUs are really a slice
    > of the SoC, and there is not enough information in the existing ACPI tables
    > to describe which bits of the slice also got removed. Without a reference
    > machine: adding this support to the spec is a wild goose chase.
    > Critically: everything described in the firmware tables must remain present.
    > For a virtual machine this is easy as all the other bits of 'virtual SoC'
    > are emulated, so they can (and do) remain present when a vCPU is 'removed'.
    > On a system that supports cpuhotplug the MADT has to describe every possible
    > CPU at boot. Under KVM, the vGIC needs to know about every possible vCPU before
    > the guest is started.
    > With these constraints, virtual-cpuhotplug is really just a hypervisor/firmware
    > policy about which CPUs can be brought online.
    > This series adds support for virtual-cpuhotplug as exactly that: firmware
    > policy. This may even work on a physical machine too; for a guest the part of
    > firmware is played by the VMM. (typically Qemu).
    > PSCI support is modified to return 'DENIED' if the CPU can't be brought
    > online/enabled yet. The CPU object's _STA method's enabled bit is used to
    > indicate firmware's current disposition. If the CPU has its enabled bit clear,
    > it will not be registered with sysfs, and attempts to bring it online will
    > fail. The notifications that _STA has changed its value then work in the same
    > way as physical hotplug, and firmware can cause the CPU to be registered some
    > time later, allowing it to be brought online.
    > This creates something that looks like cpuhotplug to user-space and the
    > kernel beyond arm64 architecture specific code, as the sysfs
    > files appear and disappear, and the udev notifications look the same.
    > One notable difference is the CPU present mask, which is exposed via sysfs.
    > Because the CPUs remain present throughout, they can still be seen in that mask.
    > This value does get used by webbrowsers to estimate the number of CPUs
    > as the CPU online mask is constantly changed on mobile phones.
    > Linux is tolerant of PSCI returning errors, as its always been allowed to do
    > that. To avoid confusing OS that can't tolerate this, we needed an additional
    > bit in the MADT GICC flags. This series copies ACPI_MADT_ONLINE_CAPABLE, which
    > appears to be for this purpose, but calls it ACPI_MADT_GICC_CPU_CAPABLE as it
    > has a different bit position in the GICC.
    > This code is unconditionally enabled for all ACPI architectures, though for
    > now only arm64 will have deferred the cpu_register() calls.
    > If folk want to play along at home, you'll need a copy of Qemu that supports this.
    > virt-cpuhp-armv8/rfc-v2
    > Replace your '-smp' argument with something like:
    > | -smp cpus=1,maxcpus=3,cores=3,threads=1,sockets=1
    > then feed the following to the Qemu montior;
    > | (qemu) device_add driver=host-arm-cpu,core-id=1,id=cpu1
    > | (qemu) device_del cpu1
    > James Morse (7):
    > ACPI: processor: Register deferred CPUs from acpi_processor_get_info()
    > ACPI: Add post_eject to struct acpi_scan_handler for cpu hotplug
    > arm64: acpi: Move get_cpu_for_acpi_id() to a header
    > irqchip/gic-v3: Don't return errors from gic_acpi_match_gicc()
    > irqchip/gic-v3: Add support for ACPI's disabled but 'online capable'
    > CPUs
    > arm64: document virtual CPU hotplug's expectations
    > cpumask: Add enabled cpumask for present CPUs that can be brought
    > online
    > Jean-Philippe Brucker (1):
    > arm64: psci: Ignore DENIED CPUs
    > Jonathan Cameron (8):
    > ACPI: processor: Simplify initial onlining to use same path for cold
    > and hotplug
    > cpu: Do not warn on arch_register_cpu() returning -EPROBE_DEFER
    > ACPI: processor: Drop duplicated check on _STA (enabled + present)
    > ACPI: processor: Move checks and availability of acpi_processor
    > earlier
    > ACPI: processor: Add acpi_get_processor_handle() helper
    > ACPI: scan: switch to flags for acpi_scan_check_and_detach()
    > arm64: arch_register_cpu() variant to check if an ACPI handle is now
    > available.
    > arm64: Kconfig: Enable hotplug CPU on arm64 if ACPI_PROCESSOR is
    > enabled.
    > .../ABI/testing/sysfs-devices-system-cpu | 6 +
    > Documentation/arch/arm64/cpu-hotplug.rst | 79 ++++++++++++
    > Documentation/arch/arm64/index.rst | 1 +
    > arch/arm64/Kconfig | 1 +
    > arch/arm64/include/asm/acpi.h | 11 ++
    > arch/arm64/kernel/acpi.c | 16 +++
    > arch/arm64/kernel/acpi_numa.c | 11 --
    > arch/arm64/kernel/psci.c | 2 +-
    > arch/arm64/kernel/smp.c | 56 ++++++++-
    > drivers/acpi/acpi_processor.c | 113 ++++++++++--------
    > drivers/acpi/processor_driver.c | 44 ++-----
    > drivers/acpi/scan.c | 47 ++++++--
    > drivers/base/cpu.c | 12 +-
    > drivers/irqchip/irq-gic-v3.c | 32 +++--
    > include/acpi/acpi_bus.h | 1 +
    > include/acpi/processor.h | 2 +-
    > include/linux/acpi.h | 10 +-
    > include/linux/cpumask.h | 25 ++++
    > kernel/cpu.c | 3 +
    > 19 files changed, 357 insertions(+), 115 deletions(-)
    > create mode 100644 Documentation/arch/arm64/cpu-hotplug.rst
    > --
    > 2.39.2

     \ /
      Last update: 2024-05-27 16:47    [W:3.778 / U:0.024 seconds]
    ©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site