lkml.org 
[lkml]   [2004]   [May]   [7]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
Subject[ANNOUNCE] [PATCH] Node Hotplug Support
Hi,

I'm announcing a new project "LHNS (Linux Hotplug Node Support)"
and patches for 2.6.5:

http://sourceforge.net/projects/lhns/



Goal
====
The main goal of LHNS is to support "node hotplug" in Linux.

Where, "node" is a container device that contains CPU, memory,
and/or I/O devices. The definition of "node" is different from one
of "NUMA node".

- "NUMA node" is: a block of memory and the CPUs, I/O, etc.
physically on the same bus as the memory (according to NUMA
project website, this is a common definition).

- "node" used here is: a hotpluggable hardware that contains
CPUs, memory, and/or I/O.

The "node" and "NUMA node" are not always the same. (I'm
sorry for the inconvenient :( For instance, the "node" could be a
device that contains:

- processors, memory, and I/O devices
- processors and memory
- processors only
- memory only
- I/O devices only
- etc

"node hotplug" allows you to hot-add and hot-remove "node"
without stopping the system.



Why?
====
Someone might think like "Why don't we invoke CPU, memory, IO
hotplug individually without node hotplug?". However, CPU,
memory, and IO hotplug cannot remove a node (container device)
from the system physically. That's a node hotplug's job. Also,
when hotplug request occurs on a node, node hotplug searchs
resouces on the node, and invokes CPU, memory, and IO hotplug
in proper order. Actually, the order is very important. For instance,
when hot-adding a NUMA node, memory should be added first
so that CPUs could allocate data in the memory while the CPUs is
being added. Otherwise, CPUs need to allocate it in other memory
on other node. This might cause performance issue.



Design
======
ACPI is used to do some hardware manipulation.
There is no general purpose interface to get hardware information
and manipulate hardware today, but hardware proprietary interfaces.
ACPI is one of them, and I decided to use it because:

- Its spec is open.
- I can use it without any hardware special knowledge:)

The following assumptions are necessary for node hotplug:

- The system has hotpluggable "node" (hardware).
- Each of "node" is defined as a container device (HID=PNP0A05)
in ACPI namespace.
- Existing CPU, memory, I/O hotplug (LHCS, LHMS, PCI HotPlug
for Linux, etc) produce a hook (function) for Node hotplug.

"node hotplug" consists of one main part and three sub parts so far:

- ACPI container device hotplug (main)
- ACPI based CPU hotplug (sub)
- ACPI based memory hotplug (sub)
- ACPI based I/O hotplug (sub)

"ACPI container device hotplug" is: the main part of node hotplug,
which is supposed to do:

[hot-addition]
1. Its handler is invoked when ACPI notify is occured by a node
attached to the system.
2. This creates data structures for the node.
If CONFIG_ACPI_NUMA_CONTAINER=y, this handles NUMA
related data also (not implemented yet).
3. This invokes ACPI based CPU/memory/IO hotplug in proper
order with proper arguments (e.g. acpi_handle of the added node)
4. This notifies userland that the node is added/removed.
5. This writes results of hotplug processing to a log file.

[hot-removal]
1. Its handler is invoked when ACPI notify is occured or user issues
hot-removal request via sysfs.
2. This deletes data structures of the node.
If CONFIG_ACPI_NUMA_CONTAINER=y, this handles NUMA
related data also (not implemented yet).
3. This invokes ACPI based CPU/memory/IO hotplug in proper
order with proper arguments (e.g. acpi_handle of the node)
4. This evaluates _EJ0 method of the node.
5. This notifies userland that the node is added/removed.
6. This writes results of hotplugging into log file.

"ACPI based CPU/memory/IO hotplug" are: the sub part, which are
supposed to do:

1. They are invoked by the main part with the argument.
2. They search ACPI namespace to check if there are any devices
on the node that they should handle. For instance, if ACPI
based CPU hotplug searches ACPI namespace to see if there
are any CPUs on an added node.
3. If they find devices, they do something and call existing hotplug
features:
o LHCS (Linux Hotplug CPU Support)
o LHMS (Linux Hotplug Memory Support)
o PCI HotPlug for Linux
o etc.
with some appropriate arguments (TBD).

Then each hotplug handles each device hotplug.

See http://lhns.sourceforge.net/ for more information and figures
(module and component diagram).



Patches
=======
The following nine patches are available at:

http://sourceforge.net/projects/lhns/
(Actually, these files are packed into .bz2 file.)

- p00001_sci_emu.patch:
SCI interrupt emulation.

- p00002_over-20030109.patch:
Override an original BIOS's DSDT with a DSDT that you make up.

- p00003_procfs_util.patch
Extension for procfs.
A new function remove_recursive_proc_entry().

- p00004_acpi_processor_bug.patch
Replace remove_proc_entry() to remove_recursive_proc_entry()
in drivers/acpi/processor.c to avoid getting Call Trace.

- p00005_acpi_hp_config.patch
Kconfigs and Makefiles for ACPI hotplug.

- p00006_acpi_core.patch
Changes to ACPI core part for hotplug.
A new function acpi_bus_scan_free(), etc...

- p00007_acpi_hp_util.patch
Utility functions for ACPI hotplug.

- p00008_acpi_container_hp.patch
The main part of node hotplug.

- p00009_acpi_container_hp_cpu.patch
The sub part (CPU) of node hotplug.



Current Status
==============
The first patches have been tested in an emulation and a virtual
environment without LHCS, LHMS, etc's codes. However, they are
very early versions of the code, so there are still many rough codes,
not Linux style codes, and TBDs. The first patches are just nice
to show my idea and how they work. Some sub parts and NUMA
node support have not been implemented yet. (Please don't
enable "Hotplug memory" and "Hotplug IO" when you're doing
"make config":)



Usage
=====
See "Documents" section at http://lhns.sourceforge.net/. There
are some documents and instructions:

- How to build the kernel with LHNS patches
- How to make a fake DSDT
- How to generate SCI interrupt (emulation)
- How to invoke hot-addition and hot-removal for container hotplug

Please feel free to take it for a spin!



Plan
====
First of all, I'd like to discuss the design of node hotplug and
interface with hotplug folks. After I get feedback from the people,
I'll update my patches. Also I'll release the following features in
the near future. Then I'll make them work all together and test
them.

- NUMA node support
- ACPI based memory hotplug
- ACPI based IO hotplug
- H2P bridge hotplug
- P2P brdige hotplug
- IOSAPIC hotplug



Any comments, bug reports, or suggestions are welcome.

Thanks,
Keiichiro Tokunaga
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:03    [W:0.111 / U:0.180 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site