Messages in this thread | | | Date | Fri, 14 Dec 2007 08:16:34 +0100 | From | Eric Dumazet <> | Subject | Re: tipc_init(), WARNING: at arch/x86/mm/highmem_32.c:52, [2.6.24-rc4-git5: Reported regressions from 2.6.23] |
| |
Christoph Lameter a écrit : > On Sat, 8 Dec 2007, Ingo Molnar wrote: > >>> Good. Although we should perhaps look at that reported performance >>> problem with SLUB. It looks like SLUB will do a memclear() for the >>> area twice (first for the whole page, then for the thing it allocated) >>> for the slow case. Maybe that exacerbates the problem. >> i dont think the SLUB problem could be explained purely via a double >> memset(). [which ought to be extremely fast anyway] We are talking about >> a 10 times slowdown on a 64-way box of a workload that is fairly >> common-sense. (tasks sending messages to each other via bog standard >> means) >> >> while i dont want to jump to conclusions without looking at some >> profiles, i think the SLUB performance regression is indicative of the >> following fallacy: "SLAB can be done significantly simpler while keeping >> the same performance". > > Well this is double crap. First of all SLUB does not do memclear twice. > There is no reason to assume that SLUB has the problem just because SLOB > hat that. A "fix" for that nonexistent problem went into Linus tree. WTH > is going on? > > SLUB was done because of a series of problem with the basic concepts of > SLAB that treaten it usability in the future. > >> I couldnt point to any particular aspect of SLAB that i could >> characterise as "needless bloat". > > I agree, SLABs architecture is pretty tight and I was one of those who > helped it along to be that way. > > However, SLAB is just fundamentally wrong for todays machine. The key > problem today is cacheline fetch latency and that problem will increase > significantly in the future. Sure under some circumstances that exploit > the fact that SLAB sometimes gets its guesses on the cpu cache right SLAB > can still win but the more processors and nodes we get the more it will > become difficult to keep SLAB around and the more it will become > difficult to establish what cachelines are in the cpu cache. > >> I think we should we make SLAB the default for v2.6.24 ... > > If you guarantee that all the regression of SLAB vs. SLUB are addressed > then thats fine but AFAICT that is not possible. > > Here is a list of some of the benefits of SLUB just in case we forgot: > > > - SLUB is performance wise much faster than SLAB. This can be more than a > factor of 10 (case of concurrent allocations / frees on multiple > processors). See http://lkml.org/lkml/2007/10/27/245 > > - Single threaded allocation speed is up to double that of SLAB > > - Remote freeing of objectcs in a NUMA systems is typically 30% faster. > > - Debugging on SLAB is difficult. Requires recompile of the kernel > and the resulting output is difficult to interpret. SLUB can apply > debugging options to a subset of the slabcaches in order to allow > the system to work with maximum speed. This is necessary to detect > difficult to reproduce race conditions. > > - SLAB can capture huge amounts of memory in its queues. The problem > gets worse the more processors and NUMA nodes are in the system. The > amount of memory limits the number of per cpu objects one can configure. > > - SLAB requires a pass through all slab caches every 2 seconds to > expire objects. This is a problem both for realtime and MPI jobs > that cannot take such a processor outage. > > - SLAB does not have a sophisticated slabinfo tool to report the > state of slab objects on the system. Can provide details of > object use. > > - SLAB requires the update of two words for freeing > and allocation. SLUB can do that by updating a single > word which allows to avoid enabling and disabling interrupts if > the processor supports an atomic instruction for that purpose. > This is important for realtime kernels where special measures > may have to be implemented if one wants to disable interrupts. > > - SLAB requires memory to be set aside for queues (processors > times number of slabs times queue size). SLUB requires none of that. > > - SLUB merges slab caches with similar characteristics to > reduce the memory footprint even further. > > - SLAB performs object level NUMA management which creates > a complex allocator complexity. SLUB manages NUMA on the level of > slab pages reducing object management overhead. > > - SLUB allows remote node defragmentation to avoid the buildup > of large partial lists on a single node. > > - SLUB can actively reduce the fragmentation of slabs through > slab cache specific callbacks (not merged yet) > > - SLUB has resiliency features that allow it to isolate a problem > object and continue after diagnostics have been performed. > > - SLUB creates rarely used DMA caches on demand instead of creating > them all on bootup (SLAB). >
Yes, SLUB should be the way to go, but some issues are not yet solved.
I had to switch back to SLAB on a production NUMA server, with 2 nodes and 8GB ram. Using a lot of sockets, so a large part of memory was used by kernel.
SLUB kernel was hitting OOM after 2 or 3 days of uptime. SLAB kernel never hit this.
Unfortunatly I dont have a test machine to reproduce the setup.
Maybe the problem is not related to SLUB at all, but an underlying VM/NUMA bug.
The /proc/buddyinfo showed that :
Node 0 contained two zones (DMA and DMA32) total 4 GB Node 1 contained one zone (Normal) total 4 GB
So Node 0 contained no (Normal) zone
part of /proc/meminfo
Slab: 3338512 kB SReclaimable: 789716 kB SUnreclaim: 2548796 kB
I remember network interrupts were taken by CPU 1, so most allocations were done by CPU 1 (node 1), and many freeing were done on CPU 0
Hope this helps
-- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |