Messages in this thread Patch in this message | | | Date | Thu, 26 Mar 2009 02:42:48 -0700 (PDT) | From | David Rientjes <> | Subject | [patch 3/3] slub: sort parital list when thrashing |
| |
Caches that are cpu slab thrashing will scan their entire partial list until a slab is found that will satisfy at least the requisite number of allocations so that it will not be considered thrashing (as defined by /sys/kernel/slab/cache/slab_thrash_ratio).
The partial list can be extremely long and its scanning requires that list_lock is held for that particular node. This can be inefficient if slabs at the head of the list are not appropriate cpu slab replacements.
When an object is freed, the number of free objects for its slab is calculated if the cpu cache is currently thrashing. If it can satisfy the requisite number of allocations so that the slab thrash ratio is exceeded, it is moved to the head of the partial list. This minimizes the time spent holding list_lock and can help cacheline optimizations for recently freed objects.
Cc: Christoph Lameter <cl@linux-foundation.org> Cc: Nick Piggin <nickpiggin@yahoo.com.au> Signed-off-by: David Rientjes <rientjes@google.com> --- mm/slub.c | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-)
diff --git a/mm/slub.c b/mm/slub.c --- a/mm/slub.c +++ b/mm/slub.c @@ -1264,6 +1264,13 @@ static void remove_partial(struct kmem_cache *s, struct page *page) spin_unlock(&n->list_lock); } +static void move_partial_to_head(struct kmem_cache_node *n, struct page *page) +{ + spin_lock(&n->list_lock); + list_move(&page->lru, &n->partial); + spin_unlock(&n->list_lock); +} + /* * Remove from the partial list. * @@ -1726,6 +1733,15 @@ checks_ok: if (unlikely(!prior)) { add_partial(get_node(s, page_to_nid(page)), page, 1); stat(c, FREE_ADD_PARTIAL); + } else if (c->slowpath_allocs >= SLAB_THRASHING_THRESHOLD) { + /* + * If the cache is actively slab thrashing, it's necessary to + * move partial slabs to the head of the list so there isn't + * excessive partial list scanning while holding list_lock. + */ + if (!skip_partial(s, page)) + move_partial_to_head(get_node(s, page_to_nid(page)), + page); } out_unlock:
| |