lkml.org 
[lkml]   [2008]   [Jun]   [9]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
    /
    Date
    From
    SubjectRe: [RFC][PATCH 1/2] memcg: res_counter hierarchy
    KAMEZAWA Hiroyuki wrote:
    > On Mon, 09 Jun 2008 15:18:47 +0530
    > Balbir Singh <balbir@linux.vnet.ibm.com> wrote:
    >
    >> KAMEZAWA Hiroyuki wrote:
    >>> A simple hard-wall hierarhcy support for res_counter.
    >>>
    >>> Changelog v2->v3
    >>> - changed the name and arguments of functions.
    >>> - rewrote to be read easily.
    >>> - named as HardWall hierarchy.
    >>>
    >>> This implements following model
    >>> - A cgroup's tree means hierarchy of resource.
    >>> - All child's resource is moved from its parents.
    >>> - The resource moved to children is charged as parent's usage.
    >>> - The resource moves when child->limit is changed.
    >>> - The sum of resource for children and its own usage is limited by "limit".
    >>>
    >>> This implies
    >>> - No dynamic automatic hierarhcy balancing in the kernel.
    >>> - Each resource is isolated completely.
    >>> - The kernel just supports resource-move-at-change-in-limit.
    >>> - The user (middle-ware) is responsible to make hierarhcy balanced well.
    >> We'd definitely like to see a user level tool/application as a demo of how this
    >> can be achieved.
    >>
    > I don't have one, now. I'll write one when I have time. Need now ?
    > Hmm...maybe I(we) need some more patches to implement useful statistics,
    > notifier to middlewares.
    >

    Yes, we need more useful statistics.

    >
    >
    >>> Good balance can be achieved by changing limit from user land.
    >>>
    >>>
    >>> Background:
    >>> Recently, there are popular resource isolation technique widely used,
    >>> i.e. Hardware-Virtualization. We can do hierarchical resource isolation
    >>> by using cgroup on it. But supporting hierarchy management in croups
    >>> has some advantages of performance, unity and costs of management.
    >>>
    >>> There are good resource management in other OSs, they support some kind of
    >>> hierarchical resource management. We wonder what kind of hierarchy policy
    >>> is good for Linux. And there is an another point. Hierarchical system can be
    >>> implemented by the kernel and user-land co-operation. So, there are various
    >>> choices to do in the kernel. Doing all in the kernel or export some proper
    >>> interfaces to the user-land. Middle-wares are tend to be used for management.
    >>> I hope there will be Open Source one.
    >>>
    >>> At supporting hierarchy in cgroup, several aspects of characteristics of
    >>> policy of hierarchy can be considered. Some needs automatic balancing
    >>> between several groups.
    >>>
    >>> - fairness ... how fairness is kept under policy
    >>>
    >>> - performance ... should be _fast_. multi-level resource balancing tend
    >>> to use much amount of CPU and can cause soft lockup.
    >>>
    >>> - predictability ... resource management are usually used for resource
    >>> isolation. the kernel must not break the isolation and
    >>> predictability of users against application's progress.
    >>>
    >>> - flexibility ... some sophisticated dynamic resource balancing with
    >>> soft-limit is welcomed when the user doesn't want strict
    >>> resource isolation or when the user cannot estimate how much
    >>> they want correctly.
    >> Soft limits has been on my plate for a while now. I'll take a crack at it. At
    >> the moment the statistics is a bit of a worry, since users/administrators need
    >> good statistics to take further action.
    >>
    > Yes, statistics is not enough now.
    >
    >
    >
    >>> Hard Wall Hierarchy.
    >>>
    >>> This patch implements a hard-wall model of hierarchy for resources.
    >>> Works well for users who want strict resource isolation.
    >>>
    >>> This model allows the move of resource only between a parent and its children.
    >>> The resource is moved to a child when it declares the amount of resources to be
    >>> used. (by limit)
    >> The other reason for preferring a shares based approach is that, the it will be
    >> more in line with the CPU controllers interfaces.
    >>
    >
    > You have to think of the major difference of tha nature of CPU and Memory.
    > We have to reclaim the resource with some feedbacks among sevral cgroups.
    > But ok, if it's can be implemented in simple way.
    > I have no objections if cost is very low. My concern is only performance.
    > (and maintenance)
    >

    True, I don't see hierarchy as adding too much additional cost.

    >
    >>> Index: temp-2.6.26-rc2-mm1/include/linux/res_counter.h
    >>> ===================================================================
    >>> --- temp-2.6.26-rc2-mm1.orig/include/linux/res_counter.h
    >>> +++ temp-2.6.26-rc2-mm1/include/linux/res_counter.h
    >>> @@ -38,6 +38,16 @@ struct res_counter {
    >>> * the number of unsuccessful attempts to consume the resource
    >>> */
    >>> unsigned long long failcnt;
    >>> +
    >>> + /*
    >>> + * hierarchy support: the parent of this resource.
    >>> + */
    >>> + struct res_counter *parent;
    >>> + /*
    >>> + * the amount of resources assigned to children.
    >>> + */
    >>> + unsigned long long for_children;
    >>> +
    >> I would prefer to use a better name, lent_out? reserved_for_children?
    >> borrowed_by_children?
    >>
    > ok. use other names.
    >
    >
    >
    >>> /*
    >>> * the lock to protect all of the above.
    >>> * the routines below consider this to be IRQ-safe
    >>> @@ -63,9 +73,20 @@ u64 res_counter_read_u64(struct res_coun
    >>> ssize_t res_counter_read(struct res_counter *counter, int member,
    >>> const char __user *buf, size_t nbytes, loff_t *pos,
    >>> int (*read_strategy)(unsigned long long val, char *s));
    >>> +
    >>> +/*
    >>> + * An interface for setting res_counter's member (ex. limit)
    >>> + * the new parameter is passed by *buf and translated by write_strategy().
    >>> + * Then, it is applied to member under the control of set_strategy().
    >>> + * If write_strategy() and set_strategy() can be NULL. see res_counter.c
    >>> + */
    >>> +
    >>> ssize_t res_counter_write(struct res_counter *counter, int member,
    >>> - const char __user *buf, size_t nbytes, loff_t *pos,
    >>> - int (*write_strategy)(char *buf, unsigned long long *val));
    >>> + const char __user *buf, size_t nbytes, loff_t *pos,
    >>> + int (*write_strategy)(char *buf, unsigned long long *val),
    >>> + int (*set_strategy)(struct res_counter *res, unsigned long long val,
    >>> + int what),
    >>> + );
    >>>
    >>> /*
    >>> * the field descriptors. one for each member of res_counter
    >>> @@ -76,15 +97,33 @@ enum {
    >>> RES_MAX_USAGE,
    >>> RES_LIMIT,
    >>> RES_FAILCNT,
    >>> + RES_FOR_CHILDREN,
    >> RES_BORROWED? RES_BORROWED_BY_CHILDREN?
    >>
    > ok, again.
    >
    >>> };
    >>>
    >>> /*
    >>> * helpers for accounting
    >>> */
    >>>
    >>> +/*
    >>> + * initialize res_counter.
    >>> + * @counter : the counter
    >>> + *
    >>> + * initialize res_counter and set default limit to very big value(unlimited)
    >>> + */
    >>> +
    >>> void res_counter_init(struct res_counter *counter);
    >>>
    >>> /*
    >>> + * initialize res_counter under hierarchy.
    >>> + * @counter : the counter
    >>> + * @parent : the parent of the counter
    >>> + *
    >>> + * initialize res_counter and set default limit to 0. and set "parent".
    >>> + */
    >>> +void res_counter_init_hierarchy(struct res_counter *counter,
    >>> + struct res_counter *parent);
    >>> +
    >>> +/*
    >>> * charge - try to consume more resource.
    >>> *
    >>> * @counter: the counter
    >>> @@ -153,4 +192,51 @@ static inline void res_counter_reset_fai
    >>> cnt->failcnt = 0;
    >>> spin_unlock_irqrestore(&cnt->lock, flags);
    >>> }
    >>> +
    >>> +/**
    >>> + * Move resources from a parent to a child.
    >>> + * At success,
    >>> + * parent->usage += val.
    >>> + * parent->for_children += val.
    >>> + * child->limit += val.
    >>> + *
    >>> + * @child: an entity to set res->limit. The parent is child->parent.
    >>> + * @val: the amount of resource to be moved.
    >>> + * @callback: called when the parent's free resource is not enough to be moved.
    >>> + * this can be NULL if no callback is necessary.
    >>> + * @retry: limit for the number of trying to callback.
    >>> + * -1 means infinite loop. At each retry, yield() is called.
    >>> + * Returns 0 at success, !0 at failure.
    >>> + *
    >>> + * The callback returns 0 at success, !0 at failure.
    >>> + *
    >>> + */
    >>> +
    >>> +int res_counter_move_resource(struct res_counter *child,
    >>> + unsigned long long val,
    >>> + int (*callback)(struct res_counter *res, unsigned long long val),
    >>> + int retry);
    >>> +
    >>> +
    >>> +/**
    >>> + * Return resource to its parent.
    >>> + * At success,
    >>> + * parent->usage -= val.
    >>> + * parent->for_children -= val.
    >>> + * child->limit -= val.
    >>> + *
    >>> + * @child: entry to resize. The parent is child->parent.
    >>> + * @val : How much does child repay to parent ? -1 means 'all'
    >>> + * @callback: A callback for decreasing resource usage of child before
    >>> + * returning. If NULL, just deceases child's limit.
    >>> + * @retry: # of retries at calling callback for freeing resource.
    >>> + * -1 means infinite loop. At each retry, yield() is called.
    >>> + * Returns 0 at success.
    >>> + */
    >>> +
    >>> +int res_counter_return_resource(struct res_counter *child,
    >>> + unsigned long long val,
    >>> + int (*callback)(struct res_counter *res, unsigned long long val),
    >>> + int retry);
    >>> +
    >>> #endif
    >>> Index: temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
    >>> ===================================================================
    >>> --- temp-2.6.26-rc2-mm1.orig/Documentation/controllers/resource_counter.txt
    >>> +++ temp-2.6.26-rc2-mm1/Documentation/controllers/resource_counter.txt
    >>> @@ -44,6 +44,13 @@ to work with it.
    >>> Protects changes of the above values.
    >>>
    >>>
    >>> + f. struct res_counter *parent
    >>> +
    >>> + Parent res_counter under hierarchy.
    >>> +
    >>> + g. unsigned long long for_children
    >>> +
    >>> + Resources assigned to children. This is included in usage.
    >>>
    >>> 2. Basic accounting routines
    >>>
    >>> @@ -179,3 +186,37 @@ counter fields. They are recommended to
    >>> still can help with it).
    >>>
    >>> c. Compile and run :)
    >>> +
    >>> +
    >>> +6. Hierarchy
    >>> + a. No Hierarchy
    >>> + each cgroup can use its own private resource.
    >>> +
    >>> + b. Hard-wall Hierarhcy
    >>> + A simple hierarchical tree system for resource isolation.
    >>> + Allows moving resources only between a parent and its children.
    >>> + A parent can move its resource to children and remember the amount to
    >>> + for_children member. A child can get new resource only from its parent.
    >>> + Limit of a child is the amount of resource which is moved from its parent.
    >>> +
    >> OK, after reading this I am totally sure I want a shares based interface. Limits
    >> are not shared like this.
    >>
    >> A child and a parent should both be capable of having a limit of 1G, but they
    >> could use different shares factors to govern, how much each children will get.
    >> Doing it this way, breaks limit semantics.
    >>
    > Not easy to use in my point of view. Can we use 'share' in proper way
    > on no-swap machine ?
    >

    Not sure I understand your question. Share represents the share of available
    resources.

    >
    >>> + When add "val" to a child,
    >>> + parent->usage += val
    >>> + parent->for_children += val
    >>> + child->limit += val
    >>> + When a child returns its resource
    >>> + parent->usage -= val
    >>> + parent->for_children -= val
    >>> + child->limit -= val.
    >>> +
    >>> + This implements resource isolation among each group. This works very well
    >>> + when you want to use strict resource isolation.
    >>> +
    >>> + Usage Hint:
    >>> + This seems for static resource assignment but dynamic resource re-assignment
    >>> + can be done by resetting "limit" of groups. When you consider "limit" as
    >>> + the amount of allowed _current_ resource, a sophisticated resource management
    >>> + system based on strict resource isolation can be implemented.
    >>> +
    >>> +c. Soft-wall Hierarchy
    >>> + TBD.
    >>> +
    >>> Index: temp-2.6.26-rc2-mm1/kernel/res_counter.c
    >>> ===================================================================
    >>> --- temp-2.6.26-rc2-mm1.orig/kernel/res_counter.c
    >>> +++ temp-2.6.26-rc2-mm1/kernel/res_counter.c
    >>> @@ -20,6 +20,14 @@ void res_counter_init(struct res_counter
    >>> counter->limit = (unsigned long long)LLONG_MAX;
    >>> }
    >>>
    >>> +void res_counter_init_hierarchy(struct res_counter *counter,
    >>> + struct res_counter *parent)
    >>> +{
    >>> + spin_lock_init(&counter->lock);
    >>> + counter->limit = 0;
    >>> + counter->parent = parent;
    >>> +}
    >>> +
    >>> int res_counter_charge_locked(struct res_counter *counter, unsigned long val)
    >>> {
    >>> if (counter->usage + val > counter->limit) {
    >>> @@ -74,6 +82,8 @@ res_counter_member(struct res_counter *c
    >>> return &counter->limit;
    >>> case RES_FAILCNT:
    >>> return &counter->failcnt;
    >>> + case RES_FOR_CHILDREN:
    >>> + return &counter->for_children;
    >>> };
    >>>
    >>> BUG();
    >>> @@ -104,7 +114,9 @@ u64 res_counter_read_u64(struct res_coun
    >>>
    >>> ssize_t res_counter_write(struct res_counter *counter, int member,
    >>> const char __user *userbuf, size_t nbytes, loff_t *pos,
    >>> - int (*write_strategy)(char *st_buf, unsigned long long *val))
    >>> + int (*write_strategy)(char *st_buf, unsigned long long *val),
    >>> + int (*set_strategy)(struct res_counter *res,
    >>> + unsigned long long val, int what))
    >>> {
    >>> int ret;
    >>> char *buf, *end;
    >>> @@ -133,13 +145,101 @@ ssize_t res_counter_write(struct res_cou
    >>> if (*end != '\0')
    >>> goto out_free;
    >>> }
    >>> - spin_lock_irqsave(&counter->lock, flags);
    >>> - val = res_counter_member(counter, member);
    >>> - *val = tmp;
    >>> - spin_unlock_irqrestore(&counter->lock, flags);
    >>> - ret = nbytes;
    >>> + if (set_strategy) {
    >>> + ret = set_strategy(res, tmp, member);
    >>
    >> I'm afraid, I don't understand the set_strategy and it's purpose.
    >>
    > Sorry. I'm now rewritten and removed this.
    >
    >

    OK

    >
    >>> + if (!ret)
    >>> + ret = nbytes;
    >>> + } else {
    >>> + spin_lock_irqsave(&counter->lock, flags);
    >>> + val = res_counter_member(counter, member);
    >>> + *val = tmp;
    >>> + spin_unlock_irqrestore(&counter->lock, flags);
    >>> + ret = nbytes;
    >>> + }
    >>> out_free:
    >>> kfree(buf);
    >>> out:
    >>> return ret;
    >>> }
    >>> +
    >>> +
    >>> +int res_counter_move_resource(struct res_counter *child,
    >>> + unsigned long long val,
    >>> + int (*callback)(struct res_counter *res, unsigned long long val),
    >>> + int retry)
    >>> +{
    >>> + struct res_counter *parent = child->parent;
    >>> + unsigned long flags;
    >>> +
    >>> + BUG_ON(!parent);
    >>> +
    >>> + while (1) {
    >>> + spin_lock_irqsave(&parent->lock, flags);
    >>> + if (parent->usage + val < parent->limit) {
    >>> + parent->for_children += val;
    >>> + parent->usage += val;
    >>> + break;
    >>> + }
    >>> + spin_unlock_irqrestore(&parent->lock, flags);
    >>> +
    >>> + if (!retry || !callback)
    >>> + goto failed;
    >>> + /* -1 means infinite loop */
    >>> + if (retry != -1)
    >>> + --retry;
    >> I don't like the idea of spinning in an infinite loop, I would prefer to fail
    >> things instead of burning CPU cycles.
    >>
    > ok, will remove "-1" case.
    >
    >
    >
    >
    >>> + yield();
    >>> + callback(parent, val);
    >> This code is not very understandable. Why do we yield before callback?
    >>
    >
    > yield() after callback() means that res_counter's state will be
    > far different from the state after callback.
    > So, we have to yield before call back and check res_coutner sooner.
    >

    But does yield() get us any guarantees of seeing the state change?

    --
    Warm Regards,
    Balbir Singh
    Linux Technology Center
    IBM, ISTL


    \
     
     \ /
      Last update: 2008-06-09 12:41    [W:0.070 / U:31.112 seconds]
    ©2003-2016 Jasper Spaans. hosted at Digital OceanAdvertise on this site