Messages in this thread | | | Date | Tue, 06 Jun 2006 18:55:30 -0400 | From | Shailabh Nagar <> | Subject | Re: Merge of per task delay accounting (was Re: 2.6.18 -mm merge plans) |
| |
Jay Lan wrote: > Shailabh Nagar wrote: > >> Balbir Singh wrote: >> >>> Andrew Morton wrote: >>> >>> >>>> per-task-delay-accounting-setup.patch >>>> per-task-delay-accounting-setup-fix-1.patch >>>> per-task-delay-accounting-setup-fix-2.patch >>>> per-task-delay-accounting-sync-block-i-o-and-swapin-delay-collection.patch >>>> >>>> >>>> per-task-delay-accounting-sync-block-i-o-and-swapin-delay-collection-fix-1.patch >>>> >>>> >>>> per-task-delay-accounting-cpu-delay-collection-via-schedstats.patch >>>> per-task-delay-accounting-cpu-delay-collection-via-schedstats-fix-1.patch >>>> >>>> per-task-delay-accounting-utilities-for-genetlink-usage.patch >>>> per-task-delay-accounting-taskstats-interface.patch >>>> per-task-delay-accounting-taskstats-interface-fix-1.patch >>>> per-task-delay-accounting-taskstats-interface-fix-2.patch >>>> per-task-delay-accounting-delay-accounting-usage-of-taskstats-interface.patch >>>> >>>> >>>> per-task-delay-accounting-delay-accounting-usage-of-taskstats-interface-use-portable-cputime-api-in-__delayacct_add_tsk.patch >>>> >>>> >>>> per-task-delay-accounting-documentation.patch >>>> per-task-delay-accounting-proc-export-of-aggregated-block-i-o-delays.patch >>>> >>>> >>>> per-task-delay-accounting-proc-export-of-aggregated-block-i-o-delays-warning-fix.patch >>>> >>>> >>>> >>>> I just don't know. There are a number of groups who pop up with >>>> various >>>> enhanced accounting requirements and patches (all quite different) >>>> but I >>>> haven't heard a lot of enthusiasm from any of them over this work, >>>> which >>>> attempts to provide an extensible framework for accumulation and >>>> querying >>>> of per-task metrics. >>>> >>>> But then again, we cannot just sit there and wait for everyone to be >>>> 100% >>>> happy. So I'm 51% inclined to push this along. >>>> >>>> Anyone else who has an interest in this sort of thing needs to be aware >>>> that there will be an expectation that any future statistics >>>> submissions >>>> should use these interfaces. So the time to pay attention is right >>>> now. >>>> >>> >>> Hi, Andrew, >>> >>> Here is a brief summary of the status of the response we have >>> received from >>> the stakeholders (some of it has been duplicated in previous postings) >>> >>> Project Response >>> >>> 1. CSA accounting/PAGG/JOB: Has agreed to use >>> taskstats >>> Jay Lan <jlan@engr.sgi.com> interface >>> >>> 2. per-process IO statistics: None >>> Levent Serinol <lserinol@gmail.com> Needs are subset of CSA >>> >>> 3. per-cpu time statistics: None (email bounced) >>> Erich Focht <efocht@ess.nec.de> Needs can be met by >>> taskstats >>> Statistics not yet >>> submitted >>> >>> 4. Microstate accounting: None >>> Peter Chubb <peterc@gelato.unsw.edu.au> overlap with delay >>> accounting >>> prefers /proc due to >>> convenience >>> taskstats can meet the >>> needs >>> >>> >>> 5. ELSA: Guillaume Thouvenin None >>> <guillaume.thouvenin@bull.net> ELSA is not a direct user >>> of new kernel statistics >>> Consumer of CSA/BSD >>> accounting >>> statistics >>> >>> 6. pnotify: Jes Sorensen <jes@sgi.com> None >>> (taken over pnotify from Erik Jacobson) Informed over private >>> email >>> that pnotify replacement is >>> being worked on. pnotify >>> or its replacement will >>> not be concerned with >>> exporting data to user >>> space >>> or collecting any >>> statistics. >>> >>> >>> 7. Scalable statistics counters with /proc Not working on it >>> reporting: anymore >>> Ravikiran G Thirumalai, >>> Dipankar Sarma <dipankar@in.ibm.com> >>> >>> Studying the responses from all stake holders, Jay Lan's was the most >>> encouraging. Peter Chubb prefers the /proc interface due to the text >>> interface >>> and ease of parsing. (in our opinion, taskstats can meet the needs >>> easily >>> and the getdelays utility can provide the same ease for parsing). >>> The others did not respond. >>> Some performance numbers of taskstats were posted at >>> http://lkml.org/lkml/2006/3/23/141. The result highlights are included >>> below >>> >>> Results highlights >>> >>> - Configuring delay accounting adds < 0.5% >>> overhead in most cases and even reduces overhead >>> in some cases >>> >>> - Enabling delay accounting has similar results >>> with a maximum overhead of 1.2% for hackbench, >>> most other overheads < 1% and reduction in >>> overhead in some cases >>> >>> These statistics are _per task_ and can be extended easily by anyone >>> who wishes to obtain per task data. An example of per task improved >>> scheduler statistics was mentioned in http://lkml.org/lkml/2006/6/1/381 >>> (I am not sure if the email refers to our per-task statistics). If not, >>> the new statistics could easily use the taskstats interface. >>> >>> These statistics can be used by software product stacks to monitor >>> usage information about the various tasks they create and control. >>> I also informally spoke to a group of students (verbally), who were >>> excited at the possibility of using the per-task statistics to do >>> dynamic deadline based power management. They want to use the delay data >>> (CPU and IO) to predict deadlines for a task and then use these results >>> for dynamically scaling CPU frequency. >>> >>> >>> The ability to monitor the CPU run and delay data and IO delay data is >>> useful. >>> >>> I would request you to consider the inclusion per-task delay accounting >>> into >>> 2.6.18. >>> >> >> >> >> Andrew, >> >> The only other new set of patches to be discussed in this context are the >> statistics-infrastructure patches from Martin Peschke. >> >> That infrastructure cannot meet the needs of delay accounting, CSA >> etc. because >> - it only provides "user pull" model of getting stats whereas "kernel >> push" is >> needed for delay accounting > > > Doesn't taskstats interface provide "user pull" request-reply model > also? Serious accounting needs to push accounting data as soon as > possible.
Yes, I meant to say "kernel push" is also needed for delay accounting. So taskstats provides both pull and push whereas statistics infrastructure, on account of use of fs-based interface, provides only user-pull. > >> - it uses a relatively slow interface unsuitable for high volumes of >> data. Each >> statistic has its own definition, needs to be read separately using >> ASCII, >> reading data continuously means open/read/close each time.....all of >> which is not very conducive to large structures being sent to userspace. > > > Yes, i second the point. It won't be able to catch up the traffic. > >> - its oriented towards sampled data whereas taskstats isn't. >> >> So, we have a good consensus from existing/potential users of >> taskstats and would >> very much appreciate it being included in 2.6.18. > > > Andrew, it has become clear that the community wants to see accounting > data processing being moved to userspace. Thus there is a need for a > common accounting interface to provide minimal works at kernel (via > hooks at fork and exit) and deliver data to userspace. > > The delayacct patchset provides a good framework and example that > i believe CSA/Job can follow and build upon to move most of our work > to userspace and thus cut off dependency of PAGG. We will submit CSA > patch soon based on the taskstats interface. > > Thanks, > - jay > > P.S. Balbir and Shailabh, Chris Sturtivant will continue the CSA work > at SGI. Please also cc Chris <csturtiv@sgi.com> in the future. > Thanks!
Sure.
Thanks, Shailabh > > >> >> --Shailabh >> >> >> >> >
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
| |