lkml.org 
[lkml]   [2016]   [Jun]   [22]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: [Cocci] [PATCH v3 3/8] coccicheck: enable parmap support
On Wed, Jun 22, 2016 at 07:25:11AM +0200, Julia Lawall wrote:
>
>
> On Wed, 22 Jun 2016, Luis R. Rodriguez wrote:
>
> > On Tue, Jun 21, 2016 at 11:44:09PM +0200, Julia Lawall wrote:
> > >
> > >
> > > On Tue, 21 Jun 2016, Luis R. Rodriguez wrote:
> > >
> > > > On Tue, Jun 21, 2016 at 11:32:11PM +0200, Julia Lawall wrote:
> > > > >
> > > > >
> > > > > On Tue, 21 Jun 2016, Luis R. Rodriguez wrote:
> > > > >
> > > > > > On Tue, Jun 21, 2016 at 11:00:53PM +0200, Nicolas Palix (LIG) wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > Le 21/06/16 à 22:43, Julia Lawall a écrit :
> > > > > > > >
> > > > > > > >
> > > > > > > >On Tue, 21 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > >
> > > > > > > >>On Tue, Jun 21, 2016 at 10:17:38PM +0200, Julia Lawall wrote:
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>On Tue, 21 Jun 2016, Luis R. Rodriguez wrote:
> > > > > > > >>>
> > > > > > > >>>>Coccinelle has had parmap support since 1.0.2, this means
> > > > > > > >>>>it supports --jobs, enabling built-in multithreaded functionality,
> > > > > > > >>>>instead of needing one to script it out. Just look for --jobs
> > > > > > > >>>>in the help output to determine if this is supported.
> > > > > > > >>>>
> > > > > > > >>>>Also enable the load balancing to be dynamic, so that if a
> > > > > > > >>>>thread finishes early we keep feeding it.
> > > > > > > >>>>
> > > > > > > >>>>Note: now that we have all things handled for us, redirect stderr to
> > > > > > > >>>>stdout as well to capture any possible errors or warnings issued by
> > > > > > > >>>>coccinelle.
> > > > > > > >>>>
> > > > > > > >>>>If --jobs is not supported we fallback to the old mechanism.
> > > > > > > >>>>This also now accepts DEBUG_FILE= to specify where you want
> > > > > > > >>>>stderr to be redirected to, by default we redirect stderr to
> > > > > > > >>>>/dev/null.
> > > > > > > >>>
> > > > > > > >>>Why do you want to do something different for standard error in the parmap
> > > > > > > >>>and nonparmap case?
> > > > > > > >>
> > > > > > > >>We should just deprecate non-parmap later.
> > > > > > > >
> > > > > > > >that's not really getting at the point. I like the DEBUG_FILE= solution.
> > > > > > > >I don't like merging stderr and stdout. So you've put what to my mind is
> > > > > > > >the good solution only in the deprecated case (to my understanding of
> > > > > > > >the commit message).
> > > > > > >
> > > > > > > I agree. You're not just "enabling parmap support". You're
> > > > > > > also changing how messages to stderr are handled.
> > > > > > > Maybe add the DEBUG_FILE mechanism in a separate patch for both
> > > > > > > modes (parmap and non-parmap).
> > > > > >
> > > > > > I'd prefer to just rip out non-parmap support and bump coccinelle
> > > > > > requiremetns to at least 1.0.3, thoughts?
> > > > >
> > > > > There are already too many changes in this patch series.
> > > > >
> > > > > Also, I don't know what the 0-day people would find convenient.
> > > >
> > > > I'd really prefer to not deal with supporting DEBUG_FILE for non-parmap
> > > > case due to the way parallelism is supported there, it uses wait(1) to
> > > > wait on the shell, and for spawning this nasty thing:
> > > >
> > > > eval "$@ --max $NPROC --index $i &"
> > > >
> > > > Specially since we are likely to be able to deprecate this sooner
> > > > rather than later I see little point in adding DEBUG_FILE into this
> > > > mess.
> > >
> > > Sorry, I didn't realize there was parallelism without parmap.
> >
> > Yea :( so is the change OK as-is then, only I need to update the commit log?
> >
> > > My thought
> > > was that if someone is running Coccinelle on only one core, then why force
> > > them to use parmap.
> >
> > Oh but that's different feedback. Sure, but why should that be an issue ?
> > It would seem that coccinelle would just do the right thing with -j 1 used.
> >
> > > Coccinelle could of course be updated to not use
> > > parmap when the number of cores is 1.
> >
> > :) Single CPU systems are probably odd bests these days, either way I can
> > update the script to avoid parmap if number of cpus is 1 since I'm respinning.
>
> Some semantic patches have to be run single core, eg due to the use of
> finalize. Perhaps there would be some reason to run them single core, if
> one had the same nmber of semantic patches as cores. This was more
> relevant before dynamic load balancing though. Single core is also better
> when using an option that takes a lot of include files and when using
> --include-headers-for-types. Then one has maximal sharing of include file
> information across the treatment of the different C files. In contrast,
> chunksize 1 is worst. In that case, there is no effective caching of
> parsed header files, because Coccinelle has no shared memory.

I've disabled parmap for 1 CPU now.

> Actually, it would be probably good to raise the default chunksize a bit
> for the latter reason. It would depend on which files get assigned to
> which chunks though how much benefit it might have.

What value do you have in mind? Or should we leave this as a separate future
change?

Luis

\
 
 \ /
  Last update: 2016-06-22 18:01    [W:0.343 / U:0.160 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site