lkml.org 
[lkml]   [2004]   [Aug]   [21]   [last100]   RSS Feed
Views: [wrap][no wrap]   [headers]  [forward] 
 
Messages in this thread
/
Date
From
SubjectRe: 2.6.8.1-mm3
--William Lee Irwin III <wli@holomorphy.com> wrote (on Friday, August 20, 2004 17:03:43 -0700):

> At some point in the past, I wrote:
>>> Parallel compilation is an extremely poor benchmark in general, as the
>>> workload is incapable of being effectively scaled to system sizes, the
>>> linking phase is inherently unparallelizable and the compilation phase
>>> too parallelizable to actually stress anything. There is also precisely
>>> zero relevance the benchmark has to anything real users would do.

The compilation phase seems to stress a whole bunch of stuff, primarily
the VM and scheduler, and a little on the VFS. Saying that things are
"too parallizable" to stress things is odd - that seems to cause more
stress, not less.

> Kernel hacking is not an end in itself, regardless of the fact there
> are some, such as myself, who use computers for no other purpose. A
> real user generally has some purpose to their activity beyond working
> on the software or hardware they are "using". e.g. various real users
> use their systems for entertainment: playing games, music, and movies.
> Others may use their systems to make money somehow, e.g. archiving
> information about customers so they can look up what they've bought
> and paid for or have yet to pay for.
>
> Regardless of the social issue, the rather serious technical deficits
> of compilation of any software as a benchmark are showstopping issues.
> Frankly, even the issues I've dredged up are nowhere near comprehensive.
> There are further issues such as that stable (i.e. not varying across
> the benchmarks being done on various systems at various times) versions
> of the software being compiled and the toolchain being used to compile
> it are lacking as components of any "kernel compile benchmarking suite"
> and worse still the variance in target architecture of the toolchain
> also defeats any attempt at meaningful benchmarking.

The benchmark is perfectly stable on one machine as long as the user
has some vague grip on the enviroment, and isn't running debian unstable
and auto-updating every night. Benchmarking across arches has always been
done with a target of i386 when Anton and I have done it, and no, it
doesn't serve much purpose apart from a fun pissing contest. However,
benchmarking different kernels with the same userspace setup on one
machine is perfectly valid, and very useful.

> If you're truly concerned about compilation speed, userspace is going
> to be the most productive area to work on anyway, as the vast majority
> of time during compilation is spent in userspace. AIUI the userspace
> algorithms in gcc are not particularly cognizant of cache locality and
> in various instances have suboptimal time and space behavior, so it's
> not as if there isn't work to be done there. Improving the compactness
> and cache locality of data structures is important in userspace also,
> and most (perhaps all) userspace programs are grossly ignorant of this.
> FWIW, there are notable kernel hackers known to use very downrev gcc
> versions due to regressions in compilation speed in subsequent versions,
> so there are already large known differences in compilation speed that
> can be obtained just by choosing a different compiler version.

Altering userspace is of course going to change the result - but has
absolutely no relevance whatsoever to whether this is a useful benchmark
or not.

The point is not to compile the kernel - the point is to get a vaguely
realistic simulation of something a user might do to stress the kernel
in interesting ways. At which I think kernbench does reasonably well.
No, it's not perfect, but it's simple, and it's useful - it's led to
many useful improvements already, and no doubt will continue to do so.

All benchmarks are crap. Some are just less crap than others.

M.

compile the kernel, but
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

\
 
 \ /
  Last update: 2005-03-22 14:05    [W:0.048 / U:2.440 seconds]
©2003-2020 Jasper Spaans|hosted at Digital Ocean and TransIP|Read the blog|Advertise on this site