Messages in this thread | | | Date | Sun, 04 Jul 1999 23:09:25 +0000 | From | Steve Underwood <> | Subject | Re: Are there kernel testing suites out there? We need them. |
| |
Alan Cox wrote:
> > That works, but adding a mechanized regression test would > > be painless, and might give us earlier warnings if and when > > a few bad things (like ext2 fs corruption) creep back into > > Automatic regression tests very rarely help. Most of the bugs that > get into a shipping kernel now are ones I can't reproduce even given a > description let alone find randomly > > > corporations love. Red Hat, for instance, might well feel > > inclined to set up a mechanized regression test to give > > it a little added certainty that it's not about to ship a lemon. > > Guess which turns up lemons best, the automated testing or the beta program.
The usual experience (not necessarily with Linux) is these two types of testing produce sets of bug reports which only partially overlap. Each seems to be better at flushing out certain types of problem. Only an idiot organisation would place full reliance on programmed regression testing (like a lot of idiot organisations actually do), but betas tend to miss problems in boring bits the beta testers don't exercise well. I guess you might say programmed tests give broader coverage, but betas give greater depth. The fact that regression tests are only partially effectice shows the limited value of test coverage analysis, which some people so love.
One thing programmed regression tests do well is to reduce reinventing the wheel. A lot of test results are open to discussion, to decide what the "correct" outcome is. If the final answer gets embedded in the test plan it tends not to get wastefully re-discussed so often.
The key issue is not whether regression tests are useful , which they are, but whether they are cost effective, which is questionable in the Linux case. Try skipping them on a large distributed system, like a cellular switch, and see what the effect is.
Steve
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |