Messages in this thread | | | Date | Tue, 08 Aug 2000 18:36:00 -0500 | From | Aaron Laffin <> | Subject | Re: [Announce] Linux Test Project |
| |
Larry, thank you for your post. This is exactly the post I was looking for to seed conversation on this subject. Thank you for your testimony into the value of regression testing. Regression testing isn't a panacea, but it can have great value if leveraged correctly.
In regard to your framework, this is very similar to some ideas we had for our project. We (SGI) have a regression testing framework in use internally that is very similar in basis to your framework. The major difference is that it has been in use and development for years now. Its quite grandiose and complete -- I think we might open source it if LTP came to need it. For now, your framework is more along the line of what we had in mind for LTP.
The exit status method is probably the simplest way of reporting test results. It makes it relatively easy for anyone to write tests (take note you lkml lurkers). Basically, writing a simple test goes something like this: Setup - get the proc environment ready to perform the test operation Test - perform the test operation Analysis - determine the success/failure of the operation Cleanup - get ready to go away, cleanup scratch space, make sure system is in a clean state Exit - report results via exit status 0-PASS,!0-FAIL
Larry, I may hit you up for your test harness. It may be a great place for us to start. We already have 90+ tests, and managing them is already a problem.
Again, anyone interested in helping us with our project is encouraged to participate: http://oss.sgi.com/projects/ltp/
--aaron
Larry McVoy wrote: > > I'm a huge fan of this idea. We have a very simple, but effective, test > harness in BitKeeper. You are welcome to it if you want it. All we do is > have directory full of "t.files" which are shell scripts. There is a > setup program called "doit" which clears out /tmp/.regression-$USER, > sets up the environment, goes to that directory, and then runs the > script. If the script exits 0, it does the next one, else it stops. > > We've got 12,000 lines of shell scripts we use to test BitKeeper and it > has been priceless. In theory, we don't fix a bug without adding a > regression, in practice that is true about 80% of the time. We're quite > happy that it is that high. The reason it is that high is that once the > harness is there, you can create a new test like so: > > cat > t/t.small > echo $N Try wrapping and unwrapping some data .......................$NL > ls / > data > bk uuwrap < data > data.uu > bk unuuwrap < data.uu | cmp -s - data > if [ $? != 0 ]; then echo Data is different when unwrapped; exit 1; fi > echo OK > ^D > > I strongly urge you to do the work to create a similar simple harness. > If it is trivial for people to create tests, they _might_ create them. > If it is anything other than trivial, it's just another good idea that > won't happen. > > If I get to Linux World I'll look you up, otherwise you can ping me > for the test harness. > > On Tue, Aug 08, 2000 at 03:06:28PM -0500, Nathan Straz wrote: > > SGI would like to announce the Linux Test Project. The goal of this > > project is to create a formalized test system for the Linux kernel. > > > > We have released a set of 96 tests on the project's website > > (http://oss.sgi.com/projects/ltp/). These tests exercise file systems > > and system calls and can be used for stress testing or sanity tests. > > > > We would like to discuss the following topics with the community. > > > > - The testing philosophy that is most important to the kernel > > developers. What approach best fits the development process? > > Regression? Functional? Stress? Performance? > > > > - What is needed immediately? Building a test suite for the kernel is > > going to take time. What tests or tools are most important? > > > > - We need to plan a development road map that works with the Linux > > kernel development road map. > > > > We are hoping to hold an unscheduled BOF at LinuxWorld on Wednesday. > > Aaron Laffin and Richard Logan will be there to discuss these issues. > > If you are interested in testing and are attending LinuxWorld, please > > keep an eye open for our BOF. > > > > Linux Test Project http://oss.sgi.com/projects/ltp/ > > LTP Mailing list ltp@oss.sgi.com > > > > - > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > > the body of a message to majordomo@vger.rutgers.edu > > Please read the FAQ at http://www.tux.org/lkml/ > > -- > --- > Larry McVoy lm@bitmover.com http://www.bitmover.com/lm > > - > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.rutgers.edu > Please read the FAQ at http://www.tux.org/lkml/
-- Aaron Laffin Silicon Graphics, Inc. OS Test Development Email: alaffin@sgi.com Voice: 651-683-5756 USA/MN/CRP/F5233/SSBU
- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.rutgers.edu Please read the FAQ at http://www.tux.org/lkml/
| |