Re: CVS commit: src

[David Holland <dholland-sourcechanges%netbsd.org@localhost>]

On Mon, Jul 05, 2010 at 08:44:10PM +0100, Julio Merino wrote:
 > > The way some of the tests are organized suggests that the intended
 > > model is one test program per test victim (or per test victim and
 > > substantially different testing harness) and then one test case per
 > > bug affecting that victim.
 > > [...]
 >
 > For unit tests: you have a test program for every source module.  Such
 > test program contains one unit test per function or one unit test per
 > function/subcases (e.g. error paths, an input array with elements, an
 > input array without elements, etc.)
 > 
 > For integration/system/whatever tests: you have a test program per
 > "semantic" unit (I guess that's what you call victims ;-) and then
 > test cases for every particular scenario you want to represent.

That's not quite what I meant by "victim"; I meant "thing being
tested", whether that's a library routine or a program or whatever.
That is, you might have one test script for sort(1) and another
compiled test program for, say, snprintf(3).

If there's a lot of mechanism for making the test work, then you want
to share it among all test instances that do the same thing.
Conversely, however, it's silly to ram a bunch of unrelated code
together into a single test program just because it all happens to be
intended to test the same thing. So if you have multiple substantially
different test harnesses, as might be the case for e.g. testing a
filesystem in puffs and also in a native kernel running on qemu, that
seems a good reason to have multiple test programs.

However, my experience with test suites for a lot of projects (some of
them highly nontrivial) is that it's always been sufficient for the
test apparatus to run some collection of test programs, possibly on
multiple inputs each, and compare the output to a corresponding known
good version. This can easily be driven with make and doing so has
always been perfectly satisfactory for me, so I don't claim to
understand the fine points of the greater structure atf provides.

-- 
David A. Holland
dholland%netbsd.org@localhost