On Tue, Nov 06, 2007 at 10:58:33AM +0100, Julio M. Merino Vidal wrote:
 > >>As a matter of fact, there are a couple of sh(1) regression tests
 > >>that fail at the moment in current, and I bet they remain unfixed
 > >>because no one actually executed the test suite to discover them
 > >>(which is understandable because it's not trivial to do).
 > >
 > >Enh, what's so hard about "cd /usr/src/regress/bin/sh && make  
 > >regress"?
 > 
 > Doing that is not hard.  But how do you collect the results the run?   
 > [...]

I'm not saying what you've done is bad or not needed, just quibbling
with the point you chose to raise. Running the tests for sh is not
hard. :-)

But improving the mechanism alone won't necessarily help the problem.
It *isn't* hard to run the regression tests for sh, so why don't more
people do so? I think the primary reason is that we tend to forget
they're there. So it's important to increase the visibility. Doing a
nightly test run and sending the results to current-users would be a
good start.

(In fact, one can start right away with something as simple as

	cd src/regress
	make regress >& LOG
	rcsdiff -u LOG | mail -s 'Nightly regress run' current-users
	ci -l -m `date +%Y%m%d` LOG

until you're ready to commit the new stuff. There are problems doing
it this way of course, but it works surprisingly well in practice.)

...

I did just discover a problem: src/regress/Makefile is missing any
reference to src/regress/bin. Maybe that's part of why the tests for
bin get no attention.

-- 
   - David A. Holland / dholland+netbsd@eecs.harvard.edu