bin/43394: adding regression tests is too complicated

[pooka%iki.fi@localhost]

>Number:         43394
>Category:       bin
>Synopsis:       adding regression tests is too complicated
>Confidential:   no
>Severity:       non-critical
>Priority:       high
>Responsible:    bin-bug-people
>State:          open
>Class:          change-request
>Submitter-Id:   net
>Arrival-Date:   Mon May 31 13:40:01 +0000 2010
>Originator:     Antti Kantee
>Release:        
>Organization:
Test'ng
>Environment:
>Description:
Adding a new atf tests requires (off the top of my head):

1) copypasting the test skeleton
2) writing the actual test
3) creating/editing a Makefile
4) creating/editing an Atffile
5) adding the test to several places in set lists

Additionally, if the test creates a new subdirectory in the
test hierarchy:

6) editing another Makefile
7) editing another Atffile
8) editing the mtree base
9) adding more lines to the set lists

Tests should be easy and simple to add.  Almost 50% of the time
in adding a test is spent appeasing various gods of build
mechanics.
>How-To-Repeat:
Attempt to make regression testing easy for yourself and others.
>Fix:
Most likely the easiest solution is to provide a script which
takes care of all the mechanical tasks.  It can take care of
everything apart from step 2.