Re: prerequisites for our atf suite

[Julio Merino <jmmv%NetBSD.org@localhost>]

On 11/10/11 3:22 PM, Paul Goyette wrote:
> On Thu, 10 Nov 2011, Greg Troxel wrote:
>
> > summary: I'm unclear on what state a machine should be in prior to
> > running ATF.
> >
> > I have a -current tree with a not-yet-committed change, and I'd like to
> > run the test suite. So I built a full release to ISO, and in virtualbox
> > used that iso to do an installation.
> >
> > I booted single user, remounted / rw, and did, more or less
> >
> > cd /usr/tests
> > atf-run . | atf-report
> >
> > and found that lots of tests are failing (with signal 6).
>
> I saw that recently on amd64 - it "went away" on its own.

Those were caused by a bogus change to the atf code that triggered an 
internal assertion.  Effectively, it was causing *all* C-based tests to 
fail.  It was fixed.

> > Here, work means "test results match the results shown on the
> > autobuild/tests page". So I would expect 2 failed tests, or
> > sometimes 1.
>
> I think it needs to be in multi user, but does not require any network
> interfaces (except maybe lo0). You will need comp, base, etc, text, and
> test sets for most things. One or two tests actually depend on the games
> set!

We should make those tests with appropriate "require.progs" so that they 
are marked as skipped instead of failed when the games set is not present.