CVS commit: othersrc/external/bsd/human2atf

["Alistair G. Crooks" <agc%netbsd.org@localhost>]

Module Name:    othersrc
Committed By:   agc
Date:           Sun May 29 20:47:56 UTC 2016

Added Files:
        othersrc/external/bsd/human2atf: Makefile README Testspec.template
            human2atf.1 human2atf.sh

Log Message:
Add human2atf, a means of generating ATF test scripts through a short test
specification file.

        human2atf is a frontend for the ATF test harness. The idea is to provide
        an easy/simple way to sepcify tests, and to provide the correct
        environment for running these test cases.

        In order to accomplish this, a specfile, (by default Testspec) is
        given to human2atf, which is a shell script which produces the test
        Makefile, and t_*.sh shell script used by atf.  The Testspec file is
        itself a shell file fragment.  It specifies the names of the files
        which are to be used in testing, which are typically input files to
        the command being tested, and expected output files.  Care should be
        taken in naming test sets so that a name that is a shell variable is
        used.

        It is advised that individual test cases are grouped into
        functionally-related test sets.  This is not enforced, but is good
        practice since the tests are used together as a regression test suite,
        and so changes to utilities may show up in certain parts of the
        utility being tested, but not in others.  Grouping indivual
        functionally-related test cases into a test set for that functionality
        is thus a good idea.

        Some hand-rolled tests in the tree use shell "here" documents, which
        can interfere with control flow unless used carefully.  To avoid this,
        human2atf encodes its input and expected output files using base64
        uuencoding, so that the contents of such files do not affect the way
        that the human2atf shell script itself works.

        For a test set, all that need to be specified are:
                + the testset name
                + any input and expected output files
                + the test cases themselves

        To use a worked example, to enable atf tests for a (possibly fictional)
        utility called b2e, a Testspec file was drawn up:

                TESTNAME=b2e

                TESTSET_1_NAME=basics
                TESTSET_1_FILES='
                1.in
                1.expected
                2.in
                2.expected
                '
                TESTSET_1_CASE_1="-s eq:0 -o file:1.expected -e empty b2e < 1.in"
                TESTSET_1_CASE_2="-s eq:0 -o file:2.expected -e empty b2e < 2.in"

                TESTSET_2_NAME=extended
                TESTSET_2_FILES='
                3.in
                3.expected
                4.in
                4.expected
                5.in
                5.expected
                '
                TESTSET_2_CASE_1="-s eq:0 -o file:3.expected -e empty b2e < 3.in"
                TESTSET_2_CASE_2="-s eq:0 -o file:4.expected -e empty b2e < 4.in"
                DISABLE_TESTSET_2_CASE_3="-s eq:0 -o file:5.expected -e empty b2e < 5.in"

        This specifies 2 test sets, the first set containing 2 test cases, and
        the second set containing 3 test cases.  The first test set is for
        basic test cases, the second set is for extended tests (see the
        TESTSET_n_NAME definition).  Test cases must be numbered in increasing
        order from 1, and may not miss numbers.  (When the human2atf test
        script finds that a test case is not defined, it stops processing).

        The same numbering restrictions are in place for test sets.

        To provide the tests a useful environment in which to function, input
        and expected output files are specified.  These are given in the
        TESTSET_n_FILES definition.

        Individual test cases are specified using the TESTSET_n_CASE_m style of
        definition.

        Occasionally, we will want to add a test and its supporting files, but
        leave it disabled (until appropriate functionality is written or
        debugged fully, for example) - for that case, the
        DISABLE_TESTSET_n_CASE_m definition is used.

        I opted to leave the atf-style of command invocation in there as the
        expected output file is specified in the test case itself but not as
        part of the command, so it seemed simpler to do it that way.

        Test execution (the directory and Makefile in tests/usr.bin had
        already been set up):

                [12:34:14] agc@netbsd-002 ~/local/human2atf-20160528 [3068] > ./human2atf -f Testspec2
                Generating Makefile for b2e tests as tests-Makefile
                Generating test harness t_b2e.sh for b2e
                [12:34:19] agc@netbsd-002 ~/local/human2atf-20160528 [3069] > sudo cp t_b2e.sh =1
                [12:34:30] agc@netbsd-002 ~/local/human2atf-20160528 [3070] > pushd +1
                /usr/tests/usr.bin/b2e ~/local/human2atf-20160528
                [12:34:33] agc@netbsd-002 ...tests/usr.bin/b2e [3071] > l
                total 20
                drwxr-xr-x   2 root  wheel   512 May 28 23:23 .
                drwxr-xr-x  35 root  wheel  1024 May 28 19:17 ..
                -rw-r--r--   1 root  wheel   136 May 28 19:18 Atffile
                -rw-r--r--   1 root  wheel   112 May 28 19:18 Makefile
                -rw-r--r--   1 root  wheel  2184 May 28 23:23 atf-run.log
                -rwxr-xr-x   1 root  wheel  3368 May 28 23:23 t_b2e
                -rw-r--r--   1 root  wheel  3751 May 29 12:34 t_b2e.sh
                [12:34:34] agc@netbsd-002 ...tests/usr.bin/b2e [3072] > sudo make
                #     build  b2e/t_b2e
                echo '#! /usr/bin/atf-sh' >t_b2e.tmp
                cat t_b2e.sh >>t_b2e.tmp
                chmod +x t_b2e.tmp
                mv t_b2e.tmp t_b2e
                [12:34:39] agc@netbsd-002 ...tests/usr.bin/b2e [3073] > sudo make test
                *** WARNING: make test is experimental
                ***
                *** Using this test does not preclude you from running the tests
                *** installed in /usr/tests.  This test run may raise false
                *** positives and/or false negatives.

                Tests root: /usr/tests/usr.bin/b2e

                t_b2e (1/1): 2 test cases
                    b2e_testset_1_basics: [0.031023s] Passed.
                    b2e_testset_2_extended: [0.026240s] Passed.
                [0.060611s]

                Summary for 1 test programs:
                    2 passed test cases.
                    0 failed test cases.
                    0 expected failed test cases.
                    0 skipped test cases.

                *** The verbatim output of atf-run has been saved to /usr/tests/usr.bin/b2e/atf-run.log
                *** Once again, note that make test is unsupported.
                [12:34:43] agc@netbsd-002 ...tests/usr.bin/b2e [3074] >

        Alistair Crooks
        Sun May 29 12:41:19 PDT 2016


To generate a diff of this commit:
cvs rdiff -u -r0 -r1.1 othersrc/external/bsd/human2atf/Makefile \
    othersrc/external/bsd/human2atf/README \
    othersrc/external/bsd/human2atf/Testspec.template \
    othersrc/external/bsd/human2atf/human2atf.1 \
    othersrc/external/bsd/human2atf/human2atf.sh

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.