How to integrate atf with autotools

["Ivan \"Rambius\" Ivanov" <rambiusparkisanius%gmail.com@localhost>]

Hello,

I started a project using autotools as its build system and ATF as a
testing framework. I am able to build the ATF-based test programs with
autoconf/automake/configure, etc. I also created an Atffile and when I
run manually "atf-run | atf-report" the tests are executed. Now I want
to automate the execution of the atf tests using the default "make
check" command provided by autotools.

The most obvious solution was to use TESTS_ENVIRONMENT and TESTS
automake variables.

Attempt 1) I specified the automake variable as

check_PROGRAMS = test_seed_prng

test_seed_prng_SOURCES = seed_prng.c test_seed_prng.c

TESTS_ENVIRONMENT = atf-run
TESTS = test_seed_prng

When I run 'make check' the test is built and executed correctly and
all successes and failures are reported correctly. Also 'make check'
fails if there are test failures. However the output was too long and
hard to read even for one test program. Sending the results to
atf-report would be better, but I do not know how to code this in
automake.

Attempt 2) In order to pipe the results to atf-report I created the
following simple script called atf-run-and-report:

#!/bin/sh
atf-run $1 | atf-report
exit $?

Then I changed the variables in Makefile.am to

TESTS_ENVIRONMENT = ./atf-run-and-report
TESTS = test_seed_prng

The output of the test execution is now less cluttered, but in case of
test failures 'make check' does not fail because the exit code of the
piped commands is the exit code of the last command. In case of ATF
this can be a nuisance:

$ atf-run
$ echo $?
1

$ atf-run | atf-report
$ echo $?
0


Attempt 3) To fix the exit code problem, I changed atf-run-and-report script to

#!/bin/sh
exec 3>&1
atf_run_stat=`((atf-run $1; echo $? >&4) | atf-report 1>&3) 4>&1`
exit $atf_run_stat

The script returns the correct error code and in case of failure 'make
check' also fails. I am not sure how good the script is and since ATF
aims at portability how portable it is. But in any case it is better
then return success code when there are real failures.

Attempt 4) Now I added some more test programs and specified them in
Makefile.am:

check_PROGRAMS = test_seed_prng test_ok test_fail

test_seed_prng_SOURCES = seed_prng.c test_seed_prng.c
test_ok_SOURCES = test_ok.c
test_fail_SOURCES = test_fail.c

TESTS_ENVIRONMENT = ./atf-run-and-report
TESTS = test_seed_prng test_ok test_fail

Now 'make check' executes the test programs, reports errors if any and
fails if there are failures.


This approach, however, has disadvantages. First, the test programs
are executed one by one separately and atf-run is called for each of
them. In case of many test programs this will have performance
drawbacks. Second, it requires double maintenance - the test programs
should be given in two place - in Atffile and in Makefile.am. Third,
the test results are given for each program separately - they are not
aggregated as when run using Atffile.

I am a newbie to autotools and may be I am not doing it in the right
way. I would be very grateful if you provide any hints on how to
integrate atf and autotools.

Regards
Rambius

-- 
Tangra Mega Rock: http://www.radiotangra.com