Re: How to integrate atf with autotools

["Ivan \"Rambius\" Ivanov" <rambiusparkisanius%gmail.com@localhost>]

Hello Julio,

Thank you for your response.

On Sun, Jul 11, 2010 at 11:11 AM, Julio Merino <jmmv%netbsd.org@localhost> 
wrote:
> On Sun, Jul 11, 2010 at 5:48 AM, Ivan "Rambius" Ivanov
> <rambiusparkisanius%gmail.com@localhost> wrote:
>> Hello,
>>
>> I started a project using autotools as its build system and ATF as a
>> testing framework. I am able to build the ATF-based test programs with
>> autoconf/automake/configure, etc. I also created an Atffile and when I
>> run manually "atf-run | atf-report" the tests are executed. Now I want
>> to automate the execution of the atf tests using the default "make
>> check" command provided by autotools.
>>
>> The most obvious solution was to use TESTS_ENVIRONMENT and TESTS
>> automake variables.
> [...]
>
> Hello Ivan,
>
> As you well say, you should not be running atf-run for every test case
> separately.  If you do so, you lose the unified report created by
> atf-run!  You must not also add atf-run to TESTS_ENVIRONMENT; doing so
> doesn't fit the variable description (as that variable is exclusively
> for environment variables) so things may break.
Yes, you are right. However, I had to start from somewhere and I saw
some examples in automake manual with perl command in
TESTS_ENVIRONMENT and I tried with atf-run.

> At the moment, integrating atf with autotools is a bit complicated.
> There is ticket #23 to remind me of this issue, but I haven't given it
> any thought so far.  As far as I know, automake can't be extended by
> third-party packages... so it won't be really possible to provide a
> clean way for integration.  There is also ticket #13 to come up with a
> way of resolving the return code issue of "atf-run | atf-report".
One of the issues of capturing the error code of the first command in
a pipe is portability: bash has ${PIPESTATUS} special array to store
the exit codes of commands in a pipe; zsh use $pipestatus array. I may
be wrong but I think that this code is most portable:

exec 3>&1
atf_run_stat=`((atf-run $1; echo $? >&4) | atf-report 1>&3) 4>&1`
exit $atf_run_stat

If you also think so I can extend it as a wrapper of atf-run and
atf-report and submit it to ticket #13.

> I suggest you to take a look at the atf Makefile.am files to see how
> the whole thing works.  You will need to create rules to install the
> test programs as well as the Atffiles.
For the time being I am not looking for installing the test programs
since my project is in its beginning. I am interested in building and
running the tests when I call 'make check'. The ATF's Makefile.am was
indeed of help to me. I added check-local and check-atf targets to my
Makefile.am:

check-local: check-atf
.PHONY: check-atf
check-atf:
        logfile=$$(pwd)/atf-run.log ; \
        atf-run > $${logfile}; \
        res=$${?}; \
        cat $${logfile} | atf-report; \
        rm $${logfile}; \
        test $${res} -eq 0


> (OK, you don't *need* to
> install any tests, but one of the features of atf is that it allows to
> do so and for good reasons! Take a look at [1].)
I like the approach of enabling the users to run the unit tests
themselves. When I installed atf it installed its tests under
$prefix/tests/atf. If I go with installing the tests of my application
should I installed them under $prefix/tests/<appname>?

Regards
Rambius

-- 
Tangra Mega Rock: http://www.radiotangra.com