Re: How to integrate atf with autotools

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Jul 12, 2010 at 6:59 AM, Ivan "Rambius" Ivanov
<rambiusparkisanius%gmail.com@localhost> wrote:
[...]
>> Hello Ivan,
>>
>> As you well say, you should not be running atf-run for every test case
>> separately.  If you do so, you lose the unified report created by
>> atf-run!  You must not also add atf-run to TESTS_ENVIRONMENT; doing so
>> doesn't fit the variable description (as that variable is exclusively
>> for environment variables) so things may break.
> Yes, you are right. However, I had to start from somewhere and I saw
> some examples in automake manual with perl command in
> TESTS_ENVIRONMENT and I tried with atf-run.

I see; interesting.  I blame the variable name then ;-)

>> At the moment, integrating atf with autotools is a bit complicated.
>> There is ticket #23 to remind me of this issue, but I haven't given it
>> any thought so far.  As far as I know, automake can't be extended by
>> third-party packages... so it won't be really possible to provide a
>> clean way for integration.  There is also ticket #13 to come up with a
>> way of resolving the return code issue of "atf-run | atf-report".
> One of the issues of capturing the error code of the first command in
> a pipe is portability: bash has ${PIPESTATUS} special array to store
> the exit codes of commands in a pipe; zsh use $pipestatus array. I may
> be wrong but I think that this code is most portable:
>
> exec 3>&1
> atf_run_stat=`((atf-run $1; echo $? >&4) | atf-report 1>&3) 4>&1`
> exit $atf_run_stat

As I have seen in the anita script, you can also do:

    { atf-run && :>/tmp/test.ok; } | atf-report && test -f /tmp/test.ok

Which is completely portable.

> If you also think so I can extend it as a wrapper of atf-run and
> atf-report and submit it to ticket #13.

Well, the problem with such simple script is: how do you pass options
to atf-run? And to atf-report?  Also, there is a much deeper issue:
the "atf-run | atf-report" suggested pipeline loses *a lot* of
information because the default output of atf-report only prints a
summary; such information is invaluable for debugging, and it must be
captured because some tests may fail only intermittently.  The
pipeline should really be "atf-run | tee somefile.log | atf-report",
but it's obvious I can't ask users to do that themselves.

My current ideas to resolve this are: merge atf-run and atf-report
into a single beast (I'm thinking of a binary called 'atf'; plain and
simple).  Such tool would run the tests as it currently does, but
would store the results of the tests on-disk for future reference (and
with historical data!).  After the execution, the unified tool would
just print the summary of the run as atf-report does now (and report a
correct exit code).  With such approach, you could be able to run the
tests getting a nice progress report but, if something fails, you'd be
able to extract details of the failure.

It would go along the lines of:

$ atf run
... blah, blah, blah, you see the same as atf-report prints now
$ echo $?
1
... oh! something failed!  What tests cases failed?
$ atf query result=failed
foo/bar/baz_test:my_test_case
foo/bar/baz_test:another_test_case
$ atf inspect foo/bar/baz_test:my_test_case
... and you get the failure reason, the stdout, the stderr of the last run
$ atf history foo/bar/baz_test:my_test_case
20100701123500: failed, foo bar
20100630161057: passed
...

or something like this.  You get the idea I hope :-)

>> I suggest you to take a look at the atf Makefile.am files to see how
>> the whole thing works.  You will need to create rules to install the
>> test programs as well as the Atffiles.
> For the time being I am not looking for installing the test programs
> since my project is in its beginning.

Making sure the tests get installed and work correctly as soon as you
add them is just a tiny amount of incremental work.  Doing all this
work as an afterthought once you have many tests will be painful, and
that's guaranteed.  So little suggestion: start adding the necessary
Makefile rules to install them now.

> I am interested in building and
> running the tests when I call 'make check'. The ATF's Makefile.am was
> indeed of help to me. I added check-local and check-atf targets to my
> Makefile.am:
>
> check-local: check-atf
> .PHONY: check-atf
> check-atf:
>        logfile=$$(pwd)/atf-run.log ; \
>        atf-run > $${logfile}; \
>        res=$${?}; \
>        cat $${logfile} | atf-report; \
>        rm $${logfile}; \
>        test $${res} -eq 0

That's fine.  If you end up installing the tests, just duplicate the
check-local as a installcheck-local target that does the same commands
but does 'cd ${testsdir}/yourproject' as the first step.  Having a
installcheck target does not prevent you from adding a check target!

>> (OK, you don't *need* to
>> install any tests, but one of the features of atf is that it allows to
>> do so and for good reasons! Take a look at [1].)
> I like the approach of enabling the users to run the unit tests
> themselves. When I installed atf it installed its tests under
> $prefix/tests/atf. If I go with installing the tests of my application
> should I installed them under $prefix/tests/<appname>?

Yep, that's correct.

-- 
Julio Merino