Re: How to integrate atf with autotools

[Julio Merino <jmmv%NetBSD.org@localhost>]

On Mon, Jul 12, 2010 at 8:53 PM, Ivan "Rambius" Ivanov
<rambiusparkisanius%gmail.com@localhost> wrote:
> Hello Julio,
>
> On Mon, Jul 12, 2010 at 11:43 AM, Julio Merino <jmmv%netbsd.org@localhost> 
> wrote:
>>> Yes, you are right. However, I had to start from somewhere and I saw
>>> some examples in automake manual with perl command in
>>> TESTS_ENVIRONMENT and I tried with atf-run.
>>
>> I see; interesting.  I blame the variable name then ;-)
> You can see the example at
> http://www.gnu.org/software/automake/manual/automake.html#Simple-Tests

Yep, I checked that before.  Confusing indeed.

>> Well, the problem with such simple script is: how do you pass options
>> to atf-run? And to atf-report?
> Yes, you are right. One way is to duplicate the options of the atf-run
> and atf-report in that script, but then it would not be a simple one
> any more.
>
>> My current ideas to resolve this are: merge atf-run and atf-report
>> into a single beast (I'm thinking of a binary called 'atf'; plain and
>> simple). Such tool would run the tests as it currently does, but
>> would store the results of the tests on-disk for future reference (and
>> with historical data!).
> This would be amazing! I have experience with junit and I struggled a
> lot to provide historical data. One of the problem was that test
> reports tended to be huge and due to space limitations I kept the ten
> latest reports. The test reports were in XML and I transformed them
> into HTML. It was the HTML files that were huge. I was able to extract
> historical data from the XML reports but no one would read XML so I
> outputted the data to HTML. JUnit is simply not designed to keep
> history, it can be extended to generate report in XML (or in any
> format), but it was the users' decision how to store and process those
> reports in the span of time - I created some ugly transformations on
> top of the XML reports.

Did you know that you can also create XML/HTML reports with atf right
now?  atf-report can output to XML, and atf ships with sample XSLT/CSS
files to process the output and generate a "nice" page.

>> [...]
>> or something like this.  You get the idea I hope :-)
> Yes, as I said above, this is very cool!

We'll eventually get there :-)  The route is not trivial but that's
certainly what I'd like to reach not so far in the future.

>> Making sure the tests get installed and work correctly as soon as you
>> add them is just a tiny amount of incremental work.  Doing all this
>> work as an afterthought once you have many tests will be painful, and
>> that's guaranteed.  So little suggestion: start adding the necessary
>> Makefile rules to install them now.
> You convinced me. I added the installcheck targets. Some new questions
> popped up. This is my latest Makefile.am
[...]
>        echo "The verbatim output of atf-run has been saved to" \
>             "$${logfile}; exit code was $${res}"; \
>        test $${res} -eq 0
> CLEANFILES += installcheck.log installcheck.fifo
>
>
> Question 1) check-atf and installcheck-atf targets contain duplicated
> code. How to remove that duplication? I know it is more of an
> make/automake question, but as I said early I am new to these tools?

A common, portable trick in Makefiles is to do the following:

# Expects 'dir' to be defined in the shell.
my_complex_script = cd $${dir} && atf-run | atf-repot

check-local:
    dir=$(top_builddir) $(my_complex_script)

installcheck-local:
    dir=$(testsdir) $(my_complex_script)

Just replace the contents of the my_complex_script variable with the
snippet you had above.

Be aware that running the tests form the source tree can be
nontrivial.  In particular, things may break if you use build
directories (note that I used top_builddir above, not srcdir).  If you
do 'make distcheck' you should be able to catch most of these
mistakes.

> Question 2) The variable check_PROGRAMS contains the test programs
> that will be executed during 'make check'. The variable tests_PROGRAMS
> contains the test programs that will be installed and executed during
> 'make installcheck'. In my case the two variables will contain the
> same information. Is acceptable to assign $(check_PROGRAMS) to
> $(tests_PROGRAMS):
>
> tests_PROGRAMS = $(check_PROGRAMS)

This should be fine.  But note that check_PROGRAMS are only built when
you do 'make check', not as part of a regular build.  Therefore, I'm
not sure if 'make install' will work well.  You can just avoid setting
check_PROGRAMS altogether because tests_PROGRAMS will be
unconditionally built anyway.

/me should collect all this information in a wiki page.

-- 
Julio Merino