Re: How to integrate atf with autotools

["Ivan \"Rambius\" Ivanov" <rambiusparkisanius%gmail.com@localhost>]

Hello Julio,

On Mon, Jul 12, 2010 at 11:43 AM, Julio Merino <jmmv%netbsd.org@localhost> 
wrote:
>> Yes, you are right. However, I had to start from somewhere and I saw
>> some examples in automake manual with perl command in
>> TESTS_ENVIRONMENT and I tried with atf-run.
>
> I see; interesting.  I blame the variable name then ;-)
You can see the example at
http://www.gnu.org/software/automake/manual/automake.html#Simple-Tests
.

> Well, the problem with such simple script is: how do you pass options
> to atf-run? And to atf-report?
Yes, you are right. One way is to duplicate the options of the atf-run
and atf-report in that script, but then it would not be a simple one
any more.

> My current ideas to resolve this are: merge atf-run and atf-report
> into a single beast (I'm thinking of a binary called 'atf'; plain and
> simple). Such tool would run the tests as it currently does, but
> would store the results of the tests on-disk for future reference (and
> with historical data!).
This would be amazing! I have experience with junit and I struggled a
lot to provide historical data. One of the problem was that test
reports tended to be huge and due to space limitations I kept the ten
latest reports. The test reports were in XML and I transformed them
into HTML. It was the HTML files that were huge. I was able to extract
historical data from the XML reports but no one would read XML so I
outputted the data to HTML. JUnit is simply not designed to keep
history, it can be extended to generate report in XML (or in any
format), but it was the users' decision how to store and process those
reports in the span of time - I created some ugly transformations on
top of the XML reports.

On the other hand, if ATF is able to provide historical data
out-of-the box it would be a great advantage.

> After the execution, the unified tool would
> just print the summary of the run as atf-report does now (and report a
> correct exit code).  With such approach, you could be able to run the
> tests getting a nice progress report but, if something fails, you'd be
> able to extract details of the failure.
>
> It would go along the lines of:
>
> $ atf run
> ... blah, blah, blah, you see the same as atf-report prints now
> $ echo $?
> 1
> ... oh! something failed!  What tests cases failed?
> $ atf query result=failed
> foo/bar/baz_test:my_test_case
> foo/bar/baz_test:another_test_case
> $ atf inspect foo/bar/baz_test:my_test_case
> ... and you get the failure reason, the stdout, the stderr of the last run
> $ atf history foo/bar/baz_test:my_test_case
> 20100701123500: failed, foo bar
> 20100630161057: passed
> ...
>
> or something like this.  You get the idea I hope :-)
Yes, as I said above, this is very cool!

> Making sure the tests get installed and work correctly as soon as you
> add them is just a tiny amount of incremental work.  Doing all this
> work as an afterthought once you have many tests will be painful, and
> that's guaranteed.  So little suggestion: start adding the necessary
> Makefile rules to install them now.
You convinced me. I added the installcheck targets. Some new questions
popped up. This is my latest Makefile.am

CLEANFILES =

bin_PROGRAMS = countnb
check_PROGRAMS = test_seed_prng

countnb_SOURCES = countnb.c

test_seed_prng_SOURCES = seed_prng.c test_seed_prng.c

testsdir = $(exec_prefix)/tests/cryptoexamples
tests_DATA = Atffile
tests_PROGRAMS = $(check_PROGRAMS)

check-local: check-atf
.PHONY: check-atf
check-atf:
        logfile=$$(pwd)/check.log; \
        fifofile=$$(pwd)/check.fifo; \
        rm -f $${fifofile}; \
        mkfifo $${fifofile}; \
        cat $${fifofile} | tee $${logfile} | atf-report & \
        atf-run >>$${fifofile}; \
        res=$${?}; \
        wait; \
        rm $${logfile}; \
        echo "The verbatim output of atf-run has been saved to" \
             "$${logfile}; exit code was $${res}"; \
        test $${res} -eq 0
CLEANFILES += check.fifo check.log

installcheck-local: installcheck-atf
.PHONY: installcheck-atf
installcheck-atf:
        logfile=$$(pwd)/installcheck.log; \
        fifofile=$$(pwd)/installcheck.fifo; \
        cd $(testsdir); \
        rm -f $${fifofile}; \
        mkfifo $${fifofile}; \
        cat $${fifofile} | tee $${logfile} | atf-report & \
        atf-run >>$${fifofile}; \
        res=$${?}; \
        wait; \
        rm $${fifofile}; \
        echo; \
        echo "The verbatim output of atf-run has been saved to" \
             "$${logfile}; exit code was $${res}"; \
        test $${res} -eq 0
CLEANFILES += installcheck.log installcheck.fifo


Question 1) check-atf and installcheck-atf targets contain duplicated
code. How to remove that duplication? I know it is more of an
make/automake question, but as I said early I am new to these tools?

Question 2) The variable check_PROGRAMS contains the test programs
that will be executed during 'make check'. The variable tests_PROGRAMS
contains the test programs that will be installed and executed during
'make installcheck'. In my case the two variables will contain the
same information. Is acceptable to assign $(check_PROGRAMS) to
$(tests_PROGRAMS):

tests_PROGRAMS = $(check_PROGRAMS)


>> If I go with installing the tests of my application
>> should I installed them under $prefix/tests/<appname>?
>
> Yep, that's correct.
OK, my Makefile.am installs them there.

Thank you very much!

Regards
Rambius

-- 
Tangra Mega Rock: http://www.radiotangra.com