Re: ATF head version make check failure, test use wrong atf-config tool

[Julio Merino <jmmv84%gmail.com@localhost>]

On Mon, Jul 20, 2009 at 4:44 PM, Nicolas Joly<njoly%pasteur.fr@localhost> wrote:
> On Mon, Jul 20, 2009 at 04:33:22PM +0000, Julio Merino wrote:
>> On Mon, Jul 20, 2009 at 4:09 PM, Nicolas Joly<njoly%pasteur.fr@localhost> 
>> wrote:
>> >
>> > Hi,
>> >
>> > I updated my ATF source tree and run the boostrap tests ... And got a
>> > failure from atf-config.
>> [...]
>> > A closer look shows that the testsuite picks up an already installed
>> > atf-config tool (version 0.6 from NetBSD base system), where it should
>> > test the new one ...
>>
>> Yeah... autotest is supposed to test installed tools as far as I know.
>
> Not quite true ... It can test whatever you want, just like any other
> shell script.
>
> By example, autoconf `make check' ensure that new shell scripts are
> syntaxically correct:
>
> AT_CHECK_SHELL_SYNTAX(["$abs_top_builddir/bin/autoconf"])
> AT_CHECK_SHELL_SYNTAX(["$abs_top_builddir/tests/autoconf"])
> AT_CHECK_SHELL_SYNTAX(["$abs_top_builddir/tests/testsuite"])

Quoting the autotest documentation (autoconf.info, 18.2 Writing testsuite.at):

-----
 -- Macro: AT_TESTED (EXECUTABLES)
     Log the file name and answer to `--version' of each program in
     space-separated list EXECUTABLES.  Several invocations register
     new executables, in other words, don't fear registering one program
     several times.

   Autotest test suites rely on `PATH' to find the tested program.
This avoids the need to generate absolute names of the various tools,
and makes it possible to test installed programs.  Therefore, knowing
which programs are being exercised is crucial to understanding problems
in the test suite itself, or its occasional misuses.  It is a good idea
to also subscribe foreign programs you depend upon, to avoid
incompatible diagnostics.
-----

>>  I will eventually take a look at that, because otherwise there is no
>> use in having make check and make installcheck as separate steps.  The
>> problem is... some things will not work if you don't run make install
>> before running make check.
>
> I always use `make check' before installatiom to validate that that
> things are not badly broken.

Yes, that's a neat idea.  But specially in these test cases, the
tested binaries may require to use data files that are not yet
installed, but they try to look for them in the installed locations.
Of course, this can be worked around by passing additional flags and
environment variables, but it may be tricky and too easy to get wrong.
 No matter what, though, it looks like this should be done.  Again,
that's the whole purpose of having check and installcheck as separate
targets.

-- 
Julio Merino