Re: ATF confusion with improperly terminated script

[Julio Merino <jmmv%homeworld.netbsd.org@localhost>]

On Sun, Feb 06, 2011 at 11:00:59AM -0800, Garrett Cooper wrote:
> On Sun, Feb 6, 2011 at 8:00 AM, Julio Merino 
> <jmmv%homeworld.netbsd.org@localhost> wrote:
> > On Sun, Feb 06, 2011 at 05:53:54AM -0800, Garrett Cooper wrote:
> >> Hi again Julio, et all,
> >> ? ? Until one runs set -x in the script below, it isn't apparent why
> >> the script was failing. Is this issue going to be fixed in the future
> >> with kyua?
> >
> > Error reporting in atf-run is weak. ?I have filed issue 13 in Kyua to
> > ensure this is not forgotten:
> >
> > http://code.google.com/p/kyua/issues/detail?id=13
> >
> > Is this lacking any detail?
> 
> I think that captures the bulk majority of my concerns, but there's
> another one I'm currently working through to try and understand why
> atf-sh isn't properly running testcases that *should* (in theory) work
> as the shell code is sane (just a bit convoluted because I'm trying to
> integrate the /bin/sh testcases from FreeBSD without converting or
> generating them all beforehand).
> 
> The result of the integration effort attempt is below and I can post
> the error log up on pastebin if desired (but before that it would be
> nice if there's a variable I could pass to atf-run to debug what's
> happening exactly from ATF's perspective when executing a testcase).

Ah, yes.  ATF does not have such feature at the moment.  That is something
I wanted to address in Kyua but have kept forgetting so far.  I have
filed issue 14 to keep track of this:

http://code.google.com/p/kyua/issues/detail?id=14

That said, atf-run should not be causing any issues here.  Your problems
most likely come from atf-sh; while not particularly nice, you could add
debugging/tracing commands to libatf-sh.subr.

> #!/usr/bin/env atf-sh

The way we do this in NetBSD is through some build magic that adds the
shebang header and makes it point to the exact atf-sh binary that ships
with the system.  Dunno if it is a good idea to make it rely on the path.

> # $FreeBSD$
> 
> export SH=${SH:=/bin/sh}
> 
> do_test() {
>         local atf_args expected_exit_status testcase_path testcase_prefix
> 
>         testcase_path=$1

If you need access to data files that live in the same directory as the test
program, you should query $(atf_get_srcdir).

>         expected_exit_status=${testcase_path##*.}
>         testcase_prefix=${testcase_path%.*}
> 
>         atf_args="-s exit:${expected_exit_status}"
> 
>         for i in e,stderr o,stdout; do
>                 output="${testcase_prefix}.${i##*,}"
>                 if [ -f "$output" ]; then
>                         atf_args="${atf_args} -${i%,*} '$output'"
>                 fi
>         done
> 
>         echo "INFO: atf_args: $atf_args, expected_exit_status:
> $expected_exit_status, testcase_path: $testcase_path, testcase_prefix:
> $testcase_prefix" >&2
> 
>         atf_check ${atf_args} ${SH} ${testcase_path}
> }
> 
> set -x
> 
> atf_init_test_cases() {
>         local testcase testcase_path
> 
>         for testcase_path in $(find -Es . -regex ".*\.[0-9]+"); do
> 
>                 testcase=$(echo "$testcase_path" | sed -e 's,^./,,g'
> -e 's,/,_,g' -e 's,\.,__,g' -e 's,-,____,g')
> 
>                 atf_test_case "test__$testcase"
>                 # XXX: would be nice if this was more descriptive,
> i.e. maybe comments
>                 # in the header, or a custom description file.
>                 eval "test__${testcase}_head() { atf_set 'descr'
> '$testcase testcase'; }"

If you feel that the descr is redundant, you don't need to set it.  I would
just skip this.  (And recently, I am personally skipping pretty much all
descriptions in favor of providing descriptive test case names.  I find
myself introducing less typos and inconsistencies this way.)

>                 eval "test__${testcase}_body() { do_test '$testcase_path'; }"
> 
>                 atf_add_test_case "test__$testcase"
> 
>         done
> }

Ouch; the code is indeed a bit convoluted!  I would suggest to start simple:
avoid all the logic to "autogenerate" the tests.  Define a single one by
hand to ensure there is no eval nastiness.  Also, run one particular test
without atf-run so that you can gain access to any generated temporary
files, etc.

What I mean is:

$ mkdir tmp
$ cd tmp
$ ../my_test_program the_test_case
$ ls
... ah ha!  the temporary files for the test program are here ...

-- 
Julio Merino