Re: ATF confusion with improperly terminated script

[Garrett Cooper <yanegomi%gmail.com@localhost>]

On Sun, Feb 6, 2011 at 12:03 PM, Julio Merino 
<jmmv%homeworld.netbsd.org@localhost> wrote:
> On Sun, Feb 06, 2011 at 11:00:59AM -0800, Garrett Cooper wrote:
>> On Sun, Feb 6, 2011 at 8:00 AM, Julio Merino 
>> <jmmv%homeworld.netbsd.org@localhost> wrote:
>> > On Sun, Feb 06, 2011 at 05:53:54AM -0800, Garrett Cooper wrote:
>> >> Hi again Julio, et all,
>> >> ? ? Until one runs set -x in the script below, it isn't apparent why
>> >> the script was failing. Is this issue going to be fixed in the future
>> >> with kyua?
>> >
>> > Error reporting in atf-run is weak. ?I have filed issue 13 in Kyua to
>> > ensure this is not forgotten:
>> >
>> > http://code.google.com/p/kyua/issues/detail?id=13
>> >
>> > Is this lacking any detail?
>>
>> I think that captures the bulk majority of my concerns, but there's
>> another one I'm currently working through to try and understand why
>> atf-sh isn't properly running testcases that *should* (in theory) work
>> as the shell code is sane (just a bit convoluted because I'm trying to
>> integrate the /bin/sh testcases from FreeBSD without converting or
>> generating them all beforehand).
>>
>> The result of the integration effort attempt is below and I can post
>> the error log up on pastebin if desired (but before that it would be
>> nice if there's a variable I could pass to atf-run to debug what's
>> happening exactly from ATF's perspective when executing a testcase).
>
> Ah, yes.  ATF does not have such feature at the moment.  That is something
> I wanted to address in Kyua but have kept forgetting so far.  I have
> filed issue 14 to keep track of this:
>
> http://code.google.com/p/kyua/issues/detail?id=14

Cool -- thanks!

> That said, atf-run should not be causing any issues here.  Your problems
> most likely come from atf-sh; while not particularly nice, you could add
> debugging/tracing commands to libatf-sh.subr.

I was hoping to avoid that, but oh well :/...

>> #!/usr/bin/env atf-sh
>
> The way we do this in NetBSD is through some build magic that adds the
> shebang header and makes it point to the exact atf-sh binary that ships
> with the system.  Dunno if it is a good idea to make it rely on the path.

It's not really so much of an issue with us, but some projects (like
the python setuputils project) use a shebang like above and replace at
install time.

I was just following Giorgos's lead to make sure I was on a similar track.

>> # $FreeBSD$
>>
>> export SH=${SH:=/bin/sh}
>>
>> do_test() {
>>         local atf_args expected_exit_status testcase_path testcase_prefix
>>
>>         testcase_path=$1
>
> If you need access to data files that live in the same directory as the test
> program, you should query $(atf_get_srcdir).

Sadly, they're in subdirectories (otherwise I wouldn't care about the
path a lot).

>>         expected_exit_status=${testcase_path##*.}
>>         testcase_prefix=${testcase_path%.*}
>>
>>         atf_args="-s exit:${expected_exit_status}"
>>
>>         for i in e,stderr o,stdout; do
>>                 output="${testcase_prefix}.${i##*,}"
>>                 if [ -f "$output" ]; then
>>                         atf_args="${atf_args} -${i%,*} '$output'"
>>                 fi
>>         done
>>
>>         echo "INFO: atf_args: $atf_args, expected_exit_status:
>> $expected_exit_status, testcase_path: $testcase_path, testcase_prefix:
>> $testcase_prefix" >&2
>>
>>         atf_check ${atf_args} ${SH} ${testcase_path}
>> }
>>
>> set -x
>>
>> atf_init_test_cases() {
>>         local testcase testcase_path
>>
>>         for testcase_path in $(find -Es . -regex ".*\.[0-9]+"); do
>>
>>                 testcase=$(echo "$testcase_path" | sed -e 's,^./,,g'
>> -e 's,/,_,g' -e 's,\.,__,g' -e 's,-,____,g')
>>
>>                 atf_test_case "test__$testcase"
>>                 # XXX: would be nice if this was more descriptive,
>> i.e. maybe comments
>>                 # in the header, or a custom description file.
>>                 eval "test__${testcase}_head() { atf_set 'descr'
>> '$testcase testcase'; }"
>
> If you feel that the descr is redundant, you don't need to set it.  I would
> just skip this.  (And recently, I am personally skipping pretty much all
> descriptions in favor of providing descriptive test case names.  I find
> myself introducing less typos and inconsistencies this way.)

Well, they'll be helpful to testers because jilles@ didn't provide
descriptions of the testcases beyond what's in commit logs in FreeBSD,
and that tends to change as time passes by.

>>                 eval "test__${testcase}_body() { do_test '$testcase_path'; }"
>>
>>                 atf_add_test_case "test__$testcase"
>>
>>         done
>> }
>
> Ouch; the code is indeed a bit convoluted!  I would suggest to start simple:
> avoid all the logic to "autogenerate" the tests.  Define a single one by
> hand to ensure there is no eval nastiness.  Also, run one particular test
> without atf-run so that you can gain access to any generated temporary
> files, etc.

These work:

atf_test_case t_true
t_true_head() { atf_set 'descr' 'dat!'; }
t_true_body() { atf_check -s exit:0 true; }

atf_test_case t__false
t__false_head() { atf_set 'descr' 'haru?'; }
t__false_body() { atf_check -s exit:1 false; }

atf_test_case t_false
t_false_head() { atf_set 'descr' 'haru??'; }
t_false_body() { atf_check -s exit:1 false; }

> What I mean is:
>
> $ mkdir tmp
> $ cd tmp
> $ ../my_test_program the_test_case
> $ ls
> ... ah ha!  the temporary files for the test program are here ...

It's not even getting that far. When I don't execute the testcase
standalone it's getting hung up on some logic in libatf-sh.subr (in
particular the testcase names aren't ending up in calls to
atf_test_case ... curious). The contents of Defined_Test_Cases is ok
for a while, but the majority of the content in the value is getting
dropped about halfway through the invocation.

There appears to be light at the end of the tunnel, but I need to make
sure it's not an oncoming train :)...

Thanks,
-Garrett