bin/48624: atf integration seriously broken

[martin%NetBSD.org@localhost]

>Number:         48624
>Category:       bin
>Synopsis:       atf integration seriously broken
>Confidential:   no
>Severity:       critical
>Priority:       high
>Responsible:    bin-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Thu Feb 27 10:20:00 +0000 2014
>Originator:     Martin Husemann
>Release:        NetBSD 6.99.32
>Organization:
The NetBSD Foundation, Inc.
>Environment:
System: NetBSD thirdstage.duskware.de 6.99.32 NetBSD 6.99.32 (MODULAR) #124: 
Wed Feb 26 22:03:53 CET 2014 
martin%thirdstage.duskware.de@localhost:/usr/src/sys/arch/sparc64/compile/MODULAR
 sparc64
Architecture: sparc64
Machine: sparc64
>Description:

Since the last ATF import, many internal ATF tests fail:


Failed (bogus) test programs:
    atf-sh/misc_helpers, tools/zero_tcs_helper, tools/misc_helpers

Test cases for known bugs:
    test-programs/c_helpers:expect_death_and_exit: Exit case
    test-programs/c_helpers:expect_death_and_signal: Signal case
    test-programs/c_helpers:expect_exit_any_and_exit: Call will exit
    test-programs/c_helpers:expect_exit_code_and_exit: Call will exit
    test-programs/c_helpers:expect_fail_and_fail_check: And fail again: 2 
checks failed as expected; see output for more details
    test-programs/c_helpers:expect_fail_and_fail_requirement: Fail reason: The 
failure
    test-programs/c_helpers:expect_signal_any_and_signal: Call will signal
    test-programs/c_helpers:expect_signal_no_and_signal: Call will signal
    test-programs/c_helpers:expect_timeout_and_hang: Will overrun
    test-programs/cpp_helpers:expect_death_and_exit: Exit case
    test-programs/cpp_helpers:expect_death_and_signal: Signal case
    test-programs/cpp_helpers:expect_exit_any_and_exit: Call will exit
    test-programs/cpp_helpers:expect_exit_code_and_exit: Call will exit
    test-programs/cpp_helpers:expect_fail_and_fail_check: And fail again: 2 
checks failed as expected; see output for more details
    test-programs/cpp_helpers:expect_fail_and_fail_requirement: Fail reason: 
The failure
    test-programs/cpp_helpers:expect_signal_any_and_signal: Call will signal
    test-programs/cpp_helpers:expect_signal_no_and_signal: Call will signal
    test-programs/cpp_helpers:expect_timeout_and_hang: Will overrun
    test-programs/sh_helpers:expect_death_and_exit: Exit case
    test-programs/sh_helpers:expect_death_and_signal: Signal case
    test-programs/sh_helpers:expect_exit_any_and_exit: Call will exit
    test-programs/sh_helpers:expect_exit_code_and_exit: Call will exit
    test-programs/sh_helpers:expect_fail_and_fail_requirement: Fail reason: The 
failure
    test-programs/sh_helpers:expect_signal_any_and_signal: Call will signal
    test-programs/sh_helpers:expect_signal_no_and_signal: Call will signal
    test-programs/sh_helpers:expect_timeout_and_hang: Will overrun
    tools/expect_helpers:death_and_exit: Exit case
    tools/expect_helpers:death_and_signal: Signal case
    tools/expect_helpers:exit_any_and_exit: Call will exit
    tools/expect_helpers:exit_code_and_exit: Call will exit
    tools/expect_helpers:fail_and_fail_check: And fail again: 2 checks failed 
as expected; see output for more details
    tools/expect_helpers:fail_and_fail_requirement: Fail reason: The failure
    tools/expect_helpers:signal_any_and_signal: Call will signal
    tools/expect_helpers:signal_no_and_signal: Call will signal
    tools/expect_helpers:timeout_and_hang: Will overrun

Failed test cases:
    atf-c/macros_test:detect_unused_tests, atf-c/pkg_config_test:version, 
atf-c++/macros_test:detect_unused_tests, atf-c++/pkg_config_test:version, 
test-programs/c_helpers:cleanup_fail, test-programs/c_helpers:cleanup_pass, 
test-programs/c_helpers:cleanup_sigterm, test-programs/c_helpers:cleanup_skip, 
test-programs/c_helpers:config_empty, 
test-programs/c_helpers:config_multi_value, 
test-programs/c_helpers:config_value, 
test-programs/c_helpers:expect_death_but_pass, 
test-programs/c_helpers:expect_exit_but_pass, 
test-programs/c_helpers:expect_fail_but_pass, 
test-programs/c_helpers:expect_pass_but_fail_check, 
test-programs/c_helpers:expect_pass_but_fail_requirement, 
test-programs/c_helpers:expect_signal_but_pass, 
test-programs/c_helpers:expect_timeout_but_pass, 
test-programs/c_helpers:result_fail, 
test-programs/c_helpers:result_newlines_fail, 
test-programs/c_helpers:srcdir_exists, test-programs/cpp_helpers:config_empty, 
test-programs/cpp_helpers:config_multi_value, test-programs
 /cpp_hel
 pers:config_value, test-programs/cpp_helpers:expect_death_but_pass, 
test-programs/cpp_helpers:expect_exit_but_pass, 
test-programs/cpp_helpers:expect_fail_but_pass, 
test-programs/cpp_helpers:expect_pass_but_fail_check, 
test-programs/cpp_helpers:expect_pass_but_fail_requirement, 
test-programs/cpp_helpers:expect_signal_but_pass, 
test-programs/cpp_helpers:expect_timeout_but_pass, 
test-programs/cpp_helpers:result_exception, 
test-programs/cpp_helpers:result_fail, 
test-programs/cpp_helpers:result_newlines_fail, 
test-programs/cpp_helpers:srcdir_exists, test-programs/sh_helpers:cleanup_fail, 
test-programs/sh_helpers:cleanup_pass, 
test-programs/sh_helpers:cleanup_sigterm, 
test-programs/sh_helpers:config_multi_value, 
test-programs/sh_helpers:config_value, 
test-programs/sh_helpers:expect_death_but_pass, 
test-programs/sh_helpers:expect_exit_but_pass, 
test-programs/sh_helpers:expect_fail_and_fail_check, 
test-programs/sh_helpers:expect_fail_but_pass, 
test-programs/sh_helpers:expect_pass_bu
 t_fail_c
 heck, test-programs/sh_helpers:expect_pass_but_fail_requirement, 
test-programs/sh_helpers:expect_signal_but_pass, 
test-programs/sh_helpers:expect_timeout_but_pass, 
test-programs/sh_helpers:result_fail, test-programs/sh_helpers:srcdir_exists, 
tools/expect_helpers:death_but_pass, tools/expect_helpers:exit_but_pass, 
tools/expect_helpers:fail_but_pass, tools/expect_helpers:pass_but_fail_check, 
tools/expect_helpers:pass_but_fail_requirement, 
tools/expect_helpers:signal_but_pass, tools/expect_helpers:timeout_but_pass, 
tools/fail_helper:main

Summary for 78 test programs:
    700 passed test cases.
    58 failed test cases.
    35 expected failed test cases.
    6 skipped test cases.

Looking at the first of those, some blatant errors show up:

# pkg-config --modversion atf-c
$(cat atf-version.txt)
# atf-version
Automated Testing Framework  (atf-)
Copyright (c) 2007 The NetBSD Foundation, Inc.


Many of the other failures are due to things like (in C):

        atf_tc_get_config_var(tc, "tmpfile")

failing (tmpfile is apparently not defined).


>How-To-Repeat:
cd /usr/tests/atf
atf-run | atf-report

>Fix:
n/a - maybe we should go back to the last working state for now?
Alternatively we can skip the atf internal tests alltogether.