I'd like to run our atf-based tests (on netbsd-6) from a script, to hook into a larger regression test system for other code. I have a few questions, not answered by the tests(7) man page on the netbsd-6 branch. I ran tests two ways. Once was a myself, and once as root. The results were somewhat different. I realize some tests fail sometimes and not others, and that's just the way it is. But there were a lot more skipped tests as a non-root user. Diffing from non-root to root, I see 57 skipped rather than 301: @@ -4612,10 +4615,10 @@ ./util/sh/t_varquote:nested_quotes_multiword: PR bin/43597: atf-check failed; see the output of the test for details Failed test cases: - ./fs/vfs/t_renamerace:lfs_renamerace_dirs, ./lib/libpthread/t_cond:cond_timedwait_race, ./lib/librumphijack/t_tcpip:http, ./libexec/ld.elf_so/t_dl_symver:dl_symver + ./fs/vfs/t_renamerace:lfs_renamerace_dirs, ./lib/libpthread/t_cond:cond_timedwait_race, ./libexec/ld.elf_so/t_dl_symver:dl_symver Summary for 500 test programs: - 2742 passed test cases. - 4 failed test cases. - 31 expected failed test cases. - 301 skipped test cases. + 2984 passed test cases. + 3 failed test cases. + 34 expected failed test cases. + 57 skipped test cases. Presumably that explains the 3 additional xfail, although I don't understand why a resize_ffs test would need privileges. @@ -4600,8 +4600,11 @@ ./lib/libm/t_pow:powf_one_neg_x: PR lib/45372: 2 checks failed as expected; see output for more details ./lib/libm/t_pow:powf_zero_x: PR port-amd64/45391: 3 checks failed as expected; see output for more details ./lib/libpthread/t_sem:before_start_one_thread: Race condition; it is probably unsafe to call sem_post from a signal handler when using the pthread version + ./lib/librt/t_sched:sched_rr_get_interval_1: PR lib/44768: /u0/n0/gdt/NetBSD-6/src/tests/lib/librt/t_sched.c:207: sched_rr_get_interval(-INT_MAX, &tv) != 0 not met ./lib/semaphore/pthread/t_sem_pth:unlink: PR kern/43452: close unlinked semaphore: Invalid argument ./sbin/ifconfig/t_nonexistent:nonexistent: PR bin/43141: atf-check failed; see the output of the test for details + ./sbin/resize_ffs/t_shrink:shrink_24M_16M_v2_4096: PR bin/44205: atf-check failed; see the output of the test for details + ./sbin/resize_ffs/t_shrink_swapped:shrink_24M_16M_v2_16384: PR bin/44205: atf-check failed; see the output of the test for details ./sbin/route/t_missing:missing: PR bin/42179: atf-check failed; see the output of the test for details ./sbin/sysctl/t_perm:sysctl_kern: PR kern/44946: atf-check failed; see the output of the test for details ./usr.bin/diff/t_diff:mallocv: PR bin/26453: atf-check failed; see the output of the test for details So my questions are: 1) Are the three failures above failing for everyone? Should they be marked XFAIL? 2) What is the best-practices recommendation? To run tests as root, or as a non-root user? tests(7) hints that _test is used, so presumably running as root drops privileges for tests that don't need it. 3) In general, are the tests thought to be independent of the state of the underlying system? Is it valid to run the on a machine at single-user with zero configuration? (Surely, that seems intended, and is what anita does.) On a multi-user machine with services running? Is it a bug if the environment matters?
Attachment:
pgpckYBd8JZYU.pgp
Description: PGP signature