Subject: toolchain/23952: "make regress" lacks critical infrastructure
To: None <gnats-bugs@gnats.NetBSD.org>
From: Martin Husemann <firstname.lastname@example.org>
Date: 01/02/2004 19:17:46
>Synopsis: "make regress" lacks critical infrastructure
>Arrival-Date: Fri Jan 02 18:18:00 UTC 2004
>Originator: Martin Husemann
>Release: NetBSD 1.6ZG
System: NetBSD night-porter.duskware.de 1.6ZG NetBSD 1.6ZG (PORTER) #0: Thu Jan 1 14:05:24 CET 2004 email@example.com:/usr/src/sys/arch/i386/compile/PORTER i386
Regression tests are important. We have "make regress" for this, and it partly
works. Just to stress the point how important this is: when I started trying
this target last week, I could imediately panic the kernel on two archs
(i386 and sparc64) by running programms in the regression suite.
The fact that this problems have been so easily detectable (by just running
"make regress") and been undiscovered for a non trivial time shows there
is something wrong.
After playing with the regression tests now for some time, I found our
infrarstructure for it lacking in several ways. Here are the most critical
ones I see, please append to the PR if you know others.
(1) There is no (optional?) automatic logging and summary at completition
This is a SMOP, should be easy to fix.
(2) It's not clear if "make regress" should be run as a ordinary user
or as root. Some tests need to run as root (regress/sys/fs/ffs)
and "just fail" if run as non-root, while others need to run as
unprivileged user (regress/lib/libc/clone) and silently (!!!!)
skip important parts of the test (because it does not apply to root)
if run as root.
(3) library tests are only build and run dynamicaly linked on archs supporting
that. Some should be compiled and run static + dynamic, because the
PIC requirements show different paths in assembler code. There should
be some generic framework to support this.
Try to get a list of regression tests failing on your arch.
I see three solutions for the root/unprivileged problem:
- add "just in time sudo" support (like pkgsrc)
- split the "make regress" into three targtes: common, root, unpriviledged
- make the tests not working for the current user prominently log the
fact that they ran only a subset or are skiped completely
The latter requires some loging/summary framework though ;-)