Re: Please RUMPify tests in sbin/ifconfig

[Greg Troxel <gdt%lexort.com@localhost>]

Jukka Ruohonen <jruohonen%iki.fi@localhost> writes:

> On Sat, Jul 18, 2020 at 06:05:28AM -0400, Greg Troxel wrote:
>> I think it's reasonable and useful to have tests, but it is not ok for
>> them to run from "atf-run ." by default.  They need to be opted into
>> with some argument that says "run tests that modify the current system,
>> which means that they cannot be used except on a system whose entire
>> current purpose is testing".
>
> I agree: and when a panic/hang/etc. is explicitly known to occur, a test
> case is always skipped. However: if you can crash a system from userland
> with standard system utilities, it is always a serious issue. (Fortunately,
> privileges are required in these tests, so the issues are not security
> bugs.) Trying to hide panics in order to comply with some unwritten test
> suite requirements is not only wrong but also dangerous.

Yes, but we are not just talking about skipping the test once the panic
is known.  We are talking about ensuring that on-by-default tests do not
change the configuration of the test system.

As an example, I have an ancient laptop running 9-stable at the moment
that I use to do (test) package builds for i386, and I just went and
started "atf-run ." in /usr/tests.   Whether run as root or a non-root
uid, I expect the system to remain on the net during the tests.

See tests(7) and note:

     [when to run tests section]

          •   Periodically, maybe from a cron(8) job, to ensure that any
               changes to the system (such as the installation of third-party
               packages or manual modifications to configuration files) do not
               introduce unexpected failures.

     Ideally, tests are self-contained and do not either depend on or perturb
     the host environment, aside from skipping tests when optional facilities
     are not available.  In reality, tests load and unload modules, and do
     other things that might cause problems.  While it is not entirely safe to
     run tests on a multi-user system, permanent problems or crashes from
     doing so are viewed as bugs and should be reported.


I realize there is a bit of fuzz, but this is the notion that end-users
are encouraged to run tests via cron on production system.  I think
that's fine - but that leads to not breaking the host.