On 01/12/14 08:01, Martin Husemann wrote:
On Mon, Dec 01, 2014 at 11:54:23AM +0900, Ryota Ozaki wrote:BTW, we may need ATF tests to ring the bell for such regressions.This is a bit tricky to do as we do not expose a sane api to create "old binaries" from source. It would boil down to copy & paste of the old structure definition and some magic to test ioctls with them. Testing the ioctls itself against a rump kernel is very simple. But then you'll have to install helpers (like rump.ifconfig but for various ioctl versions) and it quickly gets messy. So no easy/safe/scalable way there that I can see. The other option that comes to mind is running a -current kernel and doing a -7 and -6 release branch test run. Unfortunately that will include all RUMP components from that branch as well, so not test the compatibility we are looking for either. But I guess we should think about rearrangements in this direction to make it easier in the future - starting with a separate rump.tgz set, or maybe a few of them, not sure what granularity would be needed there. Antti, Justin, can we separate the rump userland from the rump kernel binaries easily and put them in different sets, and then run the userland from -6 or -7 against the (rump-)kernel from -current?
Copying prebuilt userland tarballs around sounds suboptimal. Unless you want to go cherry-picking every time something changes in the tests, you'd need all of bins/libs. And then you'd probably need to carefully tune your PATH for the tests to still work. Instead, I'd go the opposite route which allows for straightforward, self-contained automation:
You can use buildrump.sh (*) with a -6 or -7 toolchain to get a -current rump kernel. Theoretically speaking, you can then run the -current tests as an additional part of -6 and -7 test runs with LD_LIBRARY_PATH set to buildrump.sh/rump/lib. I doubt anyone has run /usr/tests with host != rump kernel since I was working on NetBSD 5.0 and testing -current, um, 4 years ago (and I was just installing -current to /), so expect some tuning to be necessary.
The downside to the above approach is that if you want to manually run the tests, you need a -6/-7 installation, but once in the blue moon when it's necessary to run the tests manually instead of relying on automation, you'd just run anita and wait a few minutes.
*) ./checkout cvs src HEAD ; ./buildrump.sh fullbuild^ assuming _on_ -6/-7. Otherwise you also need to set $CC to the appropriate cross compiler before running buildrump.sh.