Re: rump is complete

[Antti Kantee <pooka%cs.hut.fi@localhost>]

On Mon Mar 21 2011 at 12:47:24 -0700, Sanjay Lal wrote:
> I was curious about running NetBSD kernel code in user land run on kernels 
> other than NetBSD?  For instance, running the NetBSD networking stack in 
> userland on Linux etc?  What would be required to get such a system going?

This is probably a FAQ, but I can't remember answering it except in
private, so one more go is useful.  If anyone investigates further,
some notes on the wiki might be in order (the same can be said about
how to use NetBSD-current kernel drivers on a release).

The short version is that it runs just fine, but there's always a ...

1) building

   Back in 2007 when I originally ran our FFS on Linux, I built the code
   with some concoctions of build.sh, nbmake USETOOLS=no and copypasted
   command lines.  It's not very difficult, but takes some tinkering.

   Since then, Arnaud Ysmal's most excellent fs-utils required rump
   support in pkgsrc.  The package makes compilation trivial, but it
   tends to go out-of-date.  In fact, currently it's really out-of-date,
   and at least I don't really know how to update it.

   So, what I think would be required is something like bootstrap-rump.sh,
   which would be included in the source tree and would compile the
   kernel and rumpuser with the right compiler flags.  The same script
   could be used to install -current rump onto a release.

2) host resource access

   Networking is not very exciting unless you can also access a network.
   On NetBSD, there's a few different way to do this:

   a) virt(4), which accesses the host network via a tap-like device
      from /dev/tap using read/write

   b) shmif(4), which uses a memory-mapped file as an ethernet bus.  as an
      implication, communication is limited to other networking stacks
      running on the same host (unless one of them routes traffic to the
      outside world)

   c) "sockin", which actually skips the networking stack and just emulates
      the inet protocol family on top of kernel sockets.  it's most useful
      when networking is _not_ the thing you're interested in virtualizing
      (think about e.g. the nfs client)

   You're probably interested in "a".  Adjusting virt(4) to talk to a
   Linux tap interface (or equivalent) is most likely not very difficult.

3) application interface

   This is where the gloves come off.  In a normal scenario the userland
   and kernel are in agreement about the ABI.  If you try to run kernel
   code on a foreign platform, the kernel code will have its own idea
   of the ABI and the userland will have its own idea.  I first noticed
   this when running FFS on Linux and trying to pass an off_t pointer
   in the hypercall interface ...

   You need userland applications at least to configure the networking
   stack.  This includes ioctl and r/w on the routing socket.  The problem
   is that there is no "NetBSD" namespace, so e.g. the name "struct
   msghdr" may have a collision between Linux and NetBSD, PF_INET may
   have a different value, etc.

   Part of the issue is that we cannot process our headers automatically
   to create a "NetBSD" namespace.  Incidentally, the "xmlif" project
   proposed on this list yesterday would solve part of the issue.  Even if
   you cannot autogenerate types properly, you can still do reasonable
   hacks with e.g. sed.  See src/sys/rump/include/makerumpdefs.sh (which
   is mostly written for file systems, but can be augmented).

   Finally, you must translate the types.  This is not difficult, but
   slightly boring.

   Depending on what your application is, you might not have to do much
   in this department.  E.g. a router runs pretty much without a userland,
   so you'd just have to implement the types necessary for configuration.
   If you want totally transparent application use, there's some more
   work involved with making librumphijack target the right syscall names.

4) hypercall interface

   The rumpuser hypercall interface should be clean of ABI assumptions
   (apart from the bits necessary to implement "sockin").  However,
   since I haven't really tested it on non-NetBSD in years, there might
   be a few snafus lurking around.


In total, I'd estimate getting the NetBSD TCP/IP stack working on Linux
to be a day or two of work, and full application support would require
some more days to get all the translation code written.  Apart from that,
it's good to reserve some time for "unforeseen issues".

  - antti

-- 
älä karot toivorikkauttas, kyl rätei ja lumpui piisaa