Re: Lightweight virtualization - the rump approach

[Antti Kantee <pooka%cs.hut.fi@localhost>]

On Thu May 13 2010 at 18:51:16 +0200, Jean-Yves Migeon wrote:
> I am not posting this to reinstate the decades old monolithic vs 
> microkernel troll, so please avoid that field; thanks.

I hope people on these lists are adult enough to realize that argument
is pointless.  The correct answer to which is better is of course "both"
(or "neither", as code I hope get committed later today or during the
weekend will quite measurably demonstrate).

> Lights on the work of Antti, with rump. Most systems I have seen lately 
> provide the characteristics enumerated above by pulling in a general 
> purpose OS, like Linux, with its environment, just to get a specific 
> need, like an up-to-date network stack (strong push for IPv6, anyone?), 
> drivers (filesystems, usb, pci stacks, devices), etc. There is no real 
> componentization in mind.
> 
> Everything being virtual these days (see cloud computing buzzwords, or 
> hardware systems delivered with some kind of hypervisor inside - PS3, or 
> sun4v, for example -), I see Antti's work (well, all TNF work too ;) ) 
> as being a real asset to make the NetBSD's code base more widely known, 
> appreciated and used. I have yet to see a solution where you could port 
> then debug kernel code directly to userland (at least, for a general 
> purpose OS), or offer an alternative when you need to port specific 
> components, like network functionality or filesystem code.

My guess of what is going to happen in the future is that the historic
kernel/user boundary and even the OS will mostly go away and you'll just
be left with semi-independent "virtualized" application stacks running on
minimal hosts, perhaps on ASICs.  The OS is pure unnecessary overhead.
To take a food analogy: "flavour rules", not if the food was baked,
broiled, sauteed or cooked sous vide (sous vided? ;).

> For this reason, I have a few questions for the ones familiar with rump 
> technology here, especially:
> 
> - the basic architecture; how did/do you achieve such a functionality? 
> Adding an extra layer of indirection within the kernel?

There's no artificial extra layer of indirection in the code (and,
compared to other virtualization technologies, there's one less at
runtime).  It's mostly about plugging into key places and understanding
what you can use directly from the host (e.g. locking, which is of
course very intertwined with scheduling) and what you really don't want
to be emulating in userspace at all (e.g. virtual memory).  Due to some
magical mystery reason, even code which was written 20 years ago tends
to allow for easy separation.

The other part is more or less completing the work on kernel module
support in NetBSD, mostly minor issues with config left (devsw,
SRCS) and I've got those somewhat done in a corner of my source tree.
Rump components more or less follow the same functional units as kernel
modules apart from the rump{dev,net,vfs} factions, which cannot be
dynamically loaded and without which a regular kernel would not function.
Yes, I decided to use the word "faction" to describe the three midlayers
between rumpkern and the drivers.

The only real problem is the loosy-goosy use of inlines/macros in
unnecessary places.  But luckily for me, a lot of the work to clean that
up was done by Andy when he made the x86 ports modular.

> Suppose we have 
> improvements in one part of it, like TCP, IP stacks, could it directly 
> benefit the rumpnet component, or any service sitting above it?

Could you elaborate this question?

> - What kind of effort would it require to port it to other OS 
> architectures, especially when the API they offer could be a subset of 
> POSIX, or specific low level API (like Xen's hypercalls)? (closely 
> related to the work of Arnaud Ysmall in misc/rump [4])

As you probably know, rump uses the "rumpuser" hypercall interface to
access the hypervisor (which is currently just a fancy name for userland
namespace).  It pretty much evolved with the "oh I need this?  ok, I'll
add it" technique.  But, if someone wants to experiment with minimal
hosts, I think we can work on rumpuser a bit and see what qualities the
hosts have in common and what's different.  I don't expect supporting
rump to be any more difficult than for example Wombat/Iguana L4+UML.
In fact, it's probably simpler since with rump there is no notion of
things like address space -- that comes entirely from the host.

Running directly on top of Xen is an interesting idea, but the first
question I have with that is "why?", i.e. what is the gain as opposed
running directly in a process on dom0?  The only reason I can think of
is that you don't trust your dom0 OS enough, but then again you can't
really trust Xen guests either?  Besides, rump kernels do not execute
any privileged instructions, so Xen doesn't sound like the right hammer.

> - If rump could be used both for lightweight virtualization (like rump 
> fs servers), or more heavyweight one (netbsd-usermode...)?

Usermode = rump+more, although paradoxically rump = usermode+more also
applies (where more != 0 and usermode != rump ;).

I'm no longer really sure why someone would want to use a usermode
operating system on top of POSIX at all, since scheduling in them seems
quite inefficient in my experience.  But if someone does, the overlap is
that lot of the drivers written for rump can be used, for example ugenhc
which accesses host USB devices or virtif which shovels network traffic
to/from /dev/tap.  rumpfs could probably be extended into a fullblown
hostfs, etc ...

I have written some code for a transparent system call server against a
rump kernel.  It currently works for simple syscalls, but not much more.
With a little more in-kernel support (such as fork/exec/vm) and a
little clarification about what is a process, it could quite painlessly
morph into a usermode OS which is probably more efficient than a purely
userspace usermode OS.  Of course, the in-kernel helper approach limits
the hostability of the usermode OS, but current usermode OSs aren't
exactly known for their wide hostability anyway.

> - eventually, extra icing on the cake, especially with the appearance of 
> IOMMU in common x86. Could rump be used for (device) drivers to offer 
> better isolation and cause less harm to the kernel?

I assume you're talking about device drivers which do DMA by themselves,
since others (e.g. USB, bpf, audio) already work.  Theoretically yes,
but someone needs to write the code ;)

I touched this issue a little in my AsiaBSDCon paper this year.  The IOMMU
technique for unprivileged DMA'ing device drivers has been described by
LeVasseur et. al in their OSDI'04 paper.  I think rump should be able
to follow the same steps here, albeit possibly with a slightly simpler
and more lightweight virtual container.

One of the key points to understand is that rump is only half of the
solution if you want to isolate a driver used by the host.  You also need
some microkernel-like server linkback to integrate with the host.  Current
examples are p2k(3) for file systems or my various proof-of-concept pud(4)
servers for USB device drivers in src/share/examples/rump.

  - antti