Re: Lightweight virtualization - the rump approach

[Jean-Yves Migeon <jeanyves.migeon%free.fr@localhost>]

On 05/14/10 17:29, Antti Kantee wrote:
> On Thu May 13 2010 at 18:51:16 +0200, Jean-Yves Migeon wrote:
> > For this reason, I have a few questions for the ones familiar with rump
> > technology here, especially:
> >
> > - the basic architecture; how did/do you achieve such a functionality?
> > Adding an extra layer of indirection within the kernel?
>
> There's no artificial extra layer of indirection in the code (and,
> compared to other virtualization technologies, there's one less at
> runtime).  It's mostly about plugging into key places and understanding
> what you can use directly from the host (e.g. locking, which is of
> course very intertwined with scheduling) and what you really don't want
> to be emulating in userspace at all (e.g. virtual memory).  Due to some
> magical mystery reason, even code which was written 20 years ago tends
> to allow for easy separation.

"Magical mystery reason?" Could you elaborate on that one? Is it due to 
architecture decisions made in the past to build up operating systems, 
or is it rather specific to the code base you are working with?

> The other part is more or less completing the work on kernel modul
> support in NetBSD, mostly minor issues with config left (devsw,
> SRCS) and I've got those somewhat done in a corner of my source tree.
> Rump components more or less follow the same functional units as kernel
> modules apart from the rump{dev,net,vfs} factions, which cannot be
> dynamically loaded and without which a regular kernel would not function.
> Yes, I decided to use the word "faction" to describe the three midlayers
> between rumpkern and the drivers.
>
> The only real problem is the loosy-goosy use of inlines/macros in
> unnecessary places.  But luckily for me, a lot of the work to clean that
> up was done by Andy when he made the x86 ports modular.
>
> > Suppose we have
> > improvements in one part of it, like TCP, IP stacks, could it directly
> > benefit the rumpnet component, or any service sitting above it?
>
> Could you elaborate this question?

Just giving examples:
- suppose that TCPCT support is added in NetBSD's TCP stack. Is it 
trivial to port/adapt/use such improvements within rump kernel and systems?
- extending USB stack (bringing in USBv3, or UVC class) within kernel 
can be "easily" used (or ported) through rump?

On a more general perspective, consider that a specific driver, file 
system, network protocol... is ported within NetBSD. Would that make it 
easy to port to another OS via rump? Does the code has to be written 
following specific guidelines, so it can take advantage from the 
"faction" layer?

> > - What kind of effort would it require to port it to other OS
> > architectures, especially when the API they offer could be a subset of
> > POSIX, or specific low level API (like Xen's hypercalls)? (closely
> > related to the work of Arnaud Ysmall in misc/rump [4])
>
> As you probably know, rump uses the "rumpuser" hypercall interface to
> access the hypervisor (which is currently just a fancy name for userland
> namespace).  It pretty much evolved with the "oh I need this?  ok, I'll
> add it" technique.  But, if someone wants to experiment with minimal
> hosts, I think we can work on rumpuser a bit and see what qualities the
> hosts have in common and what's different.  I don't expect supporting
> rump to be any more difficult than for example Wombat/Iguana L4+UML.
> In fact, it's probably simpler since with rump there is no notion of
> things like address space -- that comes entirely from the host.
>
> Running directly on top of Xen is an interesting idea, but the first
> question I have with that is "why?", i.e. what is the gain as opposed
> running directly in a process on dom0?

- smaller code size.
- smaller components, so you reach higher assurance. Running different 
processes under one general purpose OS represents tons of states.
- having bugs within processes in a dom0 could potentially compromise 
the underlying OS, and as a consequence, the rest of the host, including 
domUs, due to dom0 privileges.

>  The only reason I can think of
> is that you don't trust your dom0 OS enough, but then again you can't
> really trust Xen guests either?

Well, the dom0 should be minimal, for security reasons. In a "perfect" 
world, all devices would be exported to "semi privileged domUs" and 
protected from others with different mechanisms, like IOMMU. The dom0 
would just be an OS running the XenStore, and some simple backends.

This brings nice functionalities too, although you already get them with 
rump: crashing, or restarting, is merely a matter of restarting a domU 
or a process.

>  Besides, rump kernels do not execute
> any privileged instructions, so Xen doesn't sound like the right hammer.

It was just an example; the Xen architecture is sufficiently well known 
around here so I could take it as an example.

> > - If rump could be used both for lightweight virtualization (like rump
> > fs servers), or more heavyweight one (netbsd-usermode...)?
>
> Usermode = rump+more, although paradoxically rump = usermode+more also
> applies (where more != 0 and usermode != rump ;).
>
> I'm no longer really sure why someone would want to use a usermode
> operating system on top of POSIX at all, since scheduling in them seems
> quite inefficient in my experience.  But if someone does, the overlap is
> that lot of the drivers written for rump can be used, for example ugenhc
> which accesses host USB devices or virtif which shovels network traffic
> to/from /dev/tap.  rumpfs could probably be extended into a fullblown
> hostfs, etc ...

Wild guess: from an admin PoV (not a developer), compared to other 
approaches like capabilities, or rump, the containers/zones approach are 
familiar; you "deprivilege" a full blown kernel in an 
implementation-defined namespace protected by the host kernel; then, you 
configure the guest in the same way as you would do it with a 
traditional general OS sitting above.

> > - eventually, extra icing on the cake, especially with the appearance of
> > IOMMU in common x86. Could rump be used for (device) drivers to offer
> > better isolation and cause less harm to the kernel?
>
> I assume you're talking about device drivers which do DMA by themselves,
> since others (e.g. USB, bpf, audio) already work.  Theoretically yes,
> but someone needs to write the code ;)
>
> I touched this issue a little in my AsiaBSDCon paper this year.  The IOMMU
> technique for unprivileged DMA'ing device drivers has been described by
> LeVasseur et. al in their OSDI'04 paper.  I think rump should be able
> to follow the same steps here, albeit possibly with a slightly simpler
> and more lightweight virtual container.

Then I will eagerly monitor that part too ;)

> One of the key points to understand is that rump is only half of the
> solution if you want to isolate a driver used by the host.  You also need
> some microkernel-like server linkback to integrate with the host.  Current
> examples are p2k(3) for file systems or my various proof-of-concept pud(4)
> servers for USB device drivers in src/share/examples/rump.

Ah yes, I completely missed this one; thanks.

--
Jean-Yves Migeon
jeanyves.migeon%free.fr@localhost