Current-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Lightweight virtualization - the rump approach

On 05/14/10 17:29, Antti Kantee wrote:
On Thu May 13 2010 at 18:51:16 +0200, Jean-Yves Migeon wrote:
For this reason, I have a few questions for the ones familiar with rump
technology here, especially:

- the basic architecture; how did/do you achieve such a functionality?
Adding an extra layer of indirection within the kernel?

There's no artificial extra layer of indirection in the code (and,
compared to other virtualization technologies, there's one less at
runtime).  It's mostly about plugging into key places and understanding
what you can use directly from the host (e.g. locking, which is of
course very intertwined with scheduling) and what you really don't want
to be emulating in userspace at all (e.g. virtual memory).  Due to some
magical mystery reason, even code which was written 20 years ago tends
to allow for easy separation.

"Magical mystery reason?" Could you elaborate on that one? Is it due to architecture decisions made in the past to build up operating systems, or is it rather specific to the code base you are working with?

The other part is more or less completing the work on kernel modul
support in NetBSD, mostly minor issues with config left (devsw,
SRCS) and I've got those somewhat done in a corner of my source tree.
Rump components more or less follow the same functional units as kernel
modules apart from the rump{dev,net,vfs} factions, which cannot be
dynamically loaded and without which a regular kernel would not function.
Yes, I decided to use the word "faction" to describe the three midlayers
between rumpkern and the drivers.

The only real problem is the loosy-goosy use of inlines/macros in
unnecessary places.  But luckily for me, a lot of the work to clean that
up was done by Andy when he made the x86 ports modular.

Suppose we have
improvements in one part of it, like TCP, IP stacks, could it directly
benefit the rumpnet component, or any service sitting above it?

Could you elaborate this question?

Just giving examples:
- suppose that TCPCT support is added in NetBSD's TCP stack. Is it trivial to port/adapt/use such improvements within rump kernel and systems? - extending USB stack (bringing in USBv3, or UVC class) within kernel can be "easily" used (or ported) through rump?

On a more general perspective, consider that a specific driver, file system, network protocol... is ported within NetBSD. Would that make it easy to port to another OS via rump? Does the code has to be written following specific guidelines, so it can take advantage from the "faction" layer?

- What kind of effort would it require to port it to other OS
architectures, especially when the API they offer could be a subset of
POSIX, or specific low level API (like Xen's hypercalls)? (closely
related to the work of Arnaud Ysmall in misc/rump [4])

As you probably know, rump uses the "rumpuser" hypercall interface to
access the hypervisor (which is currently just a fancy name for userland
namespace).  It pretty much evolved with the "oh I need this?  ok, I'll
add it" technique.  But, if someone wants to experiment with minimal
hosts, I think we can work on rumpuser a bit and see what qualities the
hosts have in common and what's different.  I don't expect supporting
rump to be any more difficult than for example Wombat/Iguana L4+UML.
In fact, it's probably simpler since with rump there is no notion of
things like address space -- that comes entirely from the host.

Running directly on top of Xen is an interesting idea, but the first
question I have with that is "why?", i.e. what is the gain as opposed
running directly in a process on dom0?

- smaller code size.
- smaller components, so you reach higher assurance. Running different processes under one general purpose OS represents tons of states. - having bugs within processes in a dom0 could potentially compromise the underlying OS, and as a consequence, the rest of the host, including domUs, due to dom0 privileges.

 The only reason I can think of
is that you don't trust your dom0 OS enough, but then again you can't
really trust Xen guests either?

Well, the dom0 should be minimal, for security reasons. In a "perfect" world, all devices would be exported to "semi privileged domUs" and protected from others with different mechanisms, like IOMMU. The dom0 would just be an OS running the XenStore, and some simple backends.

This brings nice functionalities too, although you already get them with rump: crashing, or restarting, is merely a matter of restarting a domU or a process.

 Besides, rump kernels do not execute
any privileged instructions, so Xen doesn't sound like the right hammer.

It was just an example; the Xen architecture is sufficiently well known around here so I could take it as an example.

- If rump could be used both for lightweight virtualization (like rump
fs servers), or more heavyweight one (netbsd-usermode...)?

Usermode = rump+more, although paradoxically rump = usermode+more also
applies (where more != 0 and usermode != rump ;).

I'm no longer really sure why someone would want to use a usermode
operating system on top of POSIX at all, since scheduling in them seems
quite inefficient in my experience.  But if someone does, the overlap is
that lot of the drivers written for rump can be used, for example ugenhc
which accesses host USB devices or virtif which shovels network traffic
to/from /dev/tap.  rumpfs could probably be extended into a fullblown
hostfs, etc ...

Wild guess: from an admin PoV (not a developer), compared to other approaches like capabilities, or rump, the containers/zones approach are familiar; you "deprivilege" a full blown kernel in an implementation-defined namespace protected by the host kernel; then, you configure the guest in the same way as you would do it with a traditional general OS sitting above.

- eventually, extra icing on the cake, especially with the appearance of
IOMMU in common x86. Could rump be used for (device) drivers to offer
better isolation and cause less harm to the kernel?

I assume you're talking about device drivers which do DMA by themselves,
since others (e.g. USB, bpf, audio) already work.  Theoretically yes,
but someone needs to write the code ;)

I touched this issue a little in my AsiaBSDCon paper this year.  The IOMMU
technique for unprivileged DMA'ing device drivers has been described by
LeVasseur et. al in their OSDI'04 paper.  I think rump should be able
to follow the same steps here, albeit possibly with a slightly simpler
and more lightweight virtual container.

Then I will eagerly monitor that part too ;)

One of the key points to understand is that rump is only half of the
solution if you want to isolate a driver used by the host.  You also need
some microkernel-like server linkback to integrate with the host.  Current
examples are p2k(3) for file systems or my various proof-of-concept pud(4)
servers for USB device drivers in src/share/examples/rump.

Ah yes, I completely missed this one; thanks.

Jean-Yves Migeon

Home | Main Index | Thread Index | Old Index