Current-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Status of NetBSD virtualization roadmap - support jails like features?

Hi Greg,

Am 15.04.2022 um 20:28 schrieb Greg A. Woods:
At Fri, 15 Apr 2022 07:36:15 +0200, Matthias Petermann <> wrote:
Subject: Status of NetBSD virtualization roadmap - support jails like features?

My motivation: I am looking for a particularly high performance
virtualization solution on NetBSD. Especially disk and network IO
plays a role for me.

In my experience nothing beats I/O performance of Xen with LVM in the
dom0 and the best/fastest storage available for the dom0, especially now
there's SMP support for dom0.  That's anecdotal though -- I haven't done
any real comparisons.  I just know that NFS in domUs is a lot slower
than using LVMs via xbd(4), no matter where/how-fast the NFS server is!

If I'm not too far out of touch I think there's still a wee bit more SMP
support needed in the networking code to make it possible for dom0 to
also give the best network throughput, but it's really not horrible as-is.

In theory NVMM with QEMU and virtio(4) should be about the same I would
guess, with potential for some improvement in some micro-benchmarks, but
for production use the maturity and completeness of the provisioning
support offered by Xen still seems far superior to me.

This is interesting to read and also coincides with my experience so far. LVM was also my logical choice at first, but I had some unclear crashes with it in connection with FSS (FFS2 snapshots), which is why I use VNDs as backing store at the moment. The concrete context of that time needs a re-evaluation.

Anyway, the performance drop of LVM vs. VND was not dramatic for me, since the network IO is the limiting factor on the specific platform and the weak CPU equipment (NUC5 with Celeron) could also play a role here. In the meantime, I also have other hardware (i3) available and will do some comparative tests when the opportunity arises, possibly also with NVMM / virtio.

Regardless, I still think it wouldn't hurt
if NetBSD could implement some sort of

I'm not convinced "jails" (at least in the FreeBSD form I'm most
familiar with) actually buy much without also increasing complexity
and/or introducing limitations on both the provisioning and the
"virtual" side.

With a full virtualisation as in Xen the added complexity is very well
partitioned between the provisioning side and the VMs, and there are
almost no limitations inside the VMs (assuming you are virtualising
something that fits well into a virtualised environment, i.e. with no
special direct hardware access needs) -- everything looks and feels and
is managed almost as if it is running on bare hardware and so the
management of the VM is exactly as if it were running on separate
hardware; except of course some aspects are actually easier to manage,
such as provisioning direct console access and control.  There's really
nothing new to learn other than how to spin up a new domU (and possibly
how to use LVM effectively).

From a layering perspective, this is definitely the cleanest option and I fully sympathize with it. However, I also realize that this is only a real option on systems that have enough power and are based on an architecture supported by Xen. At the moment, those are mostly x86 systems.

However FreeBSD-style jails do offer their own form of flexibility that
seems to be worth having available, and it would be nice for jails to be
available on NetBSD as well.  The impact inside the OS (kernel and
userland) is quite high though, and is itself a form of complexity
nightmare all its own, though perhaps not so horrible as Linux "cgroups"
and some other related Linux kernel namespaces are.

Yes, that's right - in one of the papers, if I remember correctly, the effort to find the many places to build in security checks for the jails was explicitly mentioned. But it was also mentioned that this effort is already done in NetBSD for Kauth.

Maybe, a full-fledged jail implementation as in FreeBSD may not even be necessary. I would be interested to see how far one can go here without unreasonable large interventions in the existing structures. If one needs a separate network stack per virtual instance, resource allocation and the greatest possible isolation, Xen is already a very good option and usually with these requirements in mind suitable hardware should be selected.

But there are use cases where Xen feels too much and chroot too little. That's why I would be curious if it would be a good approach to think about a careful expansion of chroots, with a focus on local security. A start could be to prevent the transmission of signals between processes in different chroots. This would have the benefit, for example, that you cannot accidentally kill processes in another chroot if you use the same user id in different chroots. Would something like this be feasible as a secmodel extension, analogous to curtain mode, and also in the right place?

The mapping PID->root directory seems to be easily accessible (cd /proc;ls -lad ./*/root) and maybe can be leveraged for the "is same chroot?"-check...?

Kind regards

Home | Main Index | Thread Index | Old Index