Re: Status of NetBSD virtualization roadmap - support jails like features?

[Matthias Petermann <mp%petermann-it.de@localhost>]

Hi Greg,

Am 15.04.2022 um 20:28 schrieb Greg A. Woods:
> At Fri, 15 Apr 2022 07:36:15 +0200, Matthias Petermann <mp%petermann-it.de@localhost> wrote:
> Subject: Status of NetBSD virtualization roadmap - support jails like features?
> > My motivation: I am looking for a particularly high performance
> > virtualization solution on NetBSD. Especially disk and network IO
> > plays a role for me.
>
> In my experience nothing beats I/O performance of Xen with LVM in the
> dom0 and the best/fastest storage available for the dom0, especially now
> there's SMP support for dom0.  That's anecdotal though -- I haven't done
> any real comparisons.  I just know that NFS in domUs is a lot slower
> than using LVMs via xbd(4), no matter where/how-fast the NFS server is!
>
> If I'm not too far out of touch I think there's still a wee bit more SMP
> support needed in the networking code to make it possible for dom0 to
> also give the best network throughput, but it's really not horrible as-is.
>
>
> In theory NVMM with QEMU and virtio(4) should be about the same I would
> guess, with potential for some improvement in some micro-benchmarks, but
> for production use the maturity and completeness of the provisioning
> support offered by Xen still seems far superior to me.

This is interesting to read and also coincides with my experience so 
far. LVM was also my logical choice at first, but I had some unclear 
crashes with it in connection with FSS (FFS2 snapshots), which is why I 
use VNDs as backing store at the moment. The concrete context of that 
time needs a re-evaluation.

Anyway, the performance drop of LVM vs. VND was not dramatic for me, 
since the network IO is the limiting factor on the specific platform and 
the weak CPU equipment (NUC5 with Celeron) could also play a role here. 
In the meantime, I also have other hardware (i3) available and will do 
some comparative tests when the opportunity arises, possibly also with 
NVMM / virtio.

> > Regardless, I still think it wouldn't hurt
> > if NetBSD could implement some sort of
> > jail.
>
> I'm not convinced "jails" (at least in the FreeBSD form I'm most
> familiar with) actually buy much without also increasing complexity
> and/or introducing limitations on both the provisioning and the
> "virtual" side.
>
> With a full virtualisation as in Xen the added complexity is very well
> partitioned between the provisioning side and the VMs, and there are
> almost no limitations inside the VMs (assuming you are virtualising
> something that fits well into a virtualised environment, i.e. with no
> special direct hardware access needs) -- everything looks and feels and
> is managed almost as if it is running on bare hardware and so the
> management of the VM is exactly as if it were running on separate
> hardware; except of course some aspects are actually easier to manage,
> such as provisioning direct console access and control.  There's really
> nothing new to learn other than how to spin up a new domU (and possibly
> how to use LVM effectively).

From a layering perspective, this is definitely the cleanest option and 
I fully sympathize with it. However, I also realize that this is only a 
real option on systems that have enough power and are based on an 
architecture supported by Xen. At the moment, those are mostly x86 systems.

> However FreeBSD-style jails do offer their own form of flexibility that
> seems to be worth having available, and it would be nice for jails to be
> available on NetBSD as well.  The impact inside the OS (kernel and
> userland) is quite high though, and is itself a form of complexity
> nightmare all its own, though perhaps not so horrible as Linux "cgroups"
> and some other related Linux kernel namespaces are.

Yes, that's right - in one of the papers, if I remember correctly, the 
effort to find the many places to build in security checks for the jails 
was explicitly mentioned. But it was also mentioned that this effort is 
already done in NetBSD for Kauth.

Maybe, a full-fledged jail implementation as in FreeBSD may not even be 
necessary. I would be interested to see how far one can go here without 
unreasonable large interventions in the existing structures. If one 
needs a separate network stack per virtual instance, resource allocation 
and the greatest possible isolation, Xen is already a very good option 
and usually with these requirements in mind suitable hardware should be 
selected.

But there are use cases where Xen feels too much and chroot too little. 
That's why I would be curious if it would be a good approach to think 
about a careful expansion of chroots, with a focus on local security. A 
start could be to prevent the transmission of signals between processes 
in different chroots. This would have the benefit, for example, that you 
cannot accidentally kill processes in another chroot if you use the same 
user id in different chroots. Would something like this be feasible as a 
secmodel extension, analogous to curtain mode, and also in the right place?

The mapping PID->root directory seems to be easily accessible (cd 
/proc;ls -lad ./*/root) and maybe can be leveraged for the "is same 
chroot?"-check...?

Kind regards
Matthias