NetBSD-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: NetBSD Jails

At Fri, 15 May 2020 20:18:28 -0400, "Aaron B." <> wrote:
Subject: Re: NetBSD Jails
> - Processes can "see" each other; I have to be careful not to reuse
> UID numbers. For example: if I build a chroot with an instance of nginx
> that runs as UID 2505, and then deploy multiple copies of that chroot,
> all of them can call kill(2) on a process in a different chroot.

Perhaps all that's required is a tool which extracts the minimum
required entries from the real /etc/master.passwd for each chroot?
(and some way to maintain chroot copies?)

(Another way would be a new service behind nsdispatch(3) which would
provide access through the chroot, e.g. via a socket, to the shared
/etc/master.passwd, though that would assume all chrooted programs use
only the "standard" interfaces supported by nsdispatch(3).)

> - All chroots share the same network stack. If I tell nginx to bind to
> '' or '::', the first instance will startup fine, the others
> will fail with "address already in use."

Well if you're chrooting multiple instances of the same service, isn't
it obvious that each has to listen on one and only one specific address?
If I understand correctly one could also route a subnet via a bridge
interface to each chrooted service.  Maybe a chrooted process should
also be prevented from listening to a wildcard address?

I've heard FreeBSD folks go on for days about how FreeBSD's "jails" make
network management simpler, but I still don't have any real
understanding of exactly what this means.  The only thing that seems to
be of interest is in allowing a chrooted "root" user(*) (e.g. to allow
someone inside the jail to muck around with interface addresses, routes,
etc.), but I would suggest that allowing root inside a chroot is a very
very bad idea no matter what "jails" features you might think will
protect you (i.e. I would never trust a chrooted root user, "jails" or
no jails).

(*) "chrooted root user" -- i.e. the "root" user in the jail can only do
things (as superuser ID#0) to those resources, e.g. interfaces, routes,
etc. that are delegated to the jail.

> The wiki's projects list has a
> clean solution to this particular point, which may or may not be within
> scope of jails:

Virtual network stacks seem to be a rather complex solution looking for
a problem -- i.e. in actual fact more of a problem looking for trouble.

> - Some way to set per-chroot resource limits would be helpful. I can
> manipulate ulimits, but that is basically driving screws with a hammer.
> It's simply the wrong tool.

Well here's where /etc/login.conf solves most problems for normal chroot
environments, since only ordinary users should be running inside the

(Or it could, if there were resource controls related to networking. :-))

For anything beyond that I'm pretty certain that a full virtual machine
is the real solution.  Personally I think full VMs are the K.I.S.S. of
death for "containers" (especially once you see the nightmare of
complexity that underlies them in the kernel, i.e. CGroups).  I would
much rather have the clearly structured overhead of the VM instead of
the hidden overhead and excessive complexity of "containers", or even
just some of the bits of them like virtual network stacks.

All this said though I would note that perhaps re-engineering the whole
network stack in the netgraph way (perhaps directly using netgraph[1]),
provides some form of "virtualization" for network things in a clean and
structured way.


One thing I would like to have for making VMs easier to manage though is
a filesystem that can be shared read-only as a block device --
i.e. through xbdback(4) to xbd(4), or the equivalent for other kinds of VMs.
(I say this because I abhor the use of NFS to share to VM domUs.)

					Greg A. Woods <>

Kelowna, BC     +1 250 762-7675           RoboHack <>
Planix, Inc. <>     Avoncote Farms <>

Attachment: pgpiJWgVSMtjn.pgp
Description: OpenPGP Digital Signature

Home | Main Index | Thread Index | Old Index