kern/57503: nullfs locking contention

[jperkin%pkgsrc.org@localhost]

>Number:         57503
>Category:       kern
>Synopsis:       nullfs locking contention
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    kern-bug-people
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Jul 05 10:15:00 +0000 2023
>Originator:     Jonathan Perkin
>Release:        NetBSD 10.0_BETA (GENERIC) #0: Tue Jun 27 18:56:25 UTC 2023
>Organization:
>Environment:
NetBSD pkgsrc-nb0 10.0_BETA NetBSD 10.0_BETA (GENERIC) #0: Tue Jun 27 18:56:25 UTC 2023  mkrepro%mkrepro.NetBSD.org@localhost:/usr/src/sys/arch/amd64/compile/GENERIC amd64
>Description:
I figured there would already be a ticket about this, but abs suggests that there isn't and requested I raise this one.

mount_null(8) does not appear to scale very well. Attempting to use pkgsrc/pkgtools/mksandbox to create a single chroot and run pbulk scan processes inside it runs incredibly slowly.

Switching to unpacked sets instead as suggested by Joerg and others instantly made everything around 3.5x faster, with more scope for running additional processes and improving performance even further.

Obviously this approach has many drawbacks though, and I'd much rather use null mounts if they had the same performance that loopback/bind mounts have on Linux/illumos/etc.
>How-To-Repeat:
Use mount_null(8) and observe lack of scaling with additional CPUs (my host has 16).
>Fix:
From what I recall when mentioning this on IRC a while back, one of the FreeBSD developers mentioned that they had fixed a number of similar issues in their nullfs, so you may just want to see what they did there.