pkgsrc-Users archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Re: Pbulk scan phase acceleration with tmpfs?

On Sat, Apr 10, 2021 at 10:51:49AM +0100, Jonathan Perkin wrote:
> * On 2021-04-10 at 10:37 BST, Matthias Petermann wrote:
> > So what came to my mind - would it make a significant difference if I
> > keep the pkgsrc structure of the sandboxes ready in the host's tmpfs and
> > mount it to the sandbox? Then this filesystem would be served quasi from
> > RAM - of which there would also be enough available. Does anyone have
> > experience with this?
> The two main things that will help scan speed are:
>  * Increased concurrency through additional scan processes running
>    across multiple hosts and chroots.  One per CPU is a good start.
>  * Enabling PBULK_CACHE_DIRECTORY (usually to something like /var/tmp in
> each chroot, as it must be empty prior to each run).
> Any other changes usually make negligible changes to scan times, whereas
> increased concurrency will, up to a point (the fastest I managed was around
> 5 minutes for a full scan) decrease times linearly.

At least in the NetBSD case, it can help quite a bit to have a separate
copy of root in tmpfs for a set of scan processes (and avoid nullfs)
to reduce the lock contention. Essentially, watch the system time and if
it is much higher than 30%, consider checking deeper with lockstat and


Home | Main Index | Thread Index | Old Index