Re: State of pkgsrc

[Thomas Klausner <wiz%gatalith.at@localhost>]

On Sat, Mar 08, 2025 at 03:28:36PM +0100, Dr. Thomas Orgis wrote:
> We run pkgsrc in our HPC environment and the time-consuming process of
> fixing up broken things even with quarterly releases has prevented me
> from giving timely updates to users. Twice a year would be a goal.
> 
> Compute resources are not the issue for me, as running some builds is
> small scale compared to other computing jobs, and using the university
> resources is justified as it is about ensuring working software for our
> users.
> 
> I guess it makes sense that I participate in future efforts for CI
> builds on our HPC systems (>100 nodes, 192 cores each). This would be
> Linux x86-64, NetBSD maybe via qemu VMs, I guess. I intend to test my
> own build process with our external toolchains, but I should be able to
> set up some automated builds as jobs in our batch system. Personal time
> is the main constraint. When I have a place to push results to, I hope
> the project could benefit, too.
> 
> What's the current estimate for a bulk build on how many CPU cores …
> and how far does it scale in parallel? I wonder at what interval I
> could schedule full builds.

I think full native bulk builds on NetBSD take a couple of days, less
than a week. For example, the 2024Q4 branch was cut on 2024-12-24 and
the first bulk build results on pkgsrc-bulk arrived on the 27th
(though I'm not sure if they were pre-seeded with the packages from
HEAD). The first SmartOS build I see had a supposed build time from
about 2,5 days.

So if you have a lot of computing resources, having timely bulk build
results is very possible.

I think most of the time is spent on

- disk (for the small packages - installing and deleting all
  dependencies), so working on tmpfs is a good idea if you have the
  memory

- CPU (for the big packages like libreoffice, firefox, chromium) which
  are CPU bound

Except for the big packages, it's very parallelizable.

I'd appreciate it if you could set up regular Linux bulk builds in
your environment and send the results to pkgsrc-bulk, so we can keep
track of the current state. NetBSD bulk builds are already covered by
TNF.
 Thomas