Subject: Looking for some assistance with bulk scripts
To: None <tech-pkg@netbsd.org>
From: Florian Heigl <florian.heigl@gmail.com>
List: tech-pkg
Date: 06/05/2007 17:46:34
Hi,

I tried a first bulk build on HP-UX back in April, with not too much
success, but I will have time to resume on that during the next weeks.
I should add that, even with X support missing back then I build
around 800 packages simply by trying to build everything in sight. :>

I remember a few little issues:

The usage of bulk cache (.dependstree and .index):
1. the tsort call bails because the tsort on HP-UX does not know a -q
flag, and I've not found the actual call in the bulk scripts so I
could remove -q; from what i understood it simply means 'quiet'
2. if i could remove it (i know from manually testing) the tsort will
output a comment like 'tsort: odd data' I do not really understand,
but i suppose the return code won't be 0 which might add some trouble.
The easiest way would be using a NetBSD source tsort, but I don't
really know where to find it.
3. if possible i want to preserve those .files as every run of
creating the cache takes roughly 16 hours. (it's about 12000 bmake
calls after all), but from what i saw in the documentation i can
either not use the cache (even slower in the long run) or regenerate
it (see before)
4. personally I think it would be helpful if the pkgsrc cvs would
contain a daily generated set of these cache files. sure there are
purposes, where this won't be sufficient, but if generating the cache
takes almost a day on a moderately sized host there is not much to
lose in my opinion.

I really need ANY advice to cut down the times per test to something
like 30 minutes :)
I've done all I can to tune the underlying system, but there's not
really anything left I could do.

- I tried to work with a subset of SITE_PKGS (or something like that)
for testing, i.e. to say the buildchain works once quite stable when
www/apache can be built.
when testing, it appeared that dependencies like bzip would be
noticed, but not built early on in the bulk phase.
for that reason i had added them to BULK_PREREQS, in that case they
were built on time but removed later on and not added back.
This was a bit weird and i'd plead for someone experienced with bulk
builds on non-NetBSD OSses look at my logs.

- also i'm looking for the correct way to integrate ccache from pkgsrc
as soon as possible.
- Later on I also would like to add distcc from pkgsrc into the
buildchain as I can't add more raw cpus but have two more systems with
a gigabit backend just for buildpurposes.

Maybe some of the Tru64/Solaris people still have some notes from
their bulk builds I could look at?
On NetBSD or even Linux most of these issues won't show due to the
more 'expected' set of utilities in the base operating system.

Florian

-- 
'Sie brauchen sich um Ihre Zukunft keine Gedanken zu machen'