Subject: a small step towards parallel package building....
To: None <tech-pkg@netbsd.org>
From: Greg Oster <oster@cs.usask.ca>
List: tech-pkg
Date: 03/21/2007 17:16:09
This is a multipart MIME message.

--==_Exmh_1174514967_18090
Content-Type: text/plain; charset=us-ascii


Hi All

Here is a little script that uses 'pkg_chk -p' to create a Makefile
that contains all the dependencies among all the packages.  This means
one can do a 'make -j 16' (or whatever) with the resulting Makefile, 
and it will happily build individual packages in parallel, and with 
the correct ordering (no locks needed, no package gets built more 
than once).

Where this is most useful for me (so far) is for rebuilding all 
packages on an SMP box after upgrading to a newer -current...  

Thanks to jlam@ and dsl@ for pointers to relevant pkgsrc and 'make' 
configuration bits.

Feel free to use (or not) this code however you wish :)

Later...

Greg Oster



--==_Exmh_1174514967_18090
Content-Type: text/plain ; name="genpkgmakefile.pl"; charset=us-ascii
Content-Description: genpkgmakefile.pl

#!/usr/pkg/bin/perl

# A little program to facilitate the parallel building of pkgsrc packages
# 
# by Greg Oster <oster@netbsd.org>
# 
# This program is in the Public Domain and you may do whatever you
# like with it except blame me if you have problems with it.
# 
# To use:
# 1) Create a pkg_chk.conf file appropriate for your machine
# 2) Create a directory (e.g. 'bulk') and in that directory run:
#      genpkgmakefile.pl pkg_chk.conf > Makefile
# 3) Run make:
#      make -j 2 
# 4) Sit back and wait for your packages to build.


if ($#ARGV!=0) {
    printf("Usage: genpkgmakefile.pl /path/to/pkg_chk_config_file > Makefile\n");
}

$conf = $ARGV[0];
$pkgsrc_base = "/usr/pkgsrc/";
@todo = `/usr/pkg/sbin/pkg_chk -p -C $conf`;

$count = 0;
$done = 0;
while ($pkg = pop(@todo)) {
    chop($pkg);
    if (!$visited{$pkg}) {
	$count++;
	$name{$count} = $pkg;
	$visited{$pkg} = $count;
	
	# grab the list of dependencies.
	printf(STDERR "Checking deps for $pkg\n");
	@deps = `cd $pkgsrc_base/$pkg && make show-depends-pkgpaths`;
	foreach $dep (@deps) {
	    push(@todo,$dep);
	    chop($dep);
	    $graph{$pkg}{$dep} = 1;
	}
    }
}

# print a convenient "KEY" list of all targets and the package
# to which they correspond

printf("# KEY: \n");
for($i=1;$i<=$count;$i++) {
    printf("# $i : $name{$i} \n");
}
printf("\n");

# dump out the "ALL: " target

printf("ALL: ");
for($i=1;$i<=$count;$i++) {
    printf(" \\\n\tt$visited{$name{$i}}");
}
printf("\n\n");

# and now each package dependency:

foreach $pkg (sort keys %visited) {
    printf("# $pkg\n");
    printf("t$visited{$pkg}: "); 
    @reqs = sort(keys %{$graph{$pkg}});
	foreach $req (@reqs) {
	    printf("t$visited{$req} ");
	}
	printf("\n");
	printf("\t(unset MAKEFLAGS \\\n\t && cd $pkgsrc_base$pkg \\\n\t && make -B package update clean \\\n\t) && touch t$visited{$pkg}\n\n");
}


--==_Exmh_1174514967_18090--