Re: Improving 'bmake wrapper' speed

[Jonathan Perkin <jperkin%joyent.com@localhost>]

* On 2014-08-22 at 14:32 BST, Thomas Klausner wrote:

> On Fri, Aug 22, 2014 at 02:18:42PM +0100, Jonathan Perkin wrote:
> > I have a proposed patch which significantly reduces the number of
> > execs we need to perform:
> > 
> >   http://us-east.manta.joyent.com/jperkin/public/patches/wrapper-perf.diff
> 
> These are impressive numbers.
> 
> Looking at the patch, it consists mostly of changes like these:
> 
> -            $cat << EOF
> -s|\([$_sep]\)$1\([$_sep]\)|\1$2\2|g
> -s|\([$_sep]\)$1\([$_sep]\)|\1$2\2|g
> -s|\([$_sep]\)$1$|\1$2|g
> -s|^$1\([$_sep]\)|$2\1|g
> -s|^$1$|$2|g
> -EOF
> +                     echo "s|\([$_sep]\)$1\([$_sep]\)|\1$2\2|g"
> +                     echo "s|\([$_sep]\)$1\([$_sep]\)|\1$2\2|g"
> +                     echo "s|\([$_sep]\)$1$|\1$2|g"
> +                     echo "s|^$1\([$_sep]\)|$2\1|g"
> +                     echo "s|^$1$|$2|g"

For clarity, the changes are:

 - use ${file%/*} instead of `dirname $file`, this doesn't handle all
   cases that dirname does, e.g. if file doesn't contain any "/"s, but
   as all ours should be full paths then I think this is ok.

 - test for file's existence before calling rm(1) on it - it is highly
   likely in this case that the file does not exist, so we save
   thousands of unnecessary execs of rm(1).

 - replace cat <<EOF with echo, again avoiding an exec.

The first two provide the bulk of the speedup (2x), the cat->echo
changes give us an extra 60% or so on top.

> The only I thing I wonder about this is if there is a quoting
> difference between the two versions that might be relevant for pkgsrc,
> or not.

Possibly, hopefully the full bulk build will shake out any issues with
this patch.

-- 
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com