pbulk-index performance improvement

[Jonathan Perkin <jperkin%pkgsrc.org@localhost>]

Hi,

While working on support for full scans in bob, I got annoyed again by 
slow scans in certain directories due to exploding cartesian products.

Here's a diff that in theory avoids this problem for packages that do 
not support multi-version in their PKGNAME:

  https://gist.github.com/jperkin/ce0b765a3a868389b09e235b58105a32

Testing this in parallel/slurm-wlm takes the scan time from:

  $ time bmake pbulk-index >/dev/null
  real	1m32.376s
  user	0m42.333s
  sys	0m36.232s

down to:

  $ time bmake pbulk-index >/dev/null
  real	0m2.027s
  user	0m0.938s
  sys	0m0.796s

and in a full unoptimised scan using bob with 8 sandbox threads from:

  $ bob scan
  Scanned 20165 packages in 1h 17m 49s (20164 succeeded, 1 failed)

down to:

  $ bob scan
  Scanned 20165 packages in 54m 41s (20164 succeeded, 1 failed)

I'm currently working on support for DAG output comparison so that I can 
verify that there are no DAG changes with this in place.

Thanks,

--
Jonathan Perkin                    pkgsrc.smartos.org
Open Source Complete Cloud   www.tritondatacenter.com