tech-pkg archive

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index][Old Index]

Wider testing for wrapper performance patch



Hey all,

This is earlier than I'd normally ask for review of something, but it's a huge win and I'm interested in testing on a wide variety of systems to see what the performance wins are, and if there are any regressions.

One of the slowest parts of the pkgsrc infrastructure is the wrapper phase, where symlinks for all includes, libraries, and ancillary support files are created under "${WRKDIR}/.buildlink/". Each file results in a fork of dirname(1) and ln(1).

When you have packages such as devel/boost-headers which is pulled in by many packages and alone has over 15,000 files to symlink, the terrible performance becomes clear. For example:

  $ cd x11/kde-workspace4
  $ bmake patch; ptime bmake wrapper

  real     3:20.696394375
  user       50.553556463
  sys      2:23.883823896

I'm currently testing an alternative which uses a small program to batch up the symlink creation, and is significantly faster:

  real       19.873100598
  user        8.141441461
  sys        11.740602820

You can grab it and the patch to the infrastructure from here:

  https://github.com/TritonDataCenter/pkgsrc/commit/d119b14357b26ac282c3d5088200cc9986295b6a

It currently lives in the cwrappers package for ease of testing, but I'm not sure this is a good long term home, and maybe we want to create a new "pkg-mk-tools" package or something that can house a few other useful utilities to speed up other areas of the infrastructure.

Cheers,

--
Jonathan Perkin  -  Joyent, Inc.  -  www.joyent.com


Home | Main Index | Thread Index | Old Index