Import of XZ

[Joerg Sonnenberger <joerg%britannica.bec.de@localhost>]

Hi all,
I want to import XZ for use in pkgsrc and possible for release
distribution. XZ is the successor of LZMA. It uses the same core
algorithm, but fixes the file format to be sensible (like having a
proper magic number etc). The code is mostly public domain, a few pieces
that are not relevant for our purpose (and won't be imported) like the
zdiff alternatives are GPL3.

LZMA as algorithm is essentially LZW with huge window size and better
entropy encoding. Compression time is higher than both gzip and bzip2,
decompression time is around 3-4 times of gzip. This is mostly related
to the entropy coder and less cache friendly memory window.

Compression efficency is much better for typical data. base and comp as
the two main sets shrink from 37MB / 67MB to 23MB / 34MB. Overall, the
non-xsrc sets shrink from 156MB to 89MB for amd64. For binary packages,
between 30% and 40% are seen on average too. This makes it very
attractive for any data decompressed many times and transfered over the
network.

liblzma itself is around 140KB on amd64. An LZMA decoder can fit into
less than 8KB binary code for the same platform.

Joerg