Re: Import of XZ

[Matthew Mondor <mm_lists%pulsar-zone.net@localhost>]

On Fri, 6 Aug 2010 08:35:35 +0200
Alan Barrett <apb%cequrux.com@localhost> wrote:

>        To prevent uncomfortable surprises caused by huge memory usage,
>        xz has a built-in memory usage limiter. The default limit is 40 %
>        of total physical RAM.  [...]
> [...]
> The good part is that setting a suitable memory usage limit on
> the compression side should avoid excessive memory usage on the
> decompression side.  I am not sure how I feel about the decompressor
> skipping files that would exceed its memory usage limit; but I suppose
> if I want it to continue regardless of memory usage then I can provide
> enough swap space and use a command line option to override the memory
> limit.

I think that being able to limit RAM usage is a nice thing, but don't
think that this kind of limit (a percentage of the host on which the
file is compressed) is useful to produce distribution archives (which
might be decompressed on a variety of systems, some of which with more
limited resources than expected by the distributor).  Probably that
using an explicit setting is better for distributions (which also seems
to be supported, fortunately)...

This is only an opinion, but personally, with the low cost of bandwidth
and storage I've not needed special purpose compressors (other than for
audio or video codecs, of course).

I've seen in the thread that a potential goal would be to switch to XZ
by default except for particular architectures; but why not the
opposite?  XZ could be used on amd64, the only architecture we can be
pretty confident to normally have the resources to deal with it
gracefully, with impressive compression results.  ppc/alpha64/sparc64
include a wide range of different hosts for the same architecture, and
i386 may even include i486 and i586 hosts.  As for a number of other
supported architectures, they can be embedded or are old.

Isn't availability and portability more important for NetBSD than using
the latest free compressor around?  And do we really want to tune the
compressor settings for every supported architecture we cross-build
(40% of the memory of an amd64 build host cross-compiling for a slower
host seems like a bad idea), still achieving subjective results anyway?
Or should they all default to minimal settings still slightly exceeding
what gzip or bz2 can achieve?

Thanks,
-- 
Matt