Re: revisiting the xz-only distribution idea

[Pádraig Brady]

On 20/09/16 17:00, Eric Blake wrote:
> http://www.nongnu.org/lzip/xz_inadequate.html is an interesting read why
> xz is an unsafe data format for long-time archival (bit-flips are
> particularly disastrous to recover from), and argues that lzip may be a
> better compression format.  Meanwhile, the recently-released open-source
> Zstd compression seems like it can outperform several other compression
> formats; http://facebook.github.io/zstd/.  Should we be revisiting
> coreutils' xz-only distribution policy?

Given software release archives are generally copied many times
(i.e. mirrored to various places), and independently cryptographically
verified with gpg signatures, bit flips probably aren't a big issue?
I guess an issue with the popularity of xz for software distribution,
is that people may consequently use it for other applications without much 
consideration.

Where zstd really wins is in the trade-off between time and compression ratio,
so compressed once, medium sized, widely distributed packages
is not the ideal use case for zstd.  That said it's more tuneable
and approaches the compression ratio even for that case.

As a matter of interest I used the proposed fedora zstd package
to compress the coreutils archive:
https://bugzilla.redhat.com/show_bug.cgi?id=1373218

  $ find -type f -printf '%8s %f\n'
  50585600 coreutils-8.24.tar
   5629004 coreutils-8.24.tar.xz.9
  10370559 coreutils-8.24.tar.zst.3
   6455668 coreutils-8.24.tar.zst.19
   5825204 coreutils-8.24.tar.zst.22

There is less work for the CPU to do on decompression also:
  $ time xz -cd coreutils-8.24.tar.xz.9 >/dev/null
  real  0m0.717s
  $ time zstd -cd coreutils-8.24.tar.zst.22 >/dev/null
  real  0m0.148s

cheers,
Pádraig