[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Parallel (de-)compression
From: |
Ludovic Courtès |
Subject: |
Re: Parallel (de-)compression |
Date: |
Fri, 04 Dec 2015 15:44:38 +0100 |
User-agent: |
Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) |
Andreas Enge <address@hidden> skribis:
> on my relatively slow ARM build machine with relatively fast storage (SSD),
> I notice that often there is an xz process taking 100% of CPU, while there
> is never more than 20MB/s written to disk. For instance, texlive-texmf
> takes a very long time to build and install into the store.
Are you saying that xz-compressing TeX Live to resend it to
hydra.gnu.org is too CPU-intensive?
> Would it make sense to switch to a parallel (de-)compression tool to leverage
> higher numbers of cores? We have pbzip2 already in Guix, which is compatible
> with bzip2.
Bzip2 provides a CPU/compression ratio tradeoff that is not as good as
xz, so I would avoid it.
Another option would be to trade compression ratio for reduced CPU usage
by using, say, ‘xz -2’ or ‘gzip’.
We did something similar in 5ef9d7d to reduce CPU consumption on the
front-end. Usually it’s much less important to reduce CPU consumption
on the build machines, but your experience seems to suggest otherwise.
Thoughts?
Ludo’.