[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: When substitute download + decompression is CPU-bound
From: |
Pierre Neidhardt |
Subject: |
Re: When substitute download + decompression is CPU-bound |
Date: |
Tue, 15 Dec 2020 13:31:23 +0100 |
Hi Ludo,
Ludovic Courtès <ludo@gnu.org> writes:
> Well, ‘guix publish’ would first need to create multi-member archives,
> right?
Correct, but it's trivial once the bindings have been implemented.
> Also, lzlib (which is what we use) does not implement parallel
> decompression, AIUI.
Yes it does, multi-member archives is a non-optional part of the Lzip
specs, and lzlib implemetns all the specs.
> Even if it did, would we be able to take advantage of it? Currently
> ‘restore-file’ expects to read an archive stream sequentially.
Yes it works, I just tried this:
--8<---------------cut here---------------start------------->8---
cat big-file.lz | plzip -d -o big-file -
--8<---------------cut here---------------end--------------->8---
Decompression happens in parallel.
> Even if I’m wrong :-), decompression speed would at best be doubled on
> multi-core machines (wouldn’t help much on low-end ARM devices), and
> that’s very little compared to the decompression speed achieved by zstd.
Why doubled? If the archive has more than CORE-NUMBER segments, then
the decompression duration can be divided by CORE-NUMBER.
All that said, I think we should have both:
- Parallel lzip support is the easiest to add at this point.
It's the best option for people with low bandwidth. This can benefit
most of the planet I suppose.
- zstd is best for users with high bandwidth (or with slow hardware).
We need to write the necessary bindings though, so it will take a bit
more time.
Then the users can choose which compression they prefer, mostly
depending on their hardware and bandwidth.
--
Pierre Neidhardt
https://ambrevar.xyz/
signature.asc
Description: PGP signature
- When substitute download + decompression is CPU-bound, Ludovic Courtès, 2020/12/14
- Re: When substitute download + decompression is CPU-bound, Julien Lepiller, 2020/12/14
- Re: When substitute download + decompression is CPU-bound, Nicolò Balzarotti, 2020/12/14
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2020/12/15
- Re: When substitute download + decompression is CPU-bound, Nicolò Balzarotti, 2020/12/15
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2020/12/15
- Re: When substitute download + decompression is CPU-bound, Nicolò Balzarotti, 2020/12/15
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2020/12/15
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2020/12/15
- Re: When substitute download + decompression is CPU-bound, Ludovic Courtès, 2020/12/15
- Re: When substitute download + decompression is CPU-bound,
Pierre Neidhardt <=
- Re: When substitute download + decompression is CPU-bound, Ludovic Courtès, 2020/12/18
- Re: When substitute download + decompression is CPU-bound, Pierre Neidhardt, 2020/12/18
Re: When substitute download + decompression is CPU-bound, Ludovic Courtès, 2020/12/15
Re: When substitute download + decompression is CPU-bound, Jonathan Brielmaier, 2020/12/15