Re: Parallel downloads

[Leo Famulari]

On Wed, Nov 13, 2019 at 11:16:53AM -0500, Mark H Weaver wrote:
> For these reasons, I'm inclined to think that parallel downloads is the
> wrong approach.  If a single download process is not making efficient
> use of the available bandwidth, I'd be more inclined to look carefully
> at why it's failing to do so.  For example, I'm not sure if this is the
> case (and don't have time to look right now), but if the current code
> waits until a NAR has finished downloading before asking for the next
> one, that's an issue that could be fixed by use of HTTP pipelining,
> without multiplying the memory usage.
> 
> What do you think?

I agree that parallel downloads is a kludge to work around the issue of
slow set-up and tear-down of our download code. Pipelining would help a
lot, and we could also profile the relevant Guile code to see if there
are any easy speedups.

This issue was actually discussed a year ago:

https://lists.gnu.org/archive/html/guix-devel/2018-11/msg00148.html

I'll quote Ludo's suggestion from then:

> I’d be in favor of a solution where ‘guix substitute’ is kept alive
> across substitutions (like what happens with ‘guix substitute --query’),
> which would allow it to keep connections alive and thus save the TLS
> handshake and a few extra round trips per download.