Re: [Bug-wget] [RFC] Extend concurrency support

[Ángel González]

On 20/05/14 01:23, Jure Grabnar wrote:
> 2) Try to figure out file size with Content-Length header. If size is
> unknown fallback to a single thread download. Would it be sensible to
> allow user to specify file size with some switch?
Probably. Sometimes it's available from the download page even if the
server does not provide it.

> 3) The main thread maintains a pool of available servers. It spawns at
> most N threads if N<  M or at most M threads if M<  N, where M is
> number of available mirrors. Every thread downloads each own chunk from
> each own mirror using current implementation of concurrent download
> for Metalink. If some mirror becomes unavailable during download from
> i-th thread, that threads terminates and notifies the main thread. The
> main thread spawns a new thread from available mirrors; if none is
> available at the moment, it waits until some mirror becomes available
> (whenever some other thread finishes downloading its chunk).
What should happen if there's no mirror left?



> 4) A file would be downloaded to a single temporary file as described
> here: http://lists.gnu.org/archive/html/bug-wget/2014-05/msg00025.html
> I'm still fixing the patch, because at least one memory corruption bug
> is still lurking around which is yet to be found.
Try using valgrind. I find it invaluable for these kind of bugs.


> b) Front end
> What would be a good way to specify mirror list? Specifying a switch
> and listing all mirrors could be quite awkward. Should we introduce
> some sort of a simple file format?
A plain text file with one url per line?

> I believe we should take into consideration number 2: downloading
> multiple files from multiple servers. Do we want to apply different
> switches (options) to different files?
> What about if we want to combine 1. and 2.: multiple files from multiple
> mirror list? The simplest way would be to use Metalink file for such
> purpose but is it the most elegant?
>
> All your suggestions are greatly appreciated.
>
> Best Regards,
>
> Jure Grabnar