Re: [Bug-wget] [RFC] Extend concurrency support

[Daniel Stenberg]

On Tue, 20 May 2014, Tim Ruehsen wrote:

> > Not sure what other people think about it, but I think wget2, whatever it 
> > will be, should be based on libcurl and focus the wget development on what 
> > wget does better, eg recursive downloads.
>
> Libcurl is one option (and not the worst). At least it would replace the 
> HTTP and FTP send and receive (plus the underlying TCP network handling - 
> what about DNS caching ?). This is just a small amount of Wget's code to 
> replace.

I'll start my response by clearly spell out that I am the libcurl maintainer 
and primary developer. I'm biased as hell.

FTP, HTTP, TLS and sockets seem to be about 25% of wget code (very roughly 
counted). Replacing the network layers with libcurl would not remove those 25% 
completely, as I assume there would have to be adaptions and stuff written 
anyway. Also, not everything would be provide exactly in way wget would prefer 
it since the designs are quite different.

The benefit would not just be less code in wget. libcurl offers a substantial 
amount of more functionality in the network layer than what wget has. And yes, 
libcurl has a DNS cache.

In the past I've been told arguments such that the licensing of libcurl and 
the fact that libcurl is not a GNU project to be blocking reasons against 
using it in wget. I don't know if there's any rules or guidelines that 
actually dictate this, but those libcurl facts won't change.

In my eyes, the biggest drawback with switching to libcurl is that it'll 
require quite a big ripping-out-and-replacing-the-carpet-beneath-us operation, 
and one that'll require one or more dedicated contributors to do some heavy 
lifting to get everything on track.

> - Cookie logic (incl. public suffix handling)

libcurl also provides cookie support.

> - threading abstraction API

libcurl offers parallel transfers without the use of threads so wget actually 
wouldn't need threads if based on libcurl. At least not for that reason.

All this said, I won't be the person who'd do the big part of the work so I 
won't tell you which way to go. Of course I'll respect whatever decision 
that's made and I'll help out with my little bits whichever the decision is 
and however the future turns.

--

 / daniel.haxx.se