Re: Architecture to reduce download time when pulling multiple packages – historic success with magnet URLs, BTIHs, & Aria2c!

[Josh Marshall]

How long is traditional before I can bump a thread?

On Sun, Oct 15, 2023 at 2:21 PM Josh Marshall
<joshua.r.marshall.1991@gmail.com> wrote:
>
> So it sounds like my first steps are to re-implement the downloads
> using aria2c.  This would affect the minimum base package, no?  Can I
> get some buy-in from maintainers that such changes are acceptable?
>
> On Fri, Oct 13, 2023 at 2:06 PM James R. Haigh (+ML.GNU.Guix
> subaddress) <JRHaigh+ML.GNU.Guix@runbox.com> wrote:
> >
> > Hi Josh,
> >
> > At Z-0400=2023-10-13Fri12:36:01, Josh Marshall sent:
> > > This is to parallelize connections which should never hurt downloading 
> > > but can help.  Mirroring would be parallelizing for providing packages, 
> > > what I want to implement is to parallelize obtaining packages.  Server 
> > > side vs client side.
> >
> >         Please, if you are going to do something like this, please use a 
> > torrent architecture like BitTorrent or GNUnet – I suggest Aria2c as a very 
> > good CLI download backend that can be daemonised and sent instructions over 
> > a socket to add, pause, remove downloads, etc., and it supports magnet URLs 
> > including the existing nontorrent servers (via ‘as’ parameters, iirc.).
> >
> >         I actually implemented this in a local copy of APT Daemon many 
> > years ago (circa 2011), but the change was not accepted upstream to 
> > Launchpad (because I was not on bleeding-edge; I was too slow to keep-up 
> > with the upstream development).  My fork got forgotten about, because to 
> > get the full benefit the server would have had to have added a BitTorrent 
> > Info Hash (BTIH) to the metadata of each package, along with the MD5, 
> > SHA-256, etc. that it already did (not a big ask, really).  That said, 
> > without the full benefit of having the metadata, it did provide immediate 
> > benefit and I used it for many years, not upgrading my Ubuntu 11.04 Natty 
> > Narwhal that I was using back then until I really had to.
> >
> >         The immediate benefit that it provided was exactly as you 
> > described: It allowed parallelisation of nontorrent downloads, be it from 
> > the same server or from multiple mirrors.  Iirc., I achieved this by simply 
> > passing the download list to Aria2c in daemon mode, I think I also 
> > converted all the HTTP URLs to ‘as’ parameters in magnet links, so that 
> > multiple mirrors could be passed using multiple ‘as’ parameters in each 
> > magnet link.  Then I simply relied on Aria2c being amazing at parallelising 
> > everything that I had given it!  I then also implemented progress updates 
> > such that APT Daemon could reflect where Aria2c was up to.
> >
> >         The way I implemented this using Aria2c and magnet URLs meant that 
> > if additional hashes were known, they could be used as well, and so if the 
> > server metadata made the simple addition of adding BTIHs, it allows 
> > swarming to occur, which in-turn would massively reduce load on the central 
> > servers, and allow anyone who want to be a mirror to be a mirror simply by 
> > seeding indefinitely.  A default share ratio of 1.0 means that no user is a 
> > burden on the network, unless they deliberately change that.  Users can 
> > donate to the running costs of the project simply by increasing their share 
> > ratio, which adds another means of contribution that they may find easier 
> > than the others.
> >
> >         Anyone keen to keep old packages online can simply seed them 
> > indefinitely, so this is also really great for archival purposes.  Even if 
> > the central project loses interest in the old packages and deletes them, 
> > anyone else can keep them up.  The hashes ensure that they have not been 
> > tampered with.
> >
> >         There is also a really cool benefit that occurs, or can occur, on a 
> > LAN.  An entire network of computers can all swarm locally with each other, 
> > thus needing each package to only need downloading through the metered last 
> > mile bottleneck from the WAN precisely once – providing that local 
> > broadcasting is supported.  I think this requires Avahi, and I seem to 
> > remember that Aria2c supports this but I can't remember.  I don't ever 
> > remember getting this bit working but also I did not try hard because it 
> > would have required the metadata that I didn't have until after download, 
> > so even if I got it working it would not have been directly useful unless 
> > the APT repositories that I was using would include the BTIHs.
> >
> >         So yeah, loads of great benefits to this architecture, and I 
> > highly-recommend it: convert all existing URLs to magnet links (can be done 
> > client-side as I did; or server-side); optionally add any additional 
> > mirrors as additional ‘as’ parameters (again client-side or server-side); 
> > add ‘btih’ parameters to the magnet links (the BTIH must be included in the 
> > server metadata to get the full benefit of the swarming, but conversion to 
> > magnet link format can be done client-side or server-side); then simply 
> > pass all this to a really good parallelising backend such as Aria2c; then 
> > update any progress data and relay pause, resume, cancel, etc. to the 
> > backend.
> >
> >         One final note, as I am sure that there are a lot of GNUnet fans on 
> > this list, is that I would try Aria2c first to see how well it can work, 
> > and then try GNUnet or whatever else once you have a standard to benchmark 
> > against.  Both are Free Software, so no concern there.  Aria2c is an 
> > all-round download manager CLI that works with or without swarming, i.e. it 
> > is just as good at HTTPS as it is BitTorrent, and can do both at the same 
> > time.  GNUnet has the advantage of working from SHA-256 iirc., which is 
> > generally already included in the metadata of the repositories of various 
> > distributions, but I think it lacks a lot of other features and stability 
> > and ecosystem of alternative backends, compared to the BitTorrent network.
> >
> >         Of course, there is no harm in including other hashes along with 
> > BTIH, to allow people to experiment with alternative backends, while always 
> > ensuring that what works works well.  Another hash that may be useful to 
> > include is the Tiger Tree Hash, which is structurally very similar to BTIH, 
> > but stronger, iirc..
> >
> >         The first thing that the Guix project can do to signal interest in 
> > this architecture is to simply include the BTIH of each package in the 
> > repository metadata.  Be it in magnet URL form or not does not matter 
> > because the client can later convert that as needed.  The important thing 
> > is an authoritative statement in metadata that this version of this package 
> > has this BTIH.  Once that metadata is available, the game is on to 
> > implement swarming support, be it with Aria2c as a backend (as I recommend 
> > at least starting with) or otherwise.
> >
> >         I know that this architecture works well out of first-hand 
> > experience with APT Daemon written in Python.  The only failure I had with 
> > it was lack of upstream support.  So I consider it important to first 
> > attain the upstream approval before really investing more time into this.  
> > I seem to remember suggesting this to the Nix project many years ago and 
> > didn't get anywhere, and now I don't have the energy to try to improve 
> > upstream projects if they reject my ideas, so I'll be interested to see 
> > whether you have any success with your attempt to do the same.
> >
> >         Good luck! ;-)
> >
> > Kind regards,
> > James.
> > --
> > Wealth doesn't bring happiness, but poverty brings sadness.
> > Sent from Debian with Claws Mail, using email subaddressing as an 
> > alternative to error-prone heuristical spam filtering.
> > Postal: James R. Haigh, Middle Farm, Vennington, nr. Westbury, nr. 
> > Shrewsbury, Salop, SY5 9RG, Britain