[bug#33899] [PATCH 0/5] Distributing substitutes over IPFS

[Hector Sanjuan]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, December 29, 2018 12:12 AM, Ludovic Courtès <address@hidden> wrote:

> Hello Guix!
>
> Here is a first draft adding support to distribute and retrieve substitutes
> over IPFS. This builds on discussions at the R-B Summit with Héctor Sanjuan
> of IPFS, lewo of Nix, Pierre Neidhardt, and also on the work Florian
> Paul Schmidt posted on guix-devel last month.
>
> The IPFS daemon exposes an HTTP API and the (guix ipfs) module provides
> bindings to a subset of that API. This module also implements a custom
> “directory” format to store directory trees in IPFS (IPFS already provides
> “UnixFS” and “tar” but they store too many or too few file attributes.)
>
> ‘guix publish’ and ‘guix substitute’ use (guix ipfs) to
> store and retrieve store items. Complete directory trees are stored in
> IPFS “as is”, rather than as compressed archives (nars). This allows for
> deduplication in IPFS. ‘guix publish’ adds a new “IPFS” field in
> narinfos and ‘guix substitute’ can then query those objects over IPFS.
> So the idea is that you still get narinfos over HTTP(S), and then you
> have the option of downloading substitutes over IPFS.
>
> I’ve pushed these patches in ‘wip-ipfs-substitutes’. This is rough on the
> edges and probably buggy, but the adventurous among us might want to give
> it a spin. :-)
>
> Thanks,
> Ludo’.


Hey! Happy new year! This is great news. I'm very glad to see this.
I haven't tried this yet but looking at the code there are a couple
of things to point out.

1) The doc strings usually refer to the IPFS HTTP API as GATEWAY. go-ipfs
has a read/write API (on :5001) and a read-only API that we call "gateway"
and which runs on :8080. The gateway, apart from handling most of the
read-only methods from the HTTP API, also handles paths like "/ipfs/<cid>"
or "/ipns/<name>" gracefully, and returns an autogenerated webpage for
directory-type CIDs. The gateway does not allow to "publish". Therefore I think
the doc strings should say "IPFS daemon API" rather than "GATEWAY".

2) I'm not proficient enough in schema to grasp the details of the
"directory" format. If I understand it right, you keep a separate manifest
object listing the directory structure, the contents and the executable bit
for each. Thus, when adding a store item you add all the files separately and
this manifest. And when retrieving a store item you fetch the manifest and
reconstruct the tree by fetching the contents in it (and applying the
executable flag). Is this correct? This works, but it can be improved:

You can add all the files/folders in a single request. If I'm
reading it right, now each files is added separately (and gets pinned
separately). It would probably make sense to add it all in a single request,
letting IPFS to store the directory structure as "unixfs". You can
additionally add the sexp file with the dir-structure and executable flags
as an extra file to the root folder. This would allow to fetch the whole thing
with a single request too /api/v0/get?arg=<hash>. And to pin a single hash
recursively (and not each separately). After getting the whole thing, you
will need to chmod +x things accordingly.

It will probably need some trial an error to get the multi-part right
to upload all in a single request. The Go code HTTP Clients doing
this can be found at:

https://github.com/ipfs/go-ipfs-files/blob/master/multifilereader.go#L96

As you see, a directory part in the multipart will have the content-type Header
set to "application/x-directory". The best way to see how "abspath" etc is set
is probably to sniff an `ipfs add -r <testfolder>` operation (localhost:5001).

Once UnixFSv2 lands, you will be in a position to just drop the sexp file
altogether.

Let me know if you have any doubts, I'll make my best to answer them. In the
meantime I'll try to get more familiar with Guix.

Cheers,

Hector

PS. There is a place where it says "ifps" instead of "ipfs". A very common typo.