Re: rust-team branch merged

[Jason Conroy]

Efraim Flashner <efraim@flashner.co.il> writes:

> [[PGP Signed Part:Undecided]]
> On Wed, Apr 24, 2024 at 11:58:22AM -0400, Jason Conroy wrote:
> > Efraim Flashner <efraim@flashner.co.il> writes:
> > > On the other hand, by generating it during the build of each 
> > > package we
> > > make sure to pull in all the crates which exist in the build, 
> > > so we
> > > could add into a profile/manifest just the crates listed in a 
> > > Cargo.toml
> > > and then each crate would pull in its own dependencies, and 
> > > then the
> > > profile hook could combine them all together.
> >
> > Thanks. Just to make sure I understand: it sounds like you're 
> > saying that by
> > creating the JSON index files up front, we'd be preserving some 
> > knowledge
> > about a package's dependency graph that isn't easily recovered 
> > later by
> > recursively walking through inputs and cargo-inputs for the 
> > package specs in
> > a manifest/profile?
>
> If we create it upfront it may be easier to keep track of the
> cargo-dependencies, but I suppose it would mostly depend on the
> implementation.  I'm not sure that'd actually be the case, so 
> don't
> worry about it too much.  Whatever we end up with will be better 
> than
> what we have now, and we can always make it better later if it 
> needs to
> change.

Hi Efraim, I spent some time on this and have a few thoughts 
below. In short, I made progress but also hit some roadblocks that 
give me second thoughts about the approach we discussed up-thread.

As you suggested, I kept the metadata generation code in 
cargo-build-system and used the cargo-registry profile hook to 
merge all of those package artifacts into index files. I also 
modified the hook to lift the full closure of source dependencies 
into the profile for any rust packages in the manifest, borrowing 
logic from `(guix build-system cargo)`.

However, testing revealed some issues with this approach:

1) Formerly, for any cargo package with `#:skip-build? #t` the 
crate was treated as an opaque file: it was copied to a location 
where cargo could access it, but otherwise unprocessed. But to 
generate the index file we need to unpack the crate and call 
`cargo metadata`, which performs some sanity checks on the package 
structure. The implication is that if #:skip-build? is used for 
reasons other than performance - e.g., for a package that works 
fine as a dependency but somehow fails during a standalone build - 
then we might have trouble including such a package in Cargo's 
local registry. I don't know how big an issue this is in practice, 
but I think I encountered one package that fits this scenario: 
custom_derive-0.1.7. (We can discuss the details in a separate 
thread if you're interested.)

2) When building against a local-registry, it seems like Cargo 
expects to find crates that it wouldn't for a vendor tree. As 
mentioned above, my current patch pulls source crates into the 
profile using the same logic as cargo-build-system currently does: 
for each manifest package and its cargo-development-inputs, 
recursively walk through all of the cargo-inputs. But after 
building a profile this way, I encountered the following:

 $ cargo build
 error: no matching package named `automod` found
 location searched: registry `crates-io`
 required by package `serde_json v1.0.111`
     ... which satisfies dependency `serde_json = "^1.0"` of 
     package `rand_chacha v0.3.1`
         ... which satisfies dependency `rand_chacha = "^0.3.0"` 
         of package `rand v0.8.5`
             ... which satisfies dependency `rand = "^0.8.5"` of 
             package `mytest v0.1.0 (/home/guix/mytest)`

Note that serde_json is actually a dev-dependency of rand_chacha, 
and automod is a dev-dependency of serde_json, but our algorithm 
above assumes that dev-dependencies of dependencies can be 
omitted. So in practice, it seems like building a local-registry 
that cargo will accept is even more expensive than building a 
vendor tree.

3) The build error above is interesting for another reason: cargo 
simply gives up rather than fetching the missing dependencies. In 
our earlier discussion, the motivation for using a local-registry 
over a vendor tree in a profile was to support a sort of hybrid 
workflow: the profile seeds the registry with some packages, but 
cargo can still download others on its own during `cargo 
build`. But in this test and several others, I haven't observed a 
case where cargo falls back to crates.io for missing 
packages. Efraim, could you share details on how you accomplished 
this in the past? I wonder if our discussion might have conflated 
cargo's registry cache ($CARGO_HOME/registry) with a true 
local-registry; I haven't found any good documentation on the 
former, but it has a different directory layout and index file 
format from the local-registry's, so it wouldn't surprise me if it 
had different semantics as well.

Cheers,

Jason