Re: [Mingw-cross-env-list] On creating packages, source files and checksums.

[Tony Theodore]

Apologies for the belated reply… and extra quoting for context

> On 11 Aug 2019, at 19:42, Volker Diels-Grabsch <address@hidden> wrote:
> 
> Dear Gregorio,
> 
> If I haven't missed a new feature in the MXE core, there's currently
> no way to have multiple source file per package.

Over the years, I’ve come to understand why package managers end up
with `*-dev` packages - you typically want *less* output in each
package, not more.

We now use source files across multiple packages[1]. I can’t see that
trend reversing.

> I believe in the past we solved similar issues as follows: Provide one
> dummy-package per source file, so they are downloaded and checked. The
> last ("main") package then depends on all others and performs the
> actual build(s).
> 
> Alas, I wasn't able to find a concrete example of that approach at a
> first glance.  But maybe you do.  However, it is also possible that
> all packages which previously used the above approach were replaced by
> separate builds which were percieved to be "cleaner".
> 
> If you don't want to go that route, I'd sugest to go your first
> approach: I don't see a huge problem in keeping bootstrap compilers
> around, as long as their binariy names are properly and consistently
> prefixed.

Have a look at protobuf[2]. It has a few salient features:
  - depends on source-only packages (google[mock/test])
  - unpacks those into its build tree
  - bootstraps native `protoc` installed in `$(PREFIX)/$(BUILD)/bin/`
  - uses that native `protoc` for cross-build

The bootstrap compiler is only visible to mxe in `$(PREFIX)/$(BUILD)/bin/`
and needs to be kept for subsequent cross-builds.

VTK[3] is a more extreme example that actually needs artefacts from its 
build tree to persist into the cross-build. We have to build in-situ
and keep `$(PREFIX)/$(BUILD)/vtkCompileTools` around.

> Gregorio Litenstein schrieb:
> […]
>> 
>> The reason I'm asking is because I'm trying to port rust; the problem is
>> rust requires a bootstrap compiler that cannot be built from scratch, we
>> _need_ to either download it or provide it somehow.

For mxe, we would prefer to build the bootstrap compiler. You can have multiple 
levels of bootstrapping with the current functionality, but I’m not familiar
with `rust` to provide any tips. 

>> One possible solution
>> of course would be to have a package for each bootstrap component (there's
>> 3; rustc, rust-std and cargo). That would probably work but would mean
>> keeping bootstrap compilers around on the system after building the real
>> compiler so it's far from ideal.

That’s not a problem (see above).

>> Another solution maybe would be to make a single package containing both
>> the source and the bootstrap binaries and host it somewhere; this is also
>> not ideal because there would have to be one for each different OS
>> supported by MXE.

An alternate would be to add extra requirements, but that has many issues
of its own.

>> The ideal solution of course, would be to be able to have more than one
>> source file for a package. Then I'd be able to pull all required
>> components, check their checksum and get rid of them once they're no longer
>> needed.
>> 
>> Is that last option possible?
>> 
>> And, in case it's not; is it possible to get around the no-download thing
>> (I know this is very discouraged but if there's no other solution...) I
>> promise to be good and verify checksums for anything I download!

You would have to duplicate all the checksum/update/parallel download logic
etc, that’s a lot of work and hard to maintain.

Effort aside, the underlying logic of the “no network” is to make isolated
and reproducible builds possible. Using “package” as the unit of organisation
facilitates this since a package has:
  - zero or one file download
  - zero or one build rule
  - zero or more dependencies
  - zero or more dependents
  - target-specific variants of all the above

Allowing downloads (potentially) breaks every one of those. Verifying the 
checksum isn’t part of the logical guarantee, it’s an execution detail.

Cheers,

Tony



[1] https://github.com/mxe/mxe/commit/85de732ad49a3b4827d02b8c46c98a59f41ef6c6
[2] https://github.com/mxe/mxe/blob/master/src/protobuf.mk
[3] https://github.com/mxe/mxe/blob/master/src/vtk.mk