Re: bootstrap: i686-linux now builds without binutils, gcc seeds

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

> It’s taken me a while but I successfully built 
> /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> for i686 from commit 5d8b7131c23d2285dd3546c59022dd5953508943 of
> ‘wip-bootstrap’.  :-)

That's great!

> $ guix hash -r /gnu/store/58q6748dydqxrhrw8xn2gsf60djl6gvv-hello-2.10
> 005zi21nvhmygjam6vsvkgmw9awsmyacz6a65naxcblrpbra1g94

And...you found the lucky numbers ;-)

> Previously we discussed that “-s i686-linux” on x86_64 would lead to a
> different graph compared to a native i686-linux run.  Is it still the
> case?  It looks like 80bd4a995 does the right thing in that respect.

Almost...that is to say for all packages up to gnu-make-boot0 and
all packages mentioned below: `Remove bootstrap leaks.'

As discussed on IRC, I created bug
    https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32749

> The “interesting commits” are these:
>
>   5d8b7131c * gnu: mes: Oops, use mirror://gnu download url.
>   1ff6d26ac * bootstrap: Replace GNU toolchain seeds with Mes for i686-linux.
>   ca8895bdb * bootstrap: Add Mes bootstrap.
>   be65bf958 * gnu: Add Mes bootstrap seeds.
>   45856f886 * gnu: Add linux-libre-headers-bootstrap-tarball.
>   cd9830fcd * guix: copy-linux-headers: Prepare for Mes bootstrap.
>   6265040e7 * guix: package-from-tarball: Allow PROGRAM-TO-TEST to be #f.
>   8d9c16ed3 * gnu: texinfo-boot0: Remove bootstrap leaks.
>   f9a76a79c * gnu: bison-boot0: Remove bootstrap leaks.
>   12394cc83 * gnu: m4-boot0: New variable.
>   6891da335 * gnu: perl-boot0: Remove bootstrap leaks.
>   f933b162c * gnu: file-boot0: Remove bootstrap leaks.
>   9c0912a20 * gnu: findutils-boot0: Remove bootstrap leaks.
>   53616d896 * gnu: diffutils-boot0: Remove bootstrap leaks.
>   7503cee76 * bootstrap: %bootstrap-inputs+toolchain: Prepare for Mes 
> bootstrap.
>   80bd4a995 * bootstrap: %bootstrap-inputs: Prepare for Mes bootstrap.

So we'll probably want to look into that bug and either remove all the
`Remove bootstrap leaks' patches, or rewrite the *-boot0 packages from
getext-boot0 onward (gcc-boot0-wrapped ld-wrapper-boot3 glibc-final
binutils-final zlib-final gcc-final bash-final guile-final
glibc-utf8-locales-final gnu-make-final coreutils-final grep-final
sed-final).

> I looked at the result and overall it LGTM!  So I think the next step is
> to rebase the branch on ‘core-updates’ (or merge it) and rename it
> ‘core-updates-next’.  WDYT?  Ricardo?

Amazing news!  I think it's best to "wait" until the `bootstrap leak'
thing is resolved; wip-bootstrap once more?

> Some comments on things that I think could be improved, in no particular
> order:
>
>   • It would be good to update the “Bootstrapping” section of the
>     manual—no urgency though.  It would be useful both to Guix
>     developers and to outsiders interested in bootstrapping
>     (Bootstrappable, Reproducible Builds, maybe projects like GNU/Linux
>     From Scratch, etc.)

I have added this text to wip-bootstrap, image not shown inline here, WDYT?

--8<---------------cut here---------------start------------->8---
diff --git a/doc/guix.texi b/doc/guix.texi
index 19a497c74..5a2cc8155 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -210,6 +210,7 @@ GNU Distribution
 * Package Modules::             Packages from the programmer's viewpoint.
 * Packaging Guidelines::        Growing the distribution.
 * Bootstrapping::               GNU/Linux built from scratch.
+* Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
 * Porting::                     Targeting another platform or kernel.
 
 System Installation
@@ -8657,6 +8658,7 @@ For information on porting to other architectures or 
kernels,
 * Package Modules::             Packages from the programmer's viewpoint.
 * Packaging Guidelines::        Growing the distribution.
 * Bootstrapping::               GNU/Linux built from scratch.
+* Reduced Binary Seed Bootstrap::  A Bootstrap worthy of GNU.
 * Porting::                     Targeting another platform or kernel.
 @end menu
 
@@ -23447,6 +23449,9 @@ Binutils, libc, and the other packages mentioned 
above---the
 These bootstrap binaries are ``taken for granted'', though we can also
 re-create them if needed (more on that later).
 
+For @code{i686-linux} the Guix bootstrap process is more elaborate,
address@hidden Binary Seed Bootstrap}.
+
 @unnumberedsubsec Preparing to Use the Bootstrap Binaries
 
 @c As of Emacs 24.3, Info-mode displays the image, but since it's a
@@ -23600,6 +23605,70 @@ bootstrap GCC with a sequence of assemblers, 
interpreters, and compilers
 of increasing complexity, which could be built from source starting from
 a simple and auditable assembler.  Your help is welcome!
 
address@hidden Reduced Binary Seed Bootstrap
address@hidden The Reduced Binary Seed Bootstrap
+
+Guix---like other GNU/Linux distributions---is traditionally bootstrapped from
+from a set of bootstrap binaries: Bourne shell, command-line tools provided by
+GNU Coreutils, Awk, Findutils, `sed', and `grep' and Guile, GCC, Binutils, and
+the GNU C Library (@pxref{Bootstrapping}).  Usually, these bootstrap-binaries
+are ``taken for granted.''
+
+What does this mean, really?  By taking these binaries for granted, trusting
+Guix depends on the trusting these binaries to be correct and clean.  Therein
+lies a problem: the current combined size of these bootstrap-binaries is about
+250MB (@pxref{Bootstrappable Builds,,, mes, Mes Reference Manual}).  Auditing
+or even inspecting these is next to impossible.
+
+For @code{i686-linux}, Guix now features a ``Reduced Binary Seed'' bootstrap
address@hidden would like to say: ``Full Source Bootstrap'' and while we are
+working towards that it would be a hyperbole to use that term for what we do
+now.}.
+
+The Reduced Binary Seed bootstrap removes the most critical tools---from a
+trust perspective---from the bootstrap binaries: GCC, Binutils and the GNU C
+Library are replaced by: @code{mescc-tools-seed} (a tiny assembler and linker)
address@hidden (a small Scheme Interpreter and a C compiler writen in Scheme)
+and @code{tinycc-seed} (the Mes C Library, built for TinyCC).  Using these new
+binary seeds and a new set of
address@hidden
+package address@hidden@c
+mescc-tools-boot,
+nyacc-boot,
+mes-boot,
+tcc-boot0,
+tcc-boot,
+make-mesboot0,
+diffutils-mesboot,
+binutils-mesboot0,
+gcc-core-mesboot,
+mesboot-headers,
+glibc-mesboot0,
+gcc-mesboot0,
+binutils-mesboot,
+make-mesboot,
+gcc-mesboot1,
+gcc-mesboot1-wrapper,
+glibc-headers-mesboot,
+glibc-mesboot,
+gcc-mesboot,
+gcc-mesboot-wrapper.
+}
address@hidden
+the ``missing'' Binutils, and the GNU C Library are built, from source.  From
+here on the more traditional bootstrap process resumes.  This approach has
+reduced the bootstrap binaries in size to about 130MB.  Work is ongoing to
+reduce this further.  If you are interested, join us on @code{#boottrappable}
+on the Freenode IRC network.
+
address@hidden ./pre-inst-env guix graph --type=bag -e '(begin (use-modules 
(guix packages)) (%current-system "i686-linux") (@@ (gnu packages commencement) 
gcc-mesboot))' > doc/images/gcc-mesboot-bag-graph.dot
address@hidden dot -T png doc/images/gcc-mesboot-bag-graph.dot > 
doc/images/gcc-mesboot-bag-graph.png
+
+Below is the generated dependency graph to for @code{gcc-mesboot} that builds
+the rest of GuixSD.
+
address@hidden/gcc-mesboot-bag-graph,6in,,Dependency graph of the gcc-mesboot}
+
 
 @node Porting
 @section Porting to a New Platform
--8<---------------cut here---------------end--------------->8---

>   • I think all the “mesboot” & co. packages in commencement.scam can be
>     made private–i.e., changing ‘define-public’ to ‘define’.

I agree; I kept gcc-mesboot (the final gcc) public, is that OK?

>   • There’s a couple of tests (for example in tests/debug-link.scm) that
>     rely on %gcc-bootstrap, on the assumption that building it is
>     cheap.  We should double-check that these are still okay on i686.
>
>   • IWBN to add comments for gcc-mesboot1-wrapper,
>     glibc-headers-mesboot, %bootstrap-inputs+toolchain, just a line or
>     two giving some context.

Done.  Adding the comment for gcc-mesboot1-wrapper, I wonder if this
package is currently necessary.  I "played" (ahum) a lot with building
gcc-mesboot1 (the first 4.7.4) already using --enable-shared.  At the
moment it's built using --disable-shared and gcc-mesboot (4.9.4) is the
first gcc built using --enable shared.

I probably kept gcc-mesboot1-wrapper even if it was no longer needed,
because adding/removing it is also cumbersome.

I'm testing right now to see if we can remove gcc-mesboot1-wrapper (and
will do so if it's not currently needed); that information will help
getting the comment more informational if it's still needed.

>   • I saw a couple of hanging parens in commencement.scm, they want to
>     be next to their friends.  ;-)

Oh, how sad :-(  Fixed!

>   • Minor style issues:
>     `(,@(substitute-keyword-arguments …)) → (substitute-keyword-arguments …)

Sure, removed.

>     (zero? (system* …)) → (invoke …)

Done, ah...I still overlooked some of these (wip-bootstrap began before
`invoke' I think).  Still have some (zero? (system " ....")).

>   • Could you add a couple of lines of explanation at the top of the new
>     gnu/packages/patches/*.patch files, as we do for other patches?
>     Some of them could also be simplified; for instance
>     ‘glibc-boot-2.2.5.patch’ contains the diff of what looks like a
>     leftover file.

I'm looking into this.  I tried to "document" the problems that I
encountered by capturing the build errors in a file called `adiff'.

That was an ugly but convenient hack, I agree this should removed and
put into words.

>   • For ‘binutils-mesboot0’: package/inherit → package (inherit …)
>     since replacement for ‘binutils’ certainly won’t apply to
>     ‘binutils-mesboot0’.

OK.

>   • For commit 80bd4a995 for instance, instead of “Prepare for Mes
>     bootstrap”, the message could be “Wrap input lists into thunks.” or
>     something like that to clarify what the actual change is.

Nice.  Put that in a TODO for the next squash of wip-bootstrap.

> I noticed that:
>
>   ./pre-inst-env guix graph -e '(begin (use-modules (guix utils)) 
> (%current-system "i686-linux")(@@ (gnu packages commencement) 
> gnu-make-boot0))' | dot -Tps > t.ps
>
> shows lots of repetitions, which defeats eq? memoization, but the
> problem is not new; I’ll try to look into fixing this.

Great, thanks.  Yes, I wondered about that!

> As we were discussing on IRC, we’ll have to figure out what the next
> step should be, once this branch has become ‘core-updates-next’.  One
> interesting bit is to remove the mes.M1 seed, as you wrote.

This is certainly something that Jeremiah Orians will be working on,
even if not directly by helping with the mes.M2 rewrite, certainly
indirectly by creating M2-Planet v2.0.  This has a timeframe of about
half a year.

> Another one might be to do x86_64 bootstrapping (using i686 binaries
> at the beginning of the graph, until we reach address@hidden, at which point
> we should be able to build x86_64 binaries; though maybe I’m just too
> naive here :-)).

Yes, I think Gábor wanted to look into this, he had the same kind of
remark on IRC, Gábor?

I'm also working on a wip-x86_64 branch for Mes; this has a timeframe of
at least a couple of months (well, unless our resources magically
multiply :-).  Even when Mes is 64 bits, we would need to revise the
whole bootstrap chain (think: patch tcc, replace gcc-2.95 with
gcc-3.x/4.x?).

> Another low-hanging fruit (I think!) is using Gash instead of Bash,
> though that’s completely orthogonal.

Yes!  I'd love to get this discussion going too, and start to do some
planning.  I integrated most of bournish into Gash, and then also
Timothy's Geesh came on my radar.  Interesting puzzle this...Timothy?

> Thank you for the amazing work, and again apologies for taking so long
> to get back to you!

Thanks!  To be fair, although wip-bootstrap has been "mostly working"
for quite some time now, I only got wip-bootstrap in a somewhat
reviewable state and fixed the integration with Guix since a week.

janneke

-- 
Jan Nieuwenhuizen <address@hidden> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com