Native-compilation build process (was Re: Loading tramp for dump goes into infinite regress)

[Lynn Winebarger]

On Wed, Aug 3, 2022 at 8:22 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Wed, 3 Aug 2022 05:58:17 -0400
> > Cc: Eli Zaretskii <eliz@gnu.org>, Michael Albinus <michael.albinus@gmx.de>,
> >       emacs-devel <emacs-devel@gnu.org>
> >
> > I found the problem - when I run make in the src directory, the variables 
> > for the ELN and bin destinations are
> > blank, which causes the code in loadup that is supposed to make the 
> > compilation unit names into pairs to
> > not bother.
> > Maybe these two variables should get constructed in src/Makefile.in?  
> > Running make from the src directory
> > shouldn't result in an unusable dump file.
>
> Why are you running "make" only in 'src'?  Some necessary parts of the
> build must also run "make" in 'lisp' and in other places.

I'm replicating the 3 stage build process Stefan Monnier (added to the
cc list) described by invoking a shell script from site-load, when
dump-mode is set to pdump.  The shell script produces the second stage
build - the one with a dump containing just the files loaded directly
from loadup - then uses it to compile the files loaded by site-load.
It explicitly builds the emacs.pdmp target in src, moves emacs and
emacs.pdmp to emacs-1 and emacs-1.pdmp, then invokes:
make -C ../lisp EMACS=../src/emacs-1 compile-target
TARGETS="${SL_TARGETS}" BYTE_COMPILE_EXTRA_FLAGS="--eval '(setq
comp-file-preloaded-p t)'"
where SL_TARGETS has been extracted from site-load.el using the same
procedure used in constructing lisp.mk

And in the process of getting this script right, I also sometimes
invoke make directly from src after bootstrapping has attempted to
create emacs.pdmp and failed.

>
> In any case, these variables cannot be defined _only_ in src/Makefile,
> but we could perhaps copy the definition from the top-level
> Makefile.in to src/Makefile.in.
>
Whatever you think is best.

> > I'd probably make the code in loadup either throw an error or come up with 
> > some reasonable default.  If you
> > insist on having the final installation specified, then it should just 
> > error during the build instead of producing a
> > dump guaranteed to fail when loaded.  Otherwise, either use some dummy 
> > values or just treat the build
> > directory as the install directory.
>
> The final installation _must_ be specified, yes.  Using the build
> directory won't help because Emacs already does that internally.  IOW,
> it only needs help in knowing where the stuff _will_ be installed

That is true for the dump that is going to be installed, yes, but for
intermediate stages in the build that knowledge is not required for
the resulting dump to be useful.
In any case, once loadup determines there are native compile units
loaded while preparing for the dump, either they all should have the
names fixed up, or an error should be signaled.  It would be way more
helpful to get an error during the dump procedure stating "The
--bin-dir and --eln-dir flags must be specified while dumping with
native-compilation units loaded", than to get "attempt to load an
incoherent eln" file later.  Because that will be the result.

It's not clear to me, though, why the pdmp loading process should
refuse to work if the compilation units recorded in the dump file are
just strings.  It seems to me the "dual location" is more of a "nice
to have" feature than a real requirement.  There could be a flag to
dump an image with only the (relative) final installation paths
recorded, or it could just be a build that will always be run in place
(e.g. an ordinary user build).   Or even if an ordinary user reruns
loadup in dump mode after emacs has been fully built and installed.
Why would the cons in the native compilation unit record be useful at
that point?   For user dumps, recording the absolute paths would
presumably work fine, no?

Lynn