Re: Enabling native-comp when frequently rebuilding master on a laptop

[Lynn Winebarger]

On Mon, Apr 28, 2025 at 8:29 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Lynn Winebarger <owinebar@gmail.com>
> > Date: Sun, 27 Apr 2025 16:52:55 -0400
> > Cc: emacs-devel <emacs-devel@gnu.org>, acorallo@gnu.org
> >
> > The Makefile target for building all native AOT is the same as it was in 
> > version 28, I believe.
> > I've been working on this problem for my own selfish reasons.  I have a 
> > shell script that can determine the
> > string used in calculating comp-abi-hash, and reproduce the hash value 
> > without requiring emacs itself.  The
> > next issue is to generate the build dependencies of the emacs libraries.  
> > To generate a detailed makefile for
> > the ELN files then requires computing the hash of the contents of the 
> > source library, and a hash of the
> > relative path of the library as determined by the algorithm in src/comp.c.:
> >      As installing .eln files compiled during the build changes their
> >      absolute path we need an hashing mechanism that is not sensitive
> >      to that. For this we replace if match PATH_DUMPLOADSEARCH or
> >      *PATH_REL_LOADSEARCH with '//' before computing the hash.
> > Once those three components are known, the dependencies can be written.
> > So, I think what is required is something like:
> >  * A subdirectory <hash PATH_DUMPLOADSEARCH>-<abi-hash>
> >  * One file "<source basename>.path" for every lisp library containing the 
> > path described above
> >   * One file for every lisp library named <source>-<path hash>, containing 
> > the content hash
> >   * One "<source>.mk" with the rule:
> > <Eln-cache>/<version>-<abi>/<source>-<content hash>: <source> <eln-dep1> ...
> >   * Makefile with targets that will update those files if either the 
> > relative path or the content hash of a source
> > file changes, including generating the individual source makefiles
> >   * Include all those makefiles and a phony target to depend on all the 
> > individual eln targets
> >
> > Something like that, but the details are subject to debugging.
>
> Thanks, but I think the problem is simpler, and so its solution.
>
> We only fail to recompile *.el files into the *.eln when the versioned
> subdirectory of native-lisp/ changes, which happens either when the
> Emacs version is changed or the C code changes significantly, like
> when we add or remove a primitive.  What happens in that case is that
> the build process creates a new directory under native-lisp/ whose
> name is given by comp-native-version-dir, but the *.el files which
> were not modified since the last build will not be recompiled into
> their *.eln files under the new directory.
>
> What this means is that you don't need to know the 2 8-character
> hashes of the *.eln files' base names, because there will be no *.eln
> file in the new subdirectory that corresponds to those *.el files.
> For example, there will be no files-xxxxxx-yyyyy.eln file that
> corresponds to files.el in that new directory.
>
> So what we need is a rule that takes a file FOO.el and natively
> compiles it if there's no file matching the wildcard
> "FOO-????????-????????.eln" in the subdirectory of native-lisp/ whose
> name is given by comp-native-version-dir (well, actually, in the
> 'preloaded' subdirectory of that subdirectory).  Since temacs is
> already built at that time, you can ask it about the value of
> comp-native-version-dir.

I suppose there may be 2 problems I want to solve, which may or may
not address the point Sean raises.
1.  A general issue for both byte-compiled and native-compiled files
to recompile files in a precise compile-time dependency ordering and
necessity, based on dependencies between lisp code
2. A second general issue that would prompt recompiling (both byte-and
native-) due to changes in the abstract machine that specifically
impact that code.

On the second one, if we assume the system configuration and options
haven't changed, and the invariants of the lisp machine haven't
changed (i.e. changes to the byte-code machine or a change to the
native compiler  that would change the ABI_VERSION constant, like
changes to calling conventions), then the relevant changes would be to
the subrs defined and the arity of each.  If we can determine the
subrs called from a compiled lisp file, then as long as the arity of
those subrs has not changed, both the byte code and native-code should
be compatible.  That is, we should be able to just copy an eln file
from one cache directory to the other (or hard-link them), without
even changing the name.  And we should be able to conservatively
estimate the subrs called by scanning the constant vectors of the
byte-compiled file for symbols matching subr names from the version
that compiled it. [ I'll let Andrea correct me on this, since it is my
best guess ]
Actually those ABI components can be made explicit to make in a
similar way as the relative path and content hashes, by writing one
file for each abi component only if it would change from the existing
values.  So, for example,there would be files for constants
ABI_VERSION and HASH_LENGTH, EMACS_CONFIGURATION, and
EMACS_CONFIGURATION_OPTIONS, and then for each subr, a file "<subr
name>.subr" containing the arity "(<minargs> . <max args>)'.
For a given library, find all the subr symbols in the byte-compiled
form, then make the dependencies of that file include both the lisp
file dependencies and the subr dependencies.

My understanding of Sean's issue is that it would be solved if the
native-compiled libraries were only recompiled when actually
necessary, and done so efficiently rather than all-or-nothing.

Bonus points if the byte compiler wrote out dependency information as
an option, so explicit make rules would not be required to support the
machinery.

Lynn