Re: pdumper bootstrap error

[Eli Zaretskii]

> Date: Mon, 21 Jan 2019 19:43:06 +0000
> From: Alan Third <address@hidden>
> Cc: address@hidden, address@hidden, address@hidden
> 
> > Anyway, is this really TRT?  Both emacs$(EXEEXT) and
> > bootstrap-emacs$(EXEEXT) are (or should be) identical, so why do you
> > need to copy one over the other?  It should be a no-op.
> 
> The existing code copies the pdmp file. In Makefile.in, just above the
> code I added is the comment:
> 
> ## The dumped Emacs is as functional and more efficient than
> ## bootstrap-emacs, so we replace the latter with the former.
> 
> I can see the unexec code path replacing bootstrap-emacs, but the
> pdumper code doesn’t.

That's expected: with unexec, the dumped Emacs executable is different
from bootstrap-emacs, but with pdumper they are identical.  That
comment is only correct for the unexec case.

> Then below that is this:
> 
> ifeq ($(DUMPING),pdumper)
> $(pdmp): emacs$(EXEEXT)
>       LC_ALL=C $(RUN_TEMACS) -batch $(BUILD_DETAILS) -l loadup --temacs=pdump
>       cp -f $@ $(bootstrap_pdmp)
> endif
> 
> which looks very much to me like bootstrap-emacs.pdmp is being replaced
> with the new one, which was built using the new temacs. Here’s the output:
> 
> Dumping under the name emacs.pdmp
> dumping fingerprint: 
> bc058dbc261d383930761d6e2df83fee5bba2b36bab3d69333b2f5c09c6bfc64
> Dump complete
> Byte counts: header=80 hot=10523232 discardable=117216 cold=4485912
> Reloc counts: hot=587886 discardable=4552
> Adding name emacs-27.0.50.71
> Adding name emacs-27.0.50.71.pdmp
> cp -f emacs.pdmp bootstrap-emacs.pdmp
> 
> There is no corresponding copy of (t)emacs over bootstrap-emacs, even
> though bootstrap-emacs.pdmp has now been built by temacs.

Right, because they are (or should be) identical, so the copy would be
a waste of cycles.

> > I guess I'm saying we should step back and understand better why you
> > have a mismatch of the fingerprint.  Can you show the last part of the
> > build, starting from "CCLD temacs" and all the way till the end,
> > without this change?  I'd like to understand better how the
> > fingerprints come out different (they are identical on my system).
> 
> Here’s the output (the whole thing is in the attachment in the first email
> I sent):

Thanks.  I think I see the problem.  Your build lacks one crucial
step:

>   CCLD     temacs
> [ld warnings elided]
> ../build-aux/install-sh -c -d ../etc
> rm -f emacs && cp -f temacs emacs
> LC_ALL=C ./temacs -batch  -l loadup --temacs=pdump
> Loading loadup.el (source)...
> dump mode: pdump

What is missing here is invocation of temacs with --temacs=pbootstrap.
Here's how it looks on my system:

    CCLD     temacs.exe
  /bin/mkdir -p ../etc
  cp -f temacs.exe bootstrap-emacs.exe
  rm -f bootstrap-emacs.pdmp
  ./temacs --batch  -l loadup --temacs=pbootstrap
  Loading loadup.el (source)...
  dump mode: pbootstrap
  [...]
  Finding pointers to doc strings...
  Finding pointers to doc strings...done
  Dumping under the name bootstrap-emacs.pdmp
  dumping fingerprint: 
2b248756260d5de08810b0b41d60e6f7411fb4ae16995fa3f7fcfb49907a3a9e
  Dump complete
  Byte counts: header=80 hot=6817180 discardable=107200 cold=4235996
  Reloc counts: hot=399276 discardable=4862
  [...]
  rm -f emacs.exe && cp -f temacs.exe emacs.exe
  LC_ALL=C ./temacs -batch  -l loadup --temacs=pdump
  Loading loadup.el (source)...
  dump mode: pdump

The "cp -f temacs.exe bootstrap-emacs.exe" part comes from this rule
in src/Makefile:

  bootstrap-emacs$(EXEEXT): temacs$(EXEEXT)
          $(MAKE) -C ../lisp update-subdirs
  ifeq ($(DUMPING),unexec)
          $(RUN_TEMACS) --batch $(BUILD_DETAILS) -l loadup --temacs=bootstrap
    ifneq ($(PAXCTL_dumped),)
          $(PAXCTL_dumped) emacs$(EXEEXT)
    endif
          mv -f emacs$(EXEEXT) bootstrap-emacs$(EXEEXT)
          @: Compile some files earlier to speed up further compilation.
          $(MAKE) -C ../lisp compile-first EMACS="$(bootstrap_exe)"
  else
          @: In the pdumper case, make compile-first after the dump
          cp -f temacs$(EXEEXT) bootstrap-emacs$(EXEEXT) <<<<<<<<<<<<<<<<<
  endif

and since we have just produced a new temacs, this rule is expected to
fire, and produce a new bootstrap-emacs.  But on your system, it
doesn't.  My first suspicion is that this has something to do with
time granularity of the filesystem on which you run this: Make doesn't
notice that temacs is newer than bootstrap-emacs, so it doesn't remake
the latter.  Is it possible that the time stamps of temacs and
bootstrap-emacs are such that this happens?  If so, what kind of
filesystem is that?

> make[3]: Circular bootstrap-emacs <- temacs dependency dropped.

This warning is also worrisome: I don't get it, and I fail to see any
such circular dependency in my Makefile.  Maybe it's because macuvs.h
depends on bootstrap-emacs, and on macOS macuvs.h is indeed used?

> Perhaps successful builds are simply ones where it doesn’t need to use
> bootstrap-emacs?

_Every_ build that rebuilds temacs should also rebuild
bootstrap-emacs.  That this doesn't happen for you is IMO part of the
problem.