Re: Preview: portable dumper

[Daniel Colascione]

On 02/16/2018 12:24 AM, Eli Zaretskii wrote:
> > Date: Thu, 15 Feb 2018 15:34:13 -0800
> > From: Daniel Colascione <address@hidden>
> > Cc: Eli Zaretskii <address@hidden>, Angelo Graziosi <address@hidden>,
> >   address@hidden
> >
> > I do wonder whether it makes sense to try to copy the dump into the Emacs 
> > executable itself instead of
> > leaving it as a separate file. We could do it independently of executable 
> > format by defining a data array in static
> > storage that's initially full of, say, 15MB of zeroes prefixed by a long random 
> > header (like a MIME boundary),
> > then, after we generate emacs.pdmp, copying the dump file into the executable 
> > at the place where we see
> > that random header. If the dump turns out to be bigger than that 15MB, we can 
> > fail the build and ask the user
> > to enlarge the array.
> >
> > I don't know of any executable format for which this scheme would fail.
>
> Wouldn't that make the dumper stuff less portable, in the sense that
> it would need to be compatible with low-level details of executable
> file formats on various systems?

No: I'm thinking about only options that work independently of 
particular executable formats.

> At least on non-ELF systems, AFAIK the flexibility of putting
> arbitrary sections into an executable is lower than desired.  For
> example, before Emacs 25 the MS-Windows build would create a special
> section for the initialized Emacs data, which had the annoying effect
> of running afoul of 'strip', because Binutils don't know about this
> section, and therefore stripping would produce a dysfunctional
> executable.  

That does sound annoying. The original proposal I had in mind was to 
make a normal data array:

uint8_t dump[15*1024*1024] = { embedded_dump_signature };

And then refer to this array on code. No part of the toolchain is 
allowed to remove this array. The "dump insertion" procedure we're 
considering would just amount to changing the contents of this array in 
an already-linked executable using dumb find-and-replace anchored on 
embedded_dump_signature.

> It also prevented re-dumping Emacs, something we had in
> the past and I'd like us to have again in the future.

I'd like to have it too. The data-section-array approach above wouldn't 
permit redumping unless the new dump fit into the old array. The 
concatenation approach would allow arbitrary redumping. And of course 
keeping the dump as a separate file would too.

> Wouldn't copying the dump into the executable hit the same problems,
> at least in principle?
>
> > This way, we could get rid of the current complicated dump-file location logic 
> > early in Emacs init.
>
> Can you elaborate why is that logic so complicated?

Two reasons:

1) The code to generate the dump file is complicated, especially because 
we want to support windows, and
2) with a separate dump file, it's easier to accidentally break the 
correspondence between the dump and the emacs binary itself. If we 
somehow stapled the dump to the binary, it'd be a lot harder to break 
this way.

I like the idea of being able to mv emacs emacs.old, make emacs, and 
then have emacs.old and emacs still both run and work normally. I don't 
think the numbered build mechanism you fixed completely solves the 
problem, since it only works in one specific context.