Re: Preview: portable dumper

[Daniel Colascione]

On Tue, Nov 29 2016, Eli Zaretskii wrote:
>> From: John Wiegley <address@hidden>
>> Cc: address@hidden,  address@hidden,  address@hidden
>> Date: Mon, 28 Nov 2016 14:18:58 -0800
>> 
>> > Accepting this patch means we believe in this direction, consider it a
>> > perspective one, enough so that we are willing to invest a non-trivial
>> > amount of energy into making it happen, fix the fallout, debug the results,
>> > etc. We are talking about at least one major release, maybe more, and
>> > definitely several minor releases. IOW, we are talking several years. I
>> > don't think we should gamble with such high stakes, not unless we sure we
>> > want to go there, because this is the best alternative of [...]
>> 
>> If I didn't know you were talking about pdumper vs. unexec here, I'd swear
>> this above paragraph was expressing my own concerns about merging Tromey's
>> concurrency branch.
>
> Superficially, perhaps.  But as soon as you look at the details, there
> are two important differences.
>
> First, the only piece of code in the concurrency patch that is not
> entirely trivial is a 60-line function (30% of whose line count is
> comments and ifdef's).

You're the only person I've ever seen use words like "entirely trivial"
to describe the addition of shared-memory concurrency to a legacy
single-threaded system. Even if the code for the threads system looks
simple, the concepts involved are anything but.

By contrast, the portable dumper is about marshalling, which I think is
comparatively simpler.

> That's all there is to understand there.  The
> rest is extremely simple, almost trivial: functions that directly map
> into the corresponding system threads APIs, or 2-liners for creation,
> checking, and otherwise simple management of the few new object types.
> By contrast, the proposed patch, even in its initial version, weighs
> in at 3200 lines in its core part, and almost none of it is trivial or
> simple.
>
> What's more, it comes with the need to make important and non-trivial
> decisions wrt which objects and data structures need to be dumped, and
> how, something that requires intimate knowledge of the internals and
> their use patterns, both at dump time and in interactive sessions.
> These decisions will need to be revisited when the related objects are
> added or modified, which is the part that averts me the most, because
> it means extending the Lisp interpreter, which is already not easy,
> will have become even harder.

I don't see it as a problem that the internals of Emacs need to have
knowledge of the internals of Emacs.  What matters is whether adding new
object types, which is already a very rare exercise, is made
prohibitively hard. Keep in mind that any programmer who adds a new
object type needs to *also* update the garbage collector, and GC and the
portable dumper require exactly the same kind of object description.

(I looked into actually unifying the GC and pdumper, but I don't think
it's possible without introducing significant new abstractions. It's not
worth it for something that happens so rarely as modification to core
Emacs data type representations.)

If you want to follow this reasoning to its logical conclusion, we
should switch to Boehm GC.

> The other important difference is that threads are an opt-in feature:
> if you don't start any threads, Emacs with threads still works as it
> did before, and you can just ignore anything related to threads.  IOW,
> threads are a feature we offer to Lisp programs that want to use it.
> By contrast, dumping code is a central and mandatory part of building
> Emacs and of its startup process.  It cannot be ignored or made
> optional.

The portable dumper is also optional: you can always use the legacy
unexec code, which isn't going away, or just rely on loadup (and the
hypothetical ultra-fast lread we've discussed).