Re: Preview: portable dumper

[Daniel Colascione]

On Feb 17, 2018 3:38 PM, Ken Brown <address@hidden> wrote:

On 2/15/2018 9:36 PM, Daniel Colascione wrote:
> On 02/15/2018 05:56 PM, Ken Brown wrote:
>> On 2/15/2018 6:36 PM, Daniel Colascione wrote:
>>> On Feb 15, 2018 3:31 PM, Ken Brown <address@hidden> wrote:
[...]
>>>     I just tried to build on 64-bit Cygwin, and the build fails as
>>> follows:
>>>
>>>     [...]
>>>     Dumping under the name bootstrap-emacs.pdmp
>>>     dumping fingerprint:
>>>     923050a9f611ad7ead76eea704308e4d05f152601a9134cf8d1b5ff3e0e1a986
>>>     Dump complete
>>>     Byte counts: header=80 hot=13187392 discardable=119424 cold=9086640
>>>     Reloc counts: hot=919268 discardable=5790
>>>     make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>>>     make[2]: Entering directory
>>>     '/home/kbrown/src/emacs/x86_64-pdumper/lisp'
>>>         ELC      ../../pdumper/lisp/emacs-lisp/macroexp.elc
>>>         ELC      ../../pdumper/lisp/emacs-lisp/cconv.elc
>>>         ELC      ../../pdumper/lisp/emacs-lisp/byte-opt.elc
>>>         ELC      ../../pdumper/lisp/emacs-lisp/bytecomp.elc
>>>     emacs: could not load dump file "../src/bootstrap-emacs.pdmp": out
>>>     of memory
>>>
>>>     There's probably some obvious explanation, but I don't see it at the
>>>     moment.
>>>
>>>
>>> I'm not entirely surprised to see Cygwin fall over here. We could
>>> just use the Windows memory mapping functions directly, but I'd
>>> prefer to stick with the POSIX API if we can make it work.
>>
>> I agree.
>>
>>> Any idea where in the mmap sequence we fail?
>>
>> I haven't looked at the code yet, so I don't understand the question.
>> If you give me some guidance, I'll try to investigate.
>
> Thanks. I think the trouble must be in dump_mmap_contiguous. We report
> all errors from this function as "out of memory". dump_mmap_contiguous
> takes an array of mapping descriptors (think ELF segments or something)
> and maps them all into a single contiguous region of virtual memory. On
> POSIX systems, we reserve a chunk of address space with a big PROT_NONE
> anonymous mapping, then carve it up into the separate mappings that we
> really want. Windows doesn't support atomic mmap replacement, so Cygwin
> has to emulate it, and I wouldn't be surprised if something's going
> wrong in this emulation.

You're right that the problem is in dump_mmap_contiguous.  I stepped
through it in gdb and found the following:

1. The call to dump_anonymous_allocate at pdumper.c:4432 succeeds.

2. We enter the loop at pdumper.c:4451 with i = 0.  dump_map_file is
called, which calls dump_map_file_posix with protection =
DUMP_MEMORY_ACCESS_READWRITE.

3. This calls mmap at line 4219 with mem_prot = PROT_READ | PROT_WRITE
and mem_flags = MAP_PRIVATE | MAP_FIXED.

4. This mmap call fails.

I don't know if this is a bug in Cygwin's mmap or simply a limitation
that has to be worked around.  Should I take this to the Cygwin mailing
list, or do you have other ideas as to how to proceed?

I'd definitely let the Cygwin people know. If the problem is that Cygwin doesn't properly support atomic map replacement, I think we can work around the problem by using the same unmap-before-remap approach we use for the native Windows code, but keep using the POSIX memory functions. I can come up with a patch blind, but you'd be in a better position to iterate quickly.

I don't think we want to just switch to the Windows memory mapping code. Cygwin won't copy maps we make that way through a fork, IIIRX, so the child might blow up if it accesses dump memory.

> That you're able to compile a few elisp files before bootstrap starts
> failing...

That's an illusion.  I was doing a parallel build, and the output I
posted simply showed the first few files that emacs was trying to compile.

Ah, I see.