Re: master 5252c45: Initialise unread buffer

[Mattias Engdegård]

19 sep. 2021 kl. 17.41 skrev Lars Ingebrigtsen <larsi@gnus.org>:

> I'm not absolutely sure that it's your change that's causing this weird
> error, but it seemed the most likely.  :-/  And reverting back to the
> checkout before yours made the problem go away.

It's more fun actually. Thanks to the fixed reader bug, the lexical-binding 
cookie is now honoured in loadup.el. That causes Emacs to be dumped with 
`lexical-binding` set to t which as a consequence becomes the new default.

It's a testament to the sterling work by Stefan, Stefan, you, and everybody 
else that Emacs seems to work almost flawlessly anyway, but as it really wasn't 
our intention to do the switch now I've pushed an explicit binding of 
lexical-binding around the actual dumping so we're now back to normal.

What about the failing eval-tests then? They check, among other things, that

 (lambda (&rest) 'ok)

can be interpreted, byte-compiled, and returns `ok` when called. This fails 
with lexical binding enabled but works with dynamic binding, so when the 
default was inadvertently switched to lexical, the test started failing.

But wait, you protest. eval-tests.el is marked with -*- lexical-binding: t -*-. 
Surely? No?

Actually, any code in `ert-deftest` uses dynamic binding (until the accidental 
switch), regardless of the file cookie. Not many people know that.

So what is this bug that is still present? Let's take a look:

(byte-compile (lambda (&rest) 'ok))
=>
#[128 "\300\207" [ok] 1 "..."]

Let's pick it apart:
- The number 128 means that there are no positional args but a single &rest 
argument.
- The "\300\207" are the two byte-ops
   constant ok   ; push `ok` on the stack
   return        ; return the value on top of the stack
- The number 1 is the maximum stack size required.

When byte-code execution starts, the arguments are pushed on the stack -- in 
this case, a single nil for the &rest. The first byte-op then pushes the 
constant `ok`. But the stack is only sized for one element so that overwrites 
what follows, which is a copy of the byte-code. The smashed ops are then 
executed: if we are lucky we get an error, if unlucky an Emacs abort or crash.

I'm not sure why the byte-code interpreter makes a copy of the byte code for 
each function call. It certainly doesn't help the lamentably high function call 
overhead a bit. Most likely it's programmer, not code, efficiency that was 
optimised: since GC can move string data, the pc would need to be rebased each 
time that can occur, and that might need to be done in a lot of places. We 
should probably do something about that.

If the argument is named, then we get:

(byte-compile (lambda (&rest _x) 'ok))
=>
#[128 "\300\207" [ok] 2 "..."]

Note the max depth 2 here which gives a sufficiently big stack.

There's a number of things to fix here, but this message is too long already. 
Thanks for your interest! There are books for sale in the foyer.