Re: Is add-to-list supposed to work when lexical-binding is t?

[Stefan Monnier]

>>> Section 11.9.3 (Lexical Binding) in the manual says "functions like
>>> `symbol-value', `boundp', and `set' only retrieve or modify
>>> a variable's dynamic binding".  Why?

Actually, the answer is much simpler: all the above are *functions* (as
opposed to macros or special forms), so they receive values as arguments
and know very little from their caller other than the values it passed
as arguments.

In order to do what you'd want them to do, they'd have to look at the
value of variables in the scope of their caller.  This is exactly what
dynamic scoping provides, and is what lexical scoping prevents.

>> With lexical scoping, a given variable name can have many different
>> values at the same time so this is not an option.

> I read the last bit as, "with lexical scoping, a given symbol (symbolic
> name) can be interpreted as many different variables in different scopes
> (and each variable can have multiple instances) ...", to synchronize
> terminology. And I call the things that variables are bound to "instances",
> not "bindings" like the docs do.

*at the same time* is the important part.  With dynamic scoping, only
one of the various scopes exists at a given time.  But with lexical
scoping it's a bit more tricky:

   (let ((funs (mapcar (lambda (x) (lambda (y) (+ x y)) '(1 2 3))))
     (mapcar (lambda (f) (funcall f 3)) funs))

this should return (4 5 6).  Between the two mapcars, we have stored in
`funs' 3 functions (more specifically closures), each one of them of the
kind (lambda (y) (+ x y)).  They each have their own copy of `x', so all
three copies of `x' exist at the same time in `funs'.
     
> In the scope of (let ((x 'a)) ...) with lexical binding enabled, since the
> symbol x is interpreted as a lexical variable rather than as the global
> variable (the latter bound to the global instance, the value cell for the
> symbol x), I propose a quote-lex special form, where (quote-lex x), with
> e.g. "&x" or maybe "^x" as readable syntax, returns a reference to the
> current instance

You can definitely do that.  In the current Emacs sources it's provided
under the name `gv-ref', which indeed works like C's &.

> Then you could use argument-mutating functions, including the standard
> add-to-list function, without having to convert them into macros, and do
> e.g.:
> (let ((x '(a))) (add-to-list &x 'b) x) -> (b a)

We can definitely make add-to-list work for

   (let ((x '(a))) (add-to-list (gv-ref x) 'b) x)   ===>   (b a)

That's easy and would work fine.  But (gv-ref x) is not the same as 'x
and trying to magically turn one into the other, while feasible in a few
particular cases is simply impossible in general.

>>> (let ((x '(a))) (add-to-list 'x 'b) x) -> (b a)
>> Yup, this is asking for trouble.  Use `push' or `cl-pushnew' instead.
> So, converting every argument-mutating function into a macro actually _is_
> the right thing to do?

Yes, pretty much.

> That seems unnecessarily complicated, just a way of working around the
> lack of lexical quoting.

There is no "lack of lexical quoting", as seen above.

Argument-mutating functions are relatively rare and using macros tends
to be a lot more efficient than using gv-ref, so in most cases using
macros makes more sense, despite their well known disadvantages.


        Stefan