Re: pcase map binding form expansion failure on Emacs 27 only

[Adam Porter]

Kevin Vigouroux via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> However, in my test, the macro expansion returns a good result. Why?
>
> It seems to me that I proceeded as indicated by first loading ‘map
> v3.1’ and then compiling the code.

After further digging, the problem seems to be that Emacs 27.2 is not
activating the newer version of map.el installed from ELPA.  I'm able to
reproduce the problem like so:

1.  Ensure that map.el >=2.1 is installed into ~/.emacs.d/elpa.

2.  Make file "/tmp/argh.el" with these contents:

      (require 'package)
      (message "PACKAGE-DIR: %S"
               package-user-dir)
      (package-activate-all)
      (require 'map)
      (message "MAP IS: %S"
               (package-desc-version (car (alist-get 'map package-alist))))
      (message "MAP IS AT: %S"
               (locate-library "map"))
      (message "EXPANSION TEST: %S"
               (macroexpand-all '(pcase-let* (((map :max-width) plist))
                                   max-width)))
      (byte-compile-file "argh2.el" t)

3.  Make file "/tmp/argh2.el" with these contents:

      (let ((plist '(:max-width 2)))
        (pcase-let* (((map :max-width) plist))
          (message "MAX-WIDTH: %S" max-width)))

4.  Run this command:

      emacs -q --batch -l /tmp/argh.el

On Emacs 27.2, I get this output:

  PACKAGE-DIR: "~/.emacs.d/elpa"
  MAP IS: (2 1)
  MAP IS AT: "/home/me/.emacs.d/elpa/map-2.1/map.elc"
  EXPANSION TEST: (let* ((x6 (map-elt plist ':max-width))) (progn max-width))
  In toplevel form:
  argh2.el:3:30:Warning: reference to free variable ‘max-width’
  Loading /tmp/argh2.elc...
  Symbol’s value as variable is void: max-width

You can see that, even though `locate-library' returns the map-2.1
version, the expansion is not correct.  And when I run Emacs
interactively and `eval-buffer', I get the same result.

However, when I do "M-x unload-feature RET map RET", I get this message:

  Loaded libraries ("/home/me/tmp/src/emacs/emacs/lisp/json.elc")
  depend on /home/me/tmp/src/emacs/emacs/lisp/emacs-lisp/map.elc

So even though `locate-library' returns the map-2.1 version, apparently
the non-ELPA version is what's actually loaded, and that version does
not support the pcase pattern in question.

So, changing "/tmp/argh.el" to these contents, adding a call to
`unload-feature' to force the already-loaded version to be unloaded, and
allowing the subsequent `require' to load the newer version, fixes it:

  (require 'package)
  (message "PACKAGE-DIR: %S"
           package-user-dir)
  (package-activate-all)
  (unload-feature 'map t)
  (require 'map)
  (message "MAP IS: %S"
           (package-desc-version (car (alist-get 'map package-alist))))
  (message "MAP IS AT: %S"
           (locate-library "map"))
  (message "EXPANSION TEST: %S"
           (macroexpand-all '(pcase-let* (((map :max-width) plist))
                               max-width)))
  (byte-compile-file "argh2.el" t)

When run in batch mode, that gives this output:

  PACKAGE-DIR: "~/.emacs.d/elpa"
  MAP IS: (2 1)
  MAP IS AT: "/home/me/.emacs.d/elpa/map-2.1/map.elc"
  EXPANSION TEST: (let* ((x6 (map-elt plist :max-width))) (let ((max-width x6)) 
max-width))
  Loading /tmp/argh2.elc...
  MAX-WIDTH: 2

So there seems to be a discrepancy between the version of the library
returned by `locate-library' and the version of the activated package,
and the version *actually* loaded into Emacs.  Installing newer versions
of core libraries from ELPA doesn't seem to work, or at least not
reliably, on Emacs 27.  But on Emacs 26.3 and Emacs 28.0.50, it seems to
work--for me, anyway.

Is this a bug in Emacs 27, or am I misunderstanding something?  I
thought the point of having newer versions of these core libs on ELPA
was that they could be used in older--or current--Emacs versions, but
this doesn't seem to be working reliably.