Elisp development news

[Neil Jerram]

I've completed my work for the moment on an Elisp translator (based on
work that was begun originally by Mikael Djurfeldt).  You can see
everything in detail by checking out the ossau-elisp-branch from CVS,
but for convenience I've also appended the up to date README file at
the end of this message.

I'd like to merge this work back into the main development branch,
both to encourage further use and development, and to avoid bitrot,
but there are some (possibly controversial) changes to the libguile C
code that I'd like to discuss and agree or rework first.  Those
changes are ...

- Removal of scm_lisp_nil and scm_lisp_t.  In my implementation, `nil'
  and `t' are read as #f and #t, so these aren't needed.

- Removal of nil-ify, t-ify, 0-cond, 0-ify, 1-ify, plus corresponding
  evaluator ISYMs: SCM_IM_NIL_IFY, SCM_IM_T_IFY, SCM_IM_0_COND,
  SCM_IM_0_IFY, SCM_IM_1_IFY.  I didn't need these, and found their
  presence confusing.

- Removal of nil-cons, nil-car, nil-cdr, null, nil-while, nil-eq,
  SCM_NILP, SCM_NILNULLP, SCM_NIL2EOL, SCM_EOL2NIL, SCM_EOL_IFY,
  SCM_NIL_IFY.  Again, these turned out not to be necessary.

- A new reader option `language', default value #f (=> scheme), can
  also be set to 'elisp.

- Corresponding reader modification: when the `language' option is set
  to 'elisp, the reader reads `nil' and `t' as #f and #t.

- [ Plus enhancements to scm_m_atfop to handle defalias and proper
  transformation of macro args, but there's no reason why these could
  be controversial at all, I think. ]

What do you think about these changes - specifically, whether it's OK
to merge them into the CVS main branch?  Strictly speaking, the
removed items should be deprecated first, as they were all present in
1.4.  But it's quite unlikely that anyone was actually using them, and
we could still slip the deprecation stage in before 1.6.0.

All thoughts appreciated; I may be a little slow to followup on
replies as I'm about to go away to Berlin for a week.

Thanks!
        Neil


---------- guile-core/lang/elisp/README --------------
                                                    -*- outline -*-

This directory holds the Scheme side of a translator for Emacs Lisp.

* Usage

To load up the base Elisp environment:

    (use-modules (lang elisp base))

Then you can switch into this module

    (define-module (lang elisp base))

and start typing away in Elisp, or evaluate an individual Elisp
expression from Scheme:

    (eval EXP (resolve-module '(lang elisp base)))

A more convenient, higher-level interface is provided by (lang elisp
interface):

    (use-modules (lang elisp interface))

With this interface, you can evaluate an Elisp expression

    (eval-elisp EXP)

load an Elisp file with no effect on the Scheme world

    (load-elisp-file "/home/neil/Guile/cvs/guile-core/lang/elisp/example.el")

load an Elisp file, automatically importing top level definitions into
Scheme

    (use-elisp-file "/home/neil/Guile/cvs/guile-core/lang/elisp/example.el")

export Scheme objects to Elisp

    (export-to-elisp + - * my-func 'my-var)

and try to bootstrap a complete Emacs environment:

    (load-emacs)

* Status

Please note that this is work in progress; the translator is
incomplete and not yet widely tested.

** Trying to load a complete Emacs environment.

To try this, type `(use-modules (lang elisp interface))' and then
`(load-emacs)'.  The following output shows how far I get when I try
this.

guile> (use-modules (lang elisp interface))
guile> (load-emacs)
Calling loadup.el to clothe the bare Emacs...
Loading /usr/share/emacs/20.7/lisp/loadup.el...
Using load-path ("/usr/share/emacs/20.7/lisp/" 
"/usr/share/emacs/20.7/lisp/emacs-lisp/")
Loading /usr/share/emacs/20.7/lisp/byte-run.el...
Loading /usr/share/emacs/20.7/lisp/byte-run.el...done
Loading /usr/share/emacs/20.7/lisp/subr.el...
Loading /usr/share/emacs/20.7/lisp/subr.el...done
Loading /usr/share/emacs/20.7/lisp/version.el...
Loading /usr/share/emacs/20.7/lisp/version.el...done
Loading /usr/share/emacs/20.7/lisp/map-ynp.el...
Loading /usr/share/emacs/20.7/lisp/map-ynp.el...done
Loading /usr/share/emacs/20.7/lisp/widget.el...
Loading /usr/share/emacs/20.7/lisp/emacs-lisp/cl.el...
Loading /usr/share/emacs/20.7/lisp/emacs-lisp/cl.el...done
Loading /usr/share/emacs/20.7/lisp/widget.el...done
Loading /usr/share/emacs/20.7/lisp/custom.el...
Loading /usr/share/emacs/20.7/lisp/custom.el...done
Loading /usr/share/emacs/20.7/lisp/cus-start.el...
Note, built-in variable `abbrev-all-caps' not bound
  ... [many other variable not bound messages] ...
Loading /usr/share/emacs/20.7/lisp/cus-start.el...done
Loading /usr/share/emacs/20.7/lisp/international/mule.el...
<unnamed port>: In procedure make-char-table in expression (@fop 
make-char-table (# #)):
<unnamed port>: Symbol's function definition is void
ABORT: (misc-error)

Type "(backtrace)" to get more information or "(debug)" to enter the debugger.
guile> 

That's 3279 lines ("wc -l") of Elisp code already, which isn't bad!

I think that progress beyond this point basically means implementing
multilingual and multibyte strings properly for Guile.  Which is a
_lot_ of work and requires IMO a very clear plan for Guile's role with
respect to Emacs.

* Design

When thinking about how to implement an Elisp translator for Guile, it
is important to realize that the great power of Emacs does not arise
from Elisp (seen as a language in syntactic terms) alone, but from the
combination of this language with the collection of primitives
provided by the Emacs C source code.  Therefore, to be of practical
use, an Elisp translator needs to be more than just a transformer that
translates sexps to Scheme expressions.

The finished translator should consist of several parts...

** Syntax transformation

Although syntax transformation isn't all we need, we do still need it!

This part is implemented by the (lang elisp transform) module; it is
close to complete and seems to work pretty reliably.

Note that transformed expressions use the address@hidden' and address@hidden' 
macros
provided by...

** C support for transformed expressions

For performance and historical reasons (and perhaps necessity - I
haven't thought about it enough yet), some of the transformation
support is written in C.

*** @fop

The address@hidden' macro is used to dispatch Elisp applications.  Its first
argument is a symbol, and this symbol's function slot is examined to
find a procedure or macro to apply to the remaining arguments.  address@hidden'
also handles aliasing (`defalias'): in this case the function slot
contains another symbol.

Once address@hidden' has found the appropriate procedure or macro to apply, it
returns an application expression in which that procedure or macro
replaces the address@hidden' and the original symbol.  Hence no Elisp-specific
evaluator support is required to perform the application.

*** @bind

Currently, Elisp variables are the same as Scheme variables, so
variable references are effectively untransformed.

The address@hidden' macro does Elisp-style dynamic variable binding.
Basically, it locates the named top level variables, `set!'s them to
new values, evaluates its body, and then uses `set!' again to restore
the original values.

Because of the body evaluation, address@hidden' requires evaluator support.
In fact, the address@hidden' macro code does little more than replace itself
with the memoized SCM_IM_BIND.  Most of the work is done by the
evaluator when it hits SCM_IM_BIND.

One theoretical problem with address@hidden' is that any local Scheme variable
in the same scope and with the same name as an Elisp variable will
shadow the Elisp variable.  But in practice it's difficult to set up
such a situation; an exception is the translator code itself, so there
we mangle the relevant Scheme variable names a bit to avoid the
problem.

Other possible problems with this approach are that it might not be
possible to implement buffer local variables properly, and that
address@hidden' might become too inefficient when we implement full support
for undefining Scheme variables.  So we might in future have to
transform Elisp variable references after all.

*** Truth value stuff

Lots of stuff to do with providing the special self-evaluating `nil'
and `t' symbols, and macros that convert between Scheme and Elisp
truth values, and so on.

I'm hoping that most of this will go away, but I need to show that
it's feasible first.

** Emacs editing primitives

Buffers, keymaps, text properties, windows, frames etc. etc.

Basically, everything that is implemented as a primitive in the Emacs
C code needs to be implemented either in Scheme or in C for Guile.

The Scheme files in the primitives subdirectory implement some of
these primitives in Scheme.  Not because that is the right decision,
but because this is a proof of concept and it's quicker to write badly
performing code in Scheme.

Ultimately, most of these primitive definitions should really come
from the Emacs C code itself, translated or preprocessed in a way that
makes it compile with Guile.  I think this is pretty close to the work
that Ken Raeburn has been doing on the Emacs codebase.

** Reading and printing support

Elisp is close enough to Scheme that it's convenient to coopt the
existing Guile reader rather than to write a new one from scratch, but
there are a few syntactic differences that will require adding Elisp
support to the reader.

- Character syntax is `?a' rather than `#\a'.  (Not done.  More
  precisely, `?a' in Elisp isn't character syntax but an alternative
  integer syntax.  Note that we could support most of the `?a' syntax
  simply by doing 

      (define ?a (char->integer #\a)
      (define ?b (char->integer #\b)

  and so on.)

- `nil' and `t' should be read (I think) as #f and #t.  (Done.)

- Vector syntax is `[1 2 3]' rather than `#(1 2 3)'.  (Not done.)

Correspondingly, when printing, #f and '() should be written as
`nil'.  (Not done.)

** The Elisp evaluation module (lang elisp base)

Fundamentally, Guile's module system can't be used to package Elisp
code in the same way it is used for Scheme code, because Elisp
function definitions are stored as symbol properties (in the symbol's
"function slot") and so are global.  On the other hand, it is useful
(necessary?) to associate some particular module with Elisp evaluation
because

- Elisp variables are currently implemented as Scheme variables and so
  need to live in some module

- a syntax transformer is a property of a module.

Therefore we have the (lang elisp base) module, which acts as the
repository for all Elisp variables and the site of all Elisp
evaluation.

The initial environment provided by this module is intended to be a
non-Emacs-dependent subset of Elisp.  To get the idea, imagine someone
who wants to write an extension function for, say Gnucash, and simply
prefers to write in Elisp rather than in Scheme.  He/she therefore
doesn't buffers, keymaps and so on, just the basic language syntax and
core data functions like +, *, concat, length etc., plus specific
functions made available by Gnucash.

(lang elisp base) achieves this by

- importing Scheme definitions for some Emacs primitives from the
  files in the primitives subdirectory

- then switching into Elisp syntax.

After this point, `(eval XXX (resolve-module '(lang elisp base)))'
will evaluate XXX as an Elisp expression in the (lang elisp base)
module.  (`eval-elisp' in (lang elisp interface) is a more convenient
wrapper for this.)

** Full Emacs environment

The difference between the initial (lang elisp base) environment and a
fully loaded Emacs equivalent is

- more primitives: buffers, char-tables and many others

- the bootstrap Elisp code that an undumped Emacs loads during
  installation by calling `(load "loadup.el")'.

We don't have all the missing primitives, but we can already get
through some of loadup.el.  The Elisp function `load-emacs' (defined
in (lang elisp base) initiates the loading of loadup.el; (lang elisp
interface) exports `load-emacs' to Scheme.

`load-emacs' loads so much Elisp code that it's an excellent way to
test the translator.  In current practice, it runs for a while and
then fails when it gets to an undefined primitive or a bug in the
translator.  Eventually, it should go all the way.  (And then we can
worry about adding unexec support to Guile!)  For the output that
currently results from calling `(load-emacs)', see above in the Status
section.

* nil, #f and '()

For Jim Blandy's notes on this, see the reference at the bottom of
this file.  Currently I'm investigating a different approach, which is
better IMO than Jim's proposal because it avoids requiring multiple
false values in the Scheme world.

According to my approach...

- `nil' and `t' are read (when in Elisp mode) as #f and #t.

- `(if x ...)', `(while x ...)' etc. are translated to something
  like `(if (and x (not (null? x))) ...)'.

- Functions which interpret an argument as a list --
  `cons', `setcdr', `memq', etc. -- either convert #f to '(), or
  handle the #f case specially.

- `eq' treats #f and '() as the same.

- Optionally, functions which produce '() values -- i.e. the reader
  and `cdr' -- could convert those immediately to #f.  This shouldn't
  affect the validity of any Elisp code, but it alters the balance of
  #f and '() values swimming around in that code and so affects what
  happens if two such values are returned to the Scheme world and then
  compared.  However, since you can never completely solve this
  problem (unless you are prepared to convert arbitrarily deep
  structures on entry to the Elisp world, which would kill performance),
  I'm inclined not to try to solve it at all.

* Resources

Ken Raeburn's Guile Emacs page: ???

Keisuke Nishida's Gemacs project: http://gemacs.sourceforge.net.

Jim Blandy's nil/#f/() notes: ???

---------- guile-core/lang/elisp/README --------------