preparation for hygienic macro expansion

[Dirk Herrmann]

Hi folks,

[those who already know about Marius #:module-ref suggestion can skip the
first half of this message ...]

Guile up to now uses forms like the following (taken from boot-9.scm):
  (define define-private define)
which is not legal R5RS scheme.  A correct implementation would be
  (defmacro define-private args '(define ,@args))

I had attempted to rewrite all macro definitions into the defmacro form.
However, with the CVS version of guile this does not work, as Mikael has
pointed out:  The problem is, that the defmacro syntax expansion does not
work hygienically.  If in some other module "foo" someone wants to use an
own version of define, the following would break:
  (defmacro define args
    (if top-level
        `(define-public ,@args)
        `(define-private ,@args))
The expansion would work endlessly, since with the defmacro definition of
define-private, define-private expands _literally_ to define, and not to
the binding of define that was visible during the definition of
define-private.  That is, in module foo the expansion of define would be
self-referential.

This is not a theoretical example, but actually occurs in slib.scm.

Currently, we don't have a solution for this.  It requires to implement a
hygienic macro expansion mechanism, which can cope with the existence of
different modules.

However, Marius has already suggested a mechanism which will support
hygienic macro expansion as soon as it is available.  Marius suggests to
extend the input language accepted by guile, such that guile will
understand the following expression:
  (#:module-ref <module-name> <identifier>)
This indicates that the binding for <identifier> should be taken from the
module <module-name>.  Note that #:module-ref is a _keyword_, that is, a
special object type distinct from symbols.  Therefore, the syntax above
will not conflict with any other scheme expression.  And, since the
mechanism is hard-coded into the memoizer/evaluator, there is no chance
for the user to shadow it by own definitions.

The idea behind the special syntax above is, that the macro expander could
issue such lookup-commands in order to make sure that symbols are looked
up in the correct module.  However, as long as the macro expander does
not work hygienically and does not make use of this feature, it would be
possible to instrument code manually with the corresponding instructions.

For the example above, this would look like the following code:
  (defmacro define-private args '((#:module-ref (guile) define) ,@args))
that is, the binding for define shall be looked up in module (guile).
During memoization, whenever such a #:module-ref instruction is detected,
the memoizer checks, if the identifier is bound to a macro in the
corresponding module.  If so, macro expansion is performed.

I have implemented the above instruction in my local version of guile, in
the memoizer code. Here are some examples of how it works:

A)
  guile> (#:module-ref (guile) define)
  ERROR: In procedure memoization:
  ERROR: In line 0: Misplaced syntactic keyword (#:module-ref (guile) define).
  ABORT: (syntax-error)
This example shows, how during memoization the memoizer runs over the
#:module-ref expression and does the corresponding lookup.  Since in
module (guile) the symbol define is a syntactic keyword, the code looks to
the memoizer as if the following had been entered:
  guile> define
and here the syntactic keyword appears at a place where an expression was
expected.  Thus, the error message about the misplaced syntactic keyword 
is issued.

B)
  guile> (#:module-ref (guile) identity)
  #<procedure identity (x)>
In this case, the symbol identity is looked up in module (guile), and
guile's identity function is found.

C)
  guile> (defmacro original-define args
  ...      `((#:module-ref (guile) define) ,@args))
  guile> (defmacro define args 
  ...      `(original-define ,@args))
  guile> (define (foo) (display "foo works") (newline))
  guile> (foo)
  foo works
Try the same with:
  guile> (defmacro original-define args `(define  ,@args))
  guile> (defmacro define args `(original-define ,@args))
  guile> (define (foo) (display "foo works") (newline))
and you will see :-)


Now, I am wondering if one would need to distinguish between #:module-ref
and #:module-syntax-ref.  This would help a hygienc macro system to cope
with the situation that in the referenced module after the definition of
the macro the binding for the variable changes from macro to value or
vice versa.  Assume in the following examples that defmacro would work
hygienically.
  in module foo:
    (define bar <some function>)
    (defmacro baz args `(bar ,@args))
    ;; baz should expand using the function-binding of bar
    (defmacro bar args #t)
    ;; baz should still expand using the function binding of bar, right?
  in module bla (assume that baz is imported):
    (baz #t)
    ;; this would in a first step be expanded to:
    ((#:module-ref (foo) bar) #t)
    ;; And now, we would like to make sure that the function-binding is
    ;; found, and not the syntax transformer.  However, this does not seem
    ;; to be possible if there is only #:module-ref.
In my current implementation, there is only #:module-ref.  It would be
simple, however, to add a #:module-syntax-ref directive as well.  Marius,
what do you think?  Do we need two directives instead of one?

Best regards,
Dirk Herrmann