Bigloo, Macros and the REPL

I've just spent quite a while trying to make Bigloo do what I want with macros. What I want:

1. To be able to write macros within a module which I can use while compiling code in that module or other modules which import the macro-module.
2. To be able to write macros which are imported into an repl which incorporates the module in which they are written.

Doing this has required a bit of trickery. (The difficulty is that I want the macros at 3 times: 1. When the compiler is processing the module in which they are defined. 2. When the compiler is processing other modules which use them. 3. When the repl is up and running.)

The solution is to write modules with macros like this:

(module with-gensyms
   (eval (export-exports))
   (export (with-gensyms-expander x e)))

(define (with-gensyms-expander x e)
   (match-case x
      ((with-gensyms (?sym) . ?body)
       (e `(let ((,sym (gensym)))
              ,@body) e))
      ((with-gensyms (?sym1 . ?rest) . ?body)
       (e `(let ((,sym1 (gensym)))
              (with-gensyms ,rest ,@body)) e))))

(eval '(define-expander with-gensyms with-gensyms-expander))

And use them in other modules like this:

(module f64vector
   (eval (export-all))
   (import (with-gensyms "with-gensyms.scm")
           (do-macros "do-macros.scm"))
   (load (with-gensyms "with-gensyms.scm")
         (do-macros "do-macros.scm"))
   (type (tvector f64vector (double)))
   (export (f64vector::f64vector . inits)
           (with-f64vectors-expander x e)))

(define (f64vector::f64vector . inits)
   (let* ((n (length inits))
          (v (make-f64vector n 0.0)))
      (let loop ((i 0) (list inits))
         (if (null? list)
             v
             (begin 
                (f64vector-set! v i (car list))
                (loop (+fx i 1) (cdr list)))))))

(define (with-f64vectors-expander x e)
   (match-case x
      ((with-f64vectors ?vecs (<- . ?body))
       (with-gensyms (result n i)
          (e `(let* ((,n (f64vector-length ,(car vecs)))
                     (,result (make-f64vector ,n 0.0)))
                 (do-times (,i ,n)
                    (let ,(map (lambda (sym)
                                  `(,sym (f64vector-ref ,sym ,i)))
                               vecs)
                       (f64vector-set! ,result ,i
                                       (begin ,@body))))
                 ,result) e)))
      ((with-f64vectors ?vecs . ?body)
       (with-gensyms (i n)
          (let* ((vector-syms (map (lambda (sym) (cons sym (gensym))) vecs))
                 (process-body-term
                  (lambda (term)
                     (match-case term
                        ((?result <- . ?body)
                         `(f64vector-set! ,(cdr (assoc result vector-syms))
                                          ,i
                                          (begin ,@body)))
                        (?- term)))))
             (e `(let ((,n (f64vector-length ,(car vecs))))
                    (let ,(map (lambda (sym)
                                  `(,(cdr (assoc sym vector-syms)) ,sym))
                               vecs)
                       (do-times (,i ,n)
                          (let ,(map (lambda (sym)
                                        `(,sym (f64vector-ref ,sym ,i)))
                                     vecs)
                             ,@(map process-body-term body))))) e))))))

(eval '(define-expander with-f64vectors with-f64vectors-expander))

(Note that this module uses two macro-modules---one of which is not listed above---to define a third. I'm sorry for the confusing example, but it's what I have handy.)

How does this work? Here's my understanding:

1. The compiler begins processing the with-gensyms.scm file. The compiler compiles it into with-gensyms.o which contains a module initialization routine that evaluates all the top-level commands in the module whenever the module is initialized---this ensures that the (eval '(define-expander with-gensyms ...)) runs whenever the module is initialized in compiled code (i.e. when running a custom repl).
2. The compiler processes f64vector.scm. It sees the (load (with-gensyms ...)) command in the module header, and interprets the file with-gensyms.scm, installing the with-gensyms macro before processing the code in f64vector.scm. The f64vector.scm file compiles into f64vector.o which contains a module initialization routine which will initialize the with-gensyms module first (because bigloo sees that it is required by a (import (with-gensyms ...)) in the f64vector module header) whenever the f64vector module is required from compiled code (i.e. within the main routine which implements the repl).

Thus, when compiling the code, all the macros are interpreted, but when running the code (i.e. within the repl), the macro-expanders are actually compiled! Any module which needs the macros at compile time should use the (load ...) form in the module header, and any module which wishes to install these macros at run time should use the (import ...) form. Whew! The repl in question can be implemented by

(module repl
   (import (with-gensyms "with-gensyms.scm")
    (do-macros "do-macros.scm")
    (f64vector "f64vector.scm"))
   (main main))

(define (main argv)
   (repl))

It took me a long enough time to figure this out (and I wouldn't have figured it out without some suggestions from jg malecki (see this message and its antecedents---thanks jg) that I thought I should post the explanation here so that other people don't have to go through the same difficulties that I have.