bdepend Problems Solved

I just figured out why I was having problems with module initialization. I also solved a long-standing mystery in my own mind: what the hell are those .mco files that Bigloo produces all the time? In fact, .mco files are module checksums---reproduced (I assume) every time the interface to a module changes. So, bdepend does, in fact, know about dependencies on (import ...) modules; it just uses the .mco files to compute these dependencies, not the .scm files themselves! Very smart, since the .scm file could change in a way that doesn't affect the interface and therefore wouldn't require recompilation of every module which (imports ...) said .scm file.

So the solution to my problem is to strike the -fno-mco flag on bdepend, and add a -fmco flag when bigloo is compiling a module. Ta-da! No more inconsistent module initialization, and no more complete system recompiles. Wonderful!

P.S.--Maybe someone (me?) should write something up for inclusion into the Bigloo manual about bdepend and mco files. There's very little in there right now (hence my problems making this work).

Bigloo GC and SIGINT

I've noticed in my work with Bigloo that sometimes when a process running from the REPL is sent SIGINT (C-c), the GC hangs in the function GC_lock. I don't know if it's related to dynamic-load-ing on OS X, or whether this is a general problem, and the GC is not very resistant to SIGINT. I suppose it could also be my optimization options on the compiled C code (I'm optimizing pretty heavily for my numerical work). I'm just putting it down here so I don't forget to file a bug report if I ever figure out the cause.

Inconsistent Module Initialization

I've been getting a lot of errors like the following lately from Bigloo:

*** ERROR:body:Inconsistent module initialization
Module `body' is inconsistently initialized by module `__nbody'.
At least of the two modules must be recompiled (see also -unsafev option).

and I think I finally realized what the problem is. I have a bunch of modules which are all separately compiled, and then put together by a "heap" file as described in the chapter from the Bigloo manual on libraries, but the Makefile I'm using doesn't think that the heap file depends on the separately compiled files. So, the code the heap file uses to initialize the modules it imports becomes stale whenever the exported interface of these modules changes.

This problem arises because bdepend doesn't think that import commands in the module header create dependencies. Normally this is true---bigloo modules can, in general, be separately compiled---but in the case of a heap file, it won't work. (It also won't work for the file which has the -dload-sym option on the command line if you're doing dynamic loading.) The heap file needs to be recompiled if any of the modules it imports have changed their exports. Probably it's easiest to just add a command to the Makefile so that the heap file depends on all the source files in the project; this will recompile it more than necessary, but it's easy and the heap file doesn't take very long to compile.

EDIT: Actually, this problem afflicts all files, not just the heap ones. If module A imports module B, then module A needs to be recompiled if module B's exported interface changes. I've submitted a bug report about bdepend not recognizing this to the bigloo list.

Perserverance: 1 Static Linking: 0

Finally! I've figured out how to make dynamic linking of module files work with bigloo on Mac OS X! Details follow (though I should warn you that this is a total hack right now).

There are two steps involved: 1. Alter the bigloo compilation process so it builds the proper dynamic libraries and 2. Make sure that the executable bigloo is linked against the dynamic libraries, and not the static ones.

#1: The way I know to do #1 is to edit Makefile.config after running ./configure [whatever-options-you-want]. Scroll through Makefile.config and change the following options:

LD

LD=ld

LDFLAGS

LDFLAGS=-dylib -single_module

LDLIBS

-lc -lSystem.B -ldylib1.o -lm -lc -lgcc

SHAREDSUFFIX

SHAREDSUFFIX=dylib

Now issue the command

make

The make process will halt with an error! That's OK. Re-edit the Makefile.config file by changing the LDLIBS variable to LDLIBS=-lbigloogc-2.7a -lc -lSystem.B -ldylib1.o -lm -lc -lgcc (you should alter the "-2.7a" to whatever version of bigloo you're compiling). The compile should now finish. You will have both libbigloo***-***.a and libbigloo***-***.dylib in the lib subdirectory.

#2: Though it seems like you should be able to ensure that the bigloo executable is linked against the dynamic libs by using the --sharedcompiler=yes flag to the configure script, this doesn't work. However, the fix is easy enough: edit the dboot target of the file comptime/Makefile. There will be a section looks something like

doboot: $(O_OBJECTS)
 if [ "$(SHRD_COMP)" = "yes" -a -f $(BOOTLIBDIR)/lib$(LIBRARYNAME)_s$(VERSION)$(SHAREDSUFFIX) ]; then            gc=-lbigloogc-$(RELEASE);            if [ "$(GCCUSTOM)" = "no" ]; then       gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) -L $(BOOTLIBDIR) -lbigloo_s-$(RELEASE) $$gc $(EXTRALIBS);  else            gc=$(BOOTLIBDIR)/libbigloogc-$(RELEASE).a;            if [ "$(GCCUSTOM)" = "no" ]; then               gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) $(BOOTLIBDIR)/libbigloo_s-$(RELEASE).a $$gc $(EXTRALIBS);  fi
 -$(STRIP) $(BIGLOO_DEST)$(EXE_SUFFIX)
 @ echo "$(BIGLOO_DEST)$(EXE_SUFFIX) done..."

you want to make it look like:

doboot: $(O_OBJECTS)
 if [ "$(SHRD_COMP)" = "yes" -a -f $(BOOTLIBDIR)/lib$(LIBRARYNAME)_s$(VERSION)$(SHAREDSUFFIX) ]; then            gc=-lbigloogc-$(RELEASE);            if [ "$(GCCUSTOM)" = "no" ]; then       gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) /tmp/bigloo2.7a/lib/2.7a/libbigloo_s-2.7a.dylib /tmp/bigloo2.7a/lib/2.7a/libbigloogc-2.7a.dylib  else            gc=$(BOOTLIBDIR)/libbigloogc-$(RELEASE).a;            if [ "$(GCCUSTOM)" = "no" ]; then               gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) $(BOOTLIBDIR)/libbigloo_s-$(RELEASE).a $$gc $(EXTRALIBS);  fi
 -$(STRIP) $(BIGLOO_DEST)$(EXE_SUFFIX)
 @ echo "$(BIGLOO_DEST)$(EXE_SUFFIX) done..."

(The difference is that we have explicitly included the libbigloo_s-2.7a.dylib and the libbigloogc-2.7a.dylib libraries on the $(CC) command line---including the full paths!) Now delete the old executable at bin/bigloo (relative to the unpacked bigloo directory) and re-make. Check with otool -L bin/bigloo to make sure that the bigloo executable really links against the dynamic libraries. The output of my otool looks like

SYDNEYPACIFIC-THREE-FORTY-ONE:/tmp/bigloo2.7a farr$ otool -L bin/bigloo
bin/bigloo:
        /tmp/bigloo2.7a/lib/2.7a/libbigloo_s-2.7a.dylib (compatibility version 0.0.0, current version 0.0.0)
        /tmp/bigloo2.7a/lib/2.7a/libbigloogc-2.7a.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 71.1.3)

Now, to make a loadable bundle, issue the following command (this example comes from the Bigloo manual, section 13.1 under the dynamic-load procedure documentation):

MACOSX_DEPLOYMENT_TARGET=10.3 gcc -bundle -undefined dynamic_lookup -o mod.bundle mod1.o mod2.o

This command replaces the ld -G -o lib.so mod1.o mod2.o command from the manual. You can now load the bundle with

(dynamic-load "mod.bundle")

from the REPL. You need the MACOSX_DEPLOYMENT_TARGET=10.3 so that it doesn't complain about the -undefined dynamic_lookup option; if you don't have 10.3, you can probably get away with -undefined suppress, but I haven't tried it.

Footnotes

1: I think it would be really neat if someone found a way to make all of this ad-hoc makefile-modifying happen automatically as part of the Bigloo build process on Mac OS X. I think it'll take someone more knowledgeable about the internals of the build process, but I may have a look at it later.

2:What happened here is that libbigloo requires the garbage collection library, so we have to include it in the LDLIBS variable. We can't put it in there at the beginning because the GC library doesn't exist yet, so we have to wait for the build to fail and then put it in. This is necessary because Mac OS X makes a distinction between dynamic libraries (which are treated like ELF static libraries for link purposes, but like ELF shared (or dynamic) libraries for loading at runtime) and loadable bundles (which are treated like ELF shared libraries all the time). In English: you have to have all the symbols for a .dylib at link time, though the libraries which contain them won't be loaded until run time, while you don't need the symbols for a .bundle until you use them at runtime.

3:And I don't understand why---the command that make issues has -lbigloogc-2.7a -lbigloo_s-2.7a and my documentation tells me that linking against the dynamic library version is the default in OS X, so I really don't understand what's going on here.

4: But very kludgy.

Latest in the Quest for Speed

I found an interesting post which claims that Chicken is a bit faster than Gambit on floating-point code. There's a paper by Brad Lucier which talks about using Gambit to make a PDE solver, which would seem to refute this (Chicken doesn't achieve C speeds on numerics), but I suspect that his sparse matrix-vector multiply was memory-bound, so the speed of the code didn't matter too much.

In any case, by my measurements, Bigloo is much faster than either Chicken or Gambit on floating-point code, so it's still my choice. I would love to find a syntax-case implementation for it, but define-macro is probably just as good for my uses. I posted a message about my attempts (and failures) getting dynamic loading to work with Bigloo. If I could just make that work, it would make my development much easier (it's a real pain to have to recompile the REPL every time I want to define a new class or whatever).

Gambit Isn't Fast Without (declare (unsafe))

Gambit, nice as it is about loading compiled code into an REPL, isn't fast numerically unless you compile using (declare (unsafe)). (At least it wasn't in my recent quick tests of some floating-point code.) Bigloo is much better at eliminating type checks and producing fast numerical code, even in safe mode. So, I guess it's back to Bigloo, because I don't want to have to worry about blowing things up when I misuse parts of my system. Bummer---I was really enjoying the syntax-case macros and easy-to-load code from gambit.

Gambit-C

I've just discovered a new scheme implementation which may be even better than Bigloo for my needs: Gambit-C. Gambit runs on Mac OS X, and compiles to C, just like Bigloo. However the compilation strategy is a bit different. Bigloo aims for roughly idiomatic C code (and relies on the C compiler for many optimizations), while Gambit really uses C as a portable assembly language (and therefore rarely cares about the C compiler optimizations). It looks like (just from reading results of google searches, etc) they are about equally fast in terms of numerical computation. However, Gambit has (from my point of view) a real advantage: loading dynamic libraries works on OS X! Therefore, I can compile my code, and load it into a running REPL---in Bigloo I had to recompile the REPL, and restart it to take advantage of newly compiled code. If the code changes, Gambit will compile it to XXX.o2, and XXX.o3, etc. Each time it's recompiled, the new object file is loaded, and the new functions replace the old ones---no need to stop the REPL process and restart. It feels much more like Common Lisp, but it will satisfy my advisor's need to have something which he can link into a pre-existing C library---the best of both worlds.

More Bigloo Macros

Regarding my previous post about Bigloo Macros and the REPL, I have realized that it's vital to export the macro-expander function from the module in which the macro is defined. Not doing this prevents recognizing the expander when (eval '(define-expander ....)) is evaluated. (Maybe this was obvious to everyone else, but I didn't realize it for a while.)