13 December 2005

Bitten By the Hygiene Bug

I've noticed that I haven't posted here for a while---been really busy working on the code for the paper on n-body integrators. I'm finally running some real "experiments" (the numerical kind), and am discovering the joy of tight binary orbits in the middle of a star cluster (hint: lots of small steps for the binary = lots of roundoff error over cluster timescales). Hopefully I've got that fixed up now, but you never know until it breaks :).

Anyway, here's a real life example of why hygiene matters. Consider the following code:

(define (vector-n-smallest vector n <)
   (let* ((store (vector-copy vector 0 (+ n 1))))
      (insert-sort! store <)
      (do-range (i (+ n 1) (vector-length vector))
         (vector-set! store n (vector-ref vector i))
         (insert-sort! store <))
      (vector-copy store 0 n)))
Hmm, my custom comparison function keeps complaining it's being fed integers rather than frobs! What's going on?

Turns out that I had defined do-range without macro hygiene, and it's expansion involves the symbol <. It's not rocket science, but it is a good example of why you should care about hygiene. (Note that this problem is "solved" in Common Lisp---I think---because the passed in comparison function lives in the value of the symbol <, while the CL::< numerical comparison lives in the symbol-function of the symbol <, so you have to use a funcall to get the custom comparison while (< ...) does the usual CL thing. Hygiene, on the other hand, just does the right thing.)

22 November 2005

Bigloo 2.7a Released

In case you hadn't noticed: Bigloo 2.7a was released yesterday (or maybe over the weekend). You can get my OS X package for it if you don't like to compile things yourself.

20 October 2005

Continuations in Unfriendly Environments

I'm sure many people already have seen the paper (and probably attended the conference), but have a look at this paper from Greg Pettyjohn, John Clements, Joe Marshall, Shriram Krishnamurthi, and Matthias Felleisen! It describes a technique that can be used to implement continuations in unfriendly environments where stack inspection is not available (i.e. portable ANSI C, JVM, .NET, etc.) The great thing about the technique (as opposed to, say, the CPS conversion in Chicken) is that most functions can still use the standard calling conventions and runtime stack of the host language or environment.

Here's my (probably weak) understanding of how it works: when a continuation is captured a chain of exceptions gets propagated up the stack. At each "stop" on the way up the stack, an exception handler cons-es up a closure which will continue the function whose frame is active at that point. When the exception reaches the top level of the stack, the frames are packaged up as the continuation. Activating the continuation simply calls the closures in order to resume the saved computation where it left off. All the implementation needs to provide is some way to unwind the stack, stopping at each frame along the way. In C, it would be a longjmp/setjmp; in Java it would be catch/throw; in .NET catch/throw.

There is an example with more code (and less math) here, too.

17 October 2005

Exciting and Depressing

Ever have something happen that's terribly exciting, as long as you don't think about it too much? I just had a mental breakthrough with my integrators: I realized a way to think about what's going on that makes everything perfectly clear. Then I implemented this in about 174 lines of Bigloo code. "That's great!" you say. I agree completely. Unfortunately, if I can't quite remain lost in the moment, I think of two mildly depressing things:
  1. Why didn't I think of this sooner? I've spent the last few months playing around with these things, and then I figure out that pretty much everything I was doing before was wrong in the space of two days---and it's (in retrospect) obvious!
  2. Someone else has to have had this thought! I'm almost scared to do a literature search tomorrow (time for bed now)---I'm sure I'll find half a dozen papers describing this technique and explaining why it's not better than what everyone else has been doing, etc (worst case) or that other people have had this thought and it's got promise for future implementations (much better, but still a bit of a bummer).

However, I'm still terribly excited! 174 lines!! It's so simple!!! This is great!!!!

13 October 2005

Shared State in PLT Scheme's Swindle

I just spent about six hours debugging a version of my integrator that runs in PLT Scheme (got tired of not having the class creation options available at runtime in Bigloo) because of a shared state problem. If you define a Swindle class with the slot option :initvalue <value> then <value> is only evaluated once; every object of this class shares the same value. Not good if you want to have mutable state! The thing to use is :initializer <thunk>. I wish there was an :initform slot option, like in the real CLOS.

30 September 2005

SRFI-32 for Bigloo

I needed some fast vector sorting routines for my n-body integrator. To that end, I've packaged up the withdrawn SRFI 32 for Bigloo. The SRFI is technically withdrawn, but even half of a half-standard is better than none (and this way I don't have to delve into sorting myself---I'm sure Olin Shivers is way better at that than I am).

29 September 2005

Wolfram Tones

Here's an idea sure to appeal to nerds everywhere (including me): cellphone ring tones generated by cellular automata. Though it's a bit hard to stomach how much Wolfram exaggerates his contributions to the science of complexity and the study of cellular automata and related computational algorithms, it's still an interesting idea. Discovered through this New York Times article.

In addition to the unpleasantness of enriching someone who seems---shall we say---reluctant to acknowledge other contributions to the study of cellular automata, I am too cheap to just drop $2 on a cellphone ring tone. However, I do think my custom tone is pretty neat.

Srfi-4 Vectors in Bigloo

I just thought I'd drop a quick note about srfi-4-like vectors in Bigloo. For those who don't know, srfi-4 specifies an interface for homogeneous numeric vector types. For those of us who do numerical computing, this is a big win---boxing a double, for example, every time you reference or store into a vector is just impossibly inefficient if you want to get anything numerical done. But, Bigloo doesn't have an implementation of srfi-4---so how is it that I'm such an enthusiastic user?

Here's how: Bigloo allows for the definition of typed vectors. Typed vectors are vectors that only hold one of the native Bigloo types. (Unfortunately, this cannot be done for user-created types like classes.) For example, the module clause

(module f64vector
   (type (tvector f64vector (double)))
   (eval (export-all)))
Creates homogeneous vectors of doubles labeled by the type ::f64vector and functions to access these vectors. Nearly the entire srfi-4 interface is created; the only warts are
  1. The make-f64vector procedure requires two arguments; the initializing value is not optional.
  2. The f64vector procedure is not created.
Note that the created ref and set! procedures are typed correctly, which helps immensely in the type analysis of the remainder of your program (I hardly ever end up putting in typing information except at procedure arguments).

In fact, the Bigloo source distribution contains an example of using tvectors, but it's not a widely known fact, so I thought I would mention it here for posterity's sake.

By the way, supposedly the type system in Bigloo can identify when vectors are used homogeneously, and automatically convert them to tvectors. That's nice in theory, but in practice it only works on vectors that are used in only a few procedures within the same module, and even then it has trouble if you use the vectors in some strange way or forget to explicitly initialize them in the creation statement. For things like numerical computing, where the f64vectors can get passed around through the entire program it's hopeless---just use a tvector.

24 September 2005

Economic Integration

Check this out (from an article in the New York Times):
A school board election will take place in October. While the board has continued to endorse economic integration, some supporters worry that that could change one day.

"It's not easy and it can be very contentious in the community," said Walter C. Sherlin, who retired two years ago as an associate superintendent. "Is it worth doing? Look at 91 percent at or above grade level. Look at 139 schools, all of them successful. I think the answer is obvious."

This is great! A public initiative backed by research into student performance that seems to be wildly successful (40%->80% grade-level scoring among black students over the last decade, 79%->91% overall). And, unlike race-based busing, this doesn't suppose that your performance is the result of your race (which is an internal factor that you have no control over) but rather your economic situation (which is an external factor over which you have some control). I hope that the parents of the 2.5% of children who are involuntarily bussed don't become such a vocal minority that this stops happening.


Just read an interesting paragraph in an essay about the internet bubble:
The prospect of technological leverage will of course raise the specter of unemployment. I'm surprised people still worry about this. After centuries of supposedly job-killing innovations, the number of jobs is within ten percent of the number of people who want them. This can't be a coincidence. There must be some kind of balancing mechanism.
From Paul Graham. He's probably right, but it seems possible that the mechanism is people bitching about being unemployed and holding society effectively at ransom to find jobs for them to do. (i.e. a country that has 50% unemployment usually has social unrest, too, and some sort of reforms are proposed so that the rich people and companies can get back to making money without that pesky mob at the front gate.) Probably there's someone who studies this; it would be interesting to know whether my idea is right.

22 September 2005

The Flu

Not that I have it (or usually get it, being young and in good shape). However, this article in the New York Times today points out that vaccines are "apparently ineffective" in preventing the flu for elderly people:
The fact that the vaccine study showed that inoculations have had only a modest effect in the elderly is particularly worrisome, because this is a group that tends to suffer high rates of complications and deaths from the disease and vaccination is the standard practice. In people over 65, the vaccines "are apparently ineffective" in the prevention of influenza, pneumonia and hospital admissions, although they did reduce deaths from pneumonia a bit, by "up to 30 percent," the study says.

"What you see is that marketing rules the response to influenza, and scientific evidence comes fourth or fifth," Dr. Jefferson said. "Vaccines may have a role, but they appear to have a modest effect. The best strategy to prevent the illness is to wash your hands."

Being young, I would like to see mention of the effectiveness of the vaccine in younger people (i.e. whether it's worth it for me to get one).

On a related note, how do people who don't believe in evolution at all (i.e. something made the world exactly as it is now) cope with the idea of viruses sharing genetic material and becoming resistant?

20 September 2005

Bigloo Finalization Library

I've just posted a library which allows finalization of Bigloo objects before they're reclaimed by the GC. You can view the documentation and download the library here.

Manuel provided the hint to look at doc/README.finalizer, but I didn't like the way finalization was done in the example there, and it seemed a bit out of date. So, I wrapped up my own finalization library. I worry that I may have gone overboard (both in the documentation and in the care I've taken with license), but I have had a lot of trouble using poorly-documented libraries in the past, so hopefully this will be helpful to more people because I've spent some time writing out good documentation.

Bigloo 2.7a Beta Mac OS X Package

I've just posted a Mac OS X installer for the latest (Sept 20) beta of Bigloo. You can read more about it here. It's been quite a lot of work to compile Bigloo properly on OS X (which you can read about on the Bigloo mailing list), and Manuel has been a huge help. Many of these things are happening automatically now, but I figured that, since I've already put in the effort, I might as well make things as easy as possible for those following. Instead of putting a few #defines into some header files, typing ./configure --lots-of-options and then compiling, you can now just grab the disk image and Bigloo is only 4 clicks away. Enjoy!

14 September 2005

Bigloo Internals, More Dynamic Linking and Shark...

without much Scheme-y stuff in sight.

Over the past few days Manuel Serrano has been helping me improve the speed of my n-body integrator by changing the behavior of Bigloo's threading when run in single-threaded mode on OS X. There is a function in runtime/Clib/cthread.c called denv_get which, I assume, gets some representation of the current dynamic environment. Since the cthread.c functions are really stubs for a more complete thread implementation denv_get is really short---it compiles to about 5 instructions (including prelude and post-lude) on my machine. Unfortunately, in the beta version of 2.7a from 18Aug denv_get was installed as the function to get the dynamic environment through a function pointer, preventing its inlining by the C compiler. My n-body code was spending upwards of 50% of its time in denv_get (remember: 5 instructions!). Manuel had me insert #define BGL_DYNAMIC_ENV_ALTERNATE 1 in runtime/Include/bigloo.h from the newest beta of 2.7a (12Sept)1. This apparently allowed for inlining of denv_get in single-threaded mode---presto: faster by a factor of two.

I'm mentioning this for two reasons: 1. Manuel is really cool to take so much time to help me with my performance issue. 2. If you use OS X and Bigloo 2.7a beta, you should definitely get the 12Sept version and insert #define BGL_DYNAMIC_ENV_ALTERNATE 1 into runtime/Include/bigloo.h!

In other news, Maunel has used my patch fixing a problem with the build script which was preventing the building of the compiler with shared libraries even when the --sharedcompiler=yes option was provided to configure. This makes it much simpler to build a bigloo linked against dynamic libraries on OS X. All you have to do is make sure to provide --sharedcompiler=yes (and, if you like, --sharedbde=yes) to configure. Then edit the Makefile.config, changing the following variables:

  • LD=gcc
  • LDFLAGS=-dynamiclib -single_module
  • LDLIBS=-lbigloogc-2.7a
That's it---no restarting the make in the middle after changing options or anything. (I can't remember whether this works for building the Development Environment, but I do remember that if it fails it's pretty clear what to fix.)

Finally, I'd just like to put in a plug for SHARK, Apple's profiler that comes with OS X. You can profile a running process, obtaining a call graph with who-calls-who and elapsed time in each call path, see the memory usage pattern (and page faults), look at the disassembled object code with color-coded hot spots and pipeline stalls, and get helpful advice like using the reciprocal square root estimate assembly instruction rather than the math library sqrt when you don't need super accuracy. I can't imagine the amount of effort it would have been to do the profiling I've done today and yesterday with gprof! Bigloo works quite well with SHARK, since it outputs relatively idiomatic C code (though you might have to de-mangle function names with bigloo-demangle); other compilers like Gambit-C (C used as assembler) or Chicken (C used in CPS) wouldn't give you code that SHARK would help with much.

Eventually I'll get back to Scheme-y things (I haven't written a good macro in days!), but I thought it would be good to get down some of the things I've been doing lately.

1: You can get the 12Sept beta of 2.7a from INRIA's FTP Site or its mirror at U. Nice.

07 September 2005

Hybrid Cars and Pork

In today's New York Times:
"I would like to get more hybrids out of our system because I do think it's something that is here to stay," William Clay Ford Jr., chairman and chief executive of Ford Motor, told reporters in Detroit on Aug. 23. Last fall, Ford introduced a hybrid version of its Escape sport utility vehicle, the first of several hybrids planned.

At the same time, the energy bill signed by President Bush on Aug. 8 effectively gave a break to American manufacturers by extending what could be a tax credit of as much as $3,400 per car to purchasers of the first 60,000 hybrids sold by a company. The credit phases out after that. Toyota sold more than 60,000 hybrids in the first six months of this year, so the tax law seems intended to help General Motors and Ford.


The new tax law "seems to benefit those who haven't done anything in the area of hybrids until now," Christopher Richter, a CLSA Asia-Pacific Markets auto analyst, said in Tokyo. "And it seems to penalize those who have been pioneers."

No kidding! And note that Ford has leased its hybrid technology from Toyota---no R&D costs for them. The only good news on the pork-ridden government front is that California is going to make a distinction between hybrids based on their environmental quality:
In early August, California regulators started to distinguish between fuel-efficient hybrids and "muscle" hybrids, the high-powered versions that save little gasoline. Of the seven hybrid models now on sale in the United States, owners of the Honda Civic, Honda Insight and Toyota Prius can qualify for decals allowing them to drive alone, rather than with two or more passengers, in highway commuter lanes. That reward is not extended to four hybrids not rated as exceptional energy savers: Honda Accord, Ford Escape, Toyota Highlander and Lexus RX 400h.

06 September 2005

Bigloo Tracing

I just saw that I'm being syndicated at planet scheme. Looks like it hasn't quite made it to planet lisp-like number of contributors (or probably readers). But, in honor of the nevertheless auspicious occasion I plan to say something scheme-y after the past few days of political posts (which, by the way, have probably alienated my one reader---sorry dude, but I call it the way I see it :) ).

I just noticed a very useful thing for debugging Bigloo: tracing macros! The way this works is you put (with-tracing n 'foo ...) declarations around your code and if the debug level (set by -g[2..4]) when compiling is greater than n you get a trace which outputs the symbol 'foo indented according to how many previous traces are active. You can also issue (trace-item . items) commands which display the items given at the appropriate level (though this only seems to work when the debug level is greater than 2). Want the debugging info to go away? Just lower the debug level (or---even easier---increase n) when you compile that module. It doesn't sound like much, but the indentation and (trace-item ...) are leaps and bounds better than the (print ...) statements I was constantly un-commenting and commenting out. I now have (with-tracing ...) scattered through my code---it's great.

Two warts if you use Bee in Emacs and dynamically load your compiled modules:

  1. If an error occurs within the dynamic extent of the (with-tracing ...) form, the indentation doesn't always reset, so sometimes after several errors your traces are walking off the edge of the screen. (I don't know why a simple dynamic-wind doesn't cut it here:
       (lambda () (increase-trace-indentation!))
       (lambda () 'do something)
       (lambda () (decrease-trace-indentation!)))
    but it sometimes doesn't work.)
  2. Sometimes the prefix of the trace data contains shell formatting characters that kind of mess up the output:
      |  |  |  |  |  |  |--+make-advancer-lambda
      |  |  |  |  |  |  |  |--+ calculate-new-dt
    Sometimes calling (trace-margin-set! "") clears this stuff, but sometimes it doesn't.

Anyways, even with the warts it's a pretty useful feature. Enjoy.

Update: In fact, you can get rid of all the cruft at the beginning of these lines by setting the parameter bigloo-trace-color to #f: (bigloo-trace-color-set! #f). Then everything works great within emacs.

What are they doing?

Yet more incompetence from our lovely executive branch: FEMA ignores an offer of help from Chicago.
Even before the storm hit the Gulf Coast on Monday, he [Chicago Mayor Daly] said, the city's Office of Emergency Management and Communications had contacted emergency response agencies in Illinois and Washington.

In the event of a disaster, the city offered to send 44 Chicago Fire Department rescue and medical personnel and their gear, more than 100 Chicago police officers, 140 Streets and Sanitation, 146 Public Health and 8 Human Services workers, and a fleet of vehicles including 29 trucks, two boats and a mobile clinic.

"So far FEMA has requested only one piece of equipment {ndash} [sic] a tank truck to support the Illinois Emergency Response Team, which is already down there," Daley said. "The tank truck is on its way. We are awaiting further instructions from FEMA."

Even before the storm hit the Gulf, Chicago was offering to help. Too bad FEMA doesn't believe in planning; they just lock the door after the horse has left the barn. You can read more about their obstructionism in the name of coordination from a list here. (Not that I'm opposed to coordination by one agency, but I think that you should have to demonstrate that you aren't a total fuck-up before being allowed to tell other helpful people what not to do. In any case, I'd have a lot more confidence in a press-release asking First Responders not to help unless requested by state and local authorities if it actually came from the state and local authorities, not from an ex-horse-show overseer and political supporter of the President.)

05 September 2005

Labor Day

From a NY Times Op-Ed:
...But because the French work 35 hours a week, Americans sneer, forgetting that in many years French workers have a higher productivity rate than their American counterparts - proof that you can work better by working less.

Americans also forget that going to work every day is often more a chore than a pleasure. You seem more and more disillusioned about work: only a third of you say that you love your jobs. In such conditions, it's not surprising that you spend on average two hours of your workday ... not working. Answering personal e-mail messages, shopping online, playing computer games or chatting with co-workers ... it's so much more pleasant than working, really.

Only two hours a day? Hell, I might play two hours of foosball on a good (bad?) day. Then there are the days when I work 18 hours straight. It's just the nature of an intellectual job---when you're hot, you're hot, and when you're not you play around. Fortunately, I have an employer who realizes the capricious nature of Physics research---I wish everyone was so lucky in their "boss".

P.S.---I love my job! Guess I'm part of the lucky 1/3.

04 September 2005

Science and Politics in the NY Times Magazine

There's an interesting article in the New York Times Magazine today about politics and science. The article discusses possible reasons for why most (nearly all?) scientists think the Bush administration has done an abysmal job listening to and properly reacting to scientific developments and studies. It's probably a good factual review of several instances where science was suppressed/changed for political ends in the administration, and also a good description of John Marburger's role and why he thinks he's doing good for science.

Unfortunately, I think the writer missed the point. He focused nearly entirely on the way that scientific results are incorporated into reports and policy decisions. This is surely an important component of how the administration treats science, but it's not the whole story. What I think really upsets scientists---me included---about this administration is not the specific suppression of a study in an EPA report (though this is pretty bad); it's their seeming disregard for facts and reason and their willingness to take public positions in support of pseudo-scientific, ideologically driven theories. This complaint would be there even if they had never, ever, pressured an employee to change the mention of scientific results.

Several examples of what, generally, pisses scientists off:

  1. The president's stance on Creationism (aka Intelligent Design). I know that most of the country believes this, but I would hope that the president would want to lead the country, not follow it in its ignorance. (By the way, the article doesn't mention Creationism at all.)
  2. The president's position on climate change, and the response of the administration to the overwhelming scientific consensus that CO2 is causing global warming. (This is discussed in the article, but more from a point of view of suppressing the relevant information in EPA reports and whatnot.)
  3. The rationale for tax cuts in the face of growing deficits and no significant spending cuts. You can't have your missile-defense cake and eat it, too, people. This isn't science, but it's such a glaring stupidity that scientifically minded people can't help but notice it.
The common theme here is: the article focused on the details of policy---reports, committee selections, etc---but I think scientists are more interested in the projected public attitude of the administration. I think that most scientists would be willing to forgive some pretty big omissions and commissions on the details front if you didn't get the feeling that the people in change had wooden heads. I mean, if a guy really believes in Creationism and tells the public that it's on an equal scientific footing with evolution, does it really matter if he gets all the facts right in an EPA report? Admittedly, the public pronouncements of the President don't have as much of an immediate policy effect as the committee work and rules published by the agencies, but it's the public persona that people follow; Bush and the upper levels of the administration are basically telling the country that there's two worlds: the scientific, rational one and the one that does whatever you want. And you don't have to listen to the scientists because you can just live in the other world. That attitude does more damage than the specific examples in the article, and that's what really pisses scientists (well, at least me) off.

Update: I totally forgot about issues like AIDS in Africa, but it's another one. Again the message: it doesn't matter what you scientists discover about the best way to prevent AIDS---we don't have to listen to it. It's not getting the facts wrong; it's ignoring the facts you know to be true.

Update II: I didn't realize that this article was still available, or I would have linked to it above. Favorite quote (the article is by Pulitzer Prize winner Ron Suskind):

In the summer of 2002, after I had written an article in Esquire that the White House didn't like about Bush's former communications director, Karen Hughes, I had a meeting with a senior adviser to Bush. He expressed the White House's displeasure, and then he told me something that at the time I didn't fully comprehend -- but which I now believe gets to the very heart of the Bush presidency. The aide said that guys like me were ''in what we call the reality-based community,'' which he defined as people who ''believe that solutions emerge from your judicious study of discernible reality.'' I nodded and murmured something about enlightenment principles and empiricism. He cut me off. ''That's not the way the world really works anymore,'' he continued. ''We're an empire now, and when we act, we create our own reality. And while you're studying that reality -- judiciously, as you will -- we'll act again, creating other new realities, which you can study too, and that's how things will sort out. We're history's actors . . . and you, all of you, will be left to just study what we do.''
Really! This kind of shit goes over like a lead balloon with physicists (and probably most other scientists, too). Again, the outrage isn't in the details---it's in the fundamental assumptions of the people in power.

Google Helper for Mac

Google Mail Helper is available for the Mac! I'm now using it, but I'm wracked with guilt over my desertion of the wonderful soul who created Gee. Poor Lloyd---and because it never rains but when it pours, Lloyd looks to have lost a bunch of photos in a Linux drive crash. Bummer.

I guess people may still use Gee for other Atom feeds, but mail helper has much better features for Gmail (not least that it registers Gmail as the default mail handler so that accidentally clicking on those mailto: tags doesn't open up Mail.app but instead takes you to Gmail).

Chlorine Tidbits

Some sites regarding chlorine exposure in swimmers are interesting, since I just got back into swimming. The information is somewhat sketchy on the web (and most sites are trying to sell you something---as in http://www.karinya.com/chlorine.htm), but there are some seemingly-authoritative pages. This review points out that there isn't a very good understanding of the effects of chlorine and related compounds in the air above a pool. (It does also point to the well-known beneficial effect of swimming on asthmatic children, which is reassuring.) Excessive chlorination may lower the pH of the water, leading to problems with dental enamel. A good summary of papers is here.

In any case, I think swimming is pretty fun, and it keeps me in great shape. I'm not worried overly about the chlorination, since it seems like the health benefits far outweigh the risks, but I'm not saying that I wouldn't be happy if my pool used something else.

02 September 2005

Sean Carroll's Blog

I just discovered Sean Carroll's Blog (he blogs along with several other physicists at cosmicvariance). Great stuff! I particularly enjoyed the political cartoons #1 and #2. Of course, the physics is neat, too.

I haven't hung out much with Sean (just a bit when I was visiting Chicago as a prospective grad student), but now I can follow him vicariously and wonder what Chicago would have been like. (Not that I'm unhappy here at MIT; just that it's interesting how different Sean and Ed are.)

31 August 2005

bdepend Problems Solved

I just figured out why I was having problems with module initialization. I also solved a long-standing mystery in my own mind: what the hell are those .mco files that Bigloo produces all the time? In fact, .mco files are module checksums---reproduced (I assume) every time the interface to a module changes. So, bdepend does, in fact, know about dependencies on (import ...) modules; it just uses the .mco files to compute these dependencies, not the .scm files themselves! Very smart, since the .scm file could change in a way that doesn't affect the interface and therefore wouldn't require recompilation of every module which (imports ...) said .scm file.

So the solution to my problem is to strike the -fno-mco flag on bdepend, and add a -fmco flag when bigloo is compiling a module. Ta-da! No more inconsistent module initialization, and no more complete system recompiles. Wonderful!

P.S.--Maybe someone (me?) should write something up for inclusion into the Bigloo manual about bdepend and mco files. There's very little in there right now (hence my problems making this work).

19 August 2005

Bigloo GC and SIGINT

I've noticed in my work with Bigloo that sometimes when a process running from the REPL is sent SIGINT (C-c), the GC hangs in the function GC_lock. I don't know if it's related to dynamic-load-ing on OS X, or whether this is a general problem, and the GC is not very resistant to SIGINT. I suppose it could also be my optimization options on the compiled C code (I'm optimizing pretty heavily for my numerical work). I'm just putting it down here so I don't forget to file a bug report if I ever figure out the cause.

16 August 2005

Inconsistent Module Initialization

I've been getting a lot of errors like the following lately from Bigloo:
*** ERROR:body:Inconsistent module initialization
Module `body' is inconsistently initialized by module `__nbody'.
At least of the two modules must be recompiled (see also -unsafev option).
and I think I finally realized what the problem is. I have a bunch of modules which are all separately compiled, and then put together by a "heap" file as described in the chapter from the Bigloo manual on libraries, but the Makefile I'm using doesn't think that the heap file depends on the separately compiled files. So, the code the heap file uses to initialize the modules it imports becomes stale whenever the exported interface of these modules changes.

This problem arises because bdepend doesn't think that import commands in the module header create dependencies. Normally this is true---bigloo modules can, in general, be separately compiled---but in the case of a heap file, it won't work. (It also won't work for the file which has the -dload-sym option on the command line if you're doing dynamic loading.) The heap file needs to be recompiled if any of the modules it imports have changed their exports. Probably it's easiest to just add a command to the Makefile so that the heap file depends on all the source files in the project; this will recompile it more than necessary, but it's easy and the heap file doesn't take very long to compile.

EDIT: Actually, this problem afflicts all files, not just the heap ones. If module A imports module B, then module A needs to be recompiled if module B's exported interface changes. I've submitted a bug report about bdepend not recognizing this to the bigloo list.

15 August 2005

Perserverance: 1 Static Linking: 0

Finally! I've figured out how to make dynamic linking of module files work with bigloo on Mac OS X! Details follow (though I should warn you that this is a total hack right now).

There are two steps involved: 1. Alter the bigloo compilation process so it builds the proper dynamic libraries and 2. Make sure that the executable bigloo is linked against the dynamic libraries, and not the static ones.

#1: The way I know to do #1 is to edit Makefile.config after running ./configure [whatever-options-you-want]. Scroll through Makefile.config and change the following options:

LDFLAGS=-dylib -single_module
-lc -lSystem.B -ldylib1.o -lm -lc -lgcc
Now issue the command
The make process will halt with an error! That's OK. Re-edit the Makefile.config file by changing the LDLIBS variable to LDLIBS=-lbigloogc-2.7a -lc -lSystem.B -ldylib1.o -lm -lc -lgcc (you should alter the "-2.7a" to whatever version of bigloo you're compiling). The compile should now finish. You will have both libbigloo***-***.a and libbigloo***-***.dylib in the lib subdirectory.

#2: Though it seems like you should be able to ensure that the bigloo executable is linked against the dynamic libs by using the --sharedcompiler=yes flag to the configure script, this doesn't work. However, the fix is easy enough: edit the dboot target of the file comptime/Makefile. There will be a section looks something like

doboot: $(O_OBJECTS)
 if [ "$(SHRD_COMP)" = "yes" -a -f $(BOOTLIBDIR)/lib$(LIBRARYNAME)_s$(VERSION)$(SHAREDSUFFIX) ]; then            gc=-lbigloogc-$(RELEASE);            if [ "$(GCCUSTOM)" = "no" ]; then       gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) -L $(BOOTLIBDIR) -lbigloo_s-$(RELEASE) $$gc $(EXTRALIBS);  else            gc=$(BOOTLIBDIR)/libbigloogc-$(RELEASE).a;            if [ "$(GCCUSTOM)" = "no" ]; then               gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) $(BOOTLIBDIR)/libbigloo_s-$(RELEASE).a $$gc $(EXTRALIBS);  fi
 @ echo "$(BIGLOO_DEST)$(EXE_SUFFIX) done..."
you want to make it look like:
doboot: $(O_OBJECTS)
 if [ "$(SHRD_COMP)" = "yes" -a -f $(BOOTLIBDIR)/lib$(LIBRARYNAME)_s$(VERSION)$(SHAREDSUFFIX) ]; then            gc=-lbigloogc-$(RELEASE);            if [ "$(GCCUSTOM)" = "no" ]; then       gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) /tmp/bigloo2.7a/lib/2.7a/libbigloo_s-2.7a.dylib /tmp/bigloo2.7a/lib/2.7a/libbigloogc-2.7a.dylib  else            gc=$(BOOTLIBDIR)/libbigloogc-$(RELEASE).a;            if [ "$(GCCUSTOM)" = "no" ]; then               gc=-lgc;            fi;     $(CC) $(EXTRA_LD_OPT) $(CFLAGS) -o $(BIGLOO_DEST) $(O_OBJECTS) $(BOOTLIBDIR)/libbigloo_s-$(RELEASE).a $$gc $(EXTRALIBS);  fi
 @ echo "$(BIGLOO_DEST)$(EXE_SUFFIX) done..."
(The difference is that we have explicitly included the libbigloo_s-2.7a.dylib and the libbigloogc-2.7a.dylib libraries on the $(CC) command line---including the full paths!) Now delete the old executable at bin/bigloo (relative to the unpacked bigloo directory) and re-make. Check with otool -L bin/bigloo to make sure that the bigloo executable really links against the dynamic libraries. The output of my otool looks like
SYDNEYPACIFIC-THREE-FORTY-ONE:/tmp/bigloo2.7a farr$ otool -L bin/bigloo
        /tmp/bigloo2.7a/lib/2.7a/libbigloo_s-2.7a.dylib (compatibility version 0.0.0, current version 0.0.0)
        /tmp/bigloo2.7a/lib/2.7a/libbigloogc-2.7a.dylib (compatibility version 0.0.0, current version 0.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 71.1.3)

Now, to make a loadable bundle, issue the following command (this example comes from the Bigloo manual, section 13.1 under the dynamic-load procedure documentation):

MACOSX_DEPLOYMENT_TARGET=10.3 gcc -bundle -undefined dynamic_lookup -o mod.bundle mod1.o mod2.o
This command replaces the ld -G -o lib.so mod1.o mod2.o command from the manual. You can now load the bundle with
(dynamic-load "mod.bundle")
from the REPL. You need the MACOSX_DEPLOYMENT_TARGET=10.3 so that it doesn't complain about the -undefined dynamic_lookup option; if you don't have 10.3, you can probably get away with -undefined suppress, but I haven't tried it.


1: I think it would be really neat if someone found a way to make all of this ad-hoc makefile-modifying happen automatically as part of the Bigloo build process on Mac OS X. I think it'll take someone more knowledgeable about the internals of the build process, but I may have a look at it later.

2:What happened here is that libbigloo requires the garbage collection library, so we have to include it in the LDLIBS variable. We can't put it in there at the beginning because the GC library doesn't exist yet, so we have to wait for the build to fail and then put it in. This is necessary because Mac OS X makes a distinction between dynamic libraries (which are treated like ELF static libraries for link purposes, but like ELF shared (or dynamic) libraries for loading at runtime) and loadable bundles (which are treated like ELF shared libraries all the time). In English: you have to have all the symbols for a .dylib at link time, though the libraries which contain them won't be loaded until run time, while you don't need the symbols for a .bundle until you use them at runtime.

3:And I don't understand why---the command that make issues has -lbigloogc-2.7a -lbigloo_s-2.7a and my documentation tells me that linking against the dynamic library version is the default in OS X, so I really don't understand what's going on here.

4: But very kludgy.

09 August 2005

Latest in the Quest for Speed

I found an interesting post which claims that Chicken is a bit faster than Gambit on floating-point code. There's a paper by Brad Lucier which talks about using Gambit to make a PDE solver, which would seem to refute this (Chicken doesn't achieve C speeds on numerics), but I suspect that his sparse matrix-vector multiply was memory-bound, so the speed of the code didn't matter too much.

In any case, by my measurements, Bigloo is much faster than either Chicken or Gambit on floating-point code, so it's still my choice. I would love to find a syntax-case implementation for it, but define-macro is probably just as good for my uses. I posted a message about my attempts (and failures) getting dynamic loading to work with Bigloo. If I could just make that work, it would make my development much easier (it's a real pain to have to recompile the REPL every time I want to define a new class or whatever).

Gambit Isn't Fast Without (declare (unsafe))

Gambit, nice as it is about loading compiled code into an REPL, isn't fast numerically unless you compile using (declare (unsafe)). (At least it wasn't in my recent quick tests of some floating-point code.) Bigloo is much better at eliminating type checks and producing fast numerical code, even in safe mode. So, I guess it's back to Bigloo, because I don't want to have to worry about blowing things up when I misuse parts of my system. Bummer---I was really enjoying the syntax-case macros and easy-to-load code from gambit.

07 August 2005


I've just discovered a new scheme implementation which may be even better than Bigloo for my needs: Gambit-C. Gambit runs on Mac OS X, and compiles to C, just like Bigloo. However the compilation strategy is a bit different. Bigloo aims for roughly idiomatic C code (and relies on the C compiler for many optimizations), while Gambit really uses C as a portable assembly language (and therefore rarely cares about the C compiler optimizations). It looks like (just from reading results of google searches, etc) they are about equally fast in terms of numerical computation. However, Gambit has (from my point of view) a real advantage: loading dynamic libraries works on OS X! Therefore, I can compile my code, and load it into a running REPL---in Bigloo I had to recompile the REPL, and restart it to take advantage of newly compiled code. If the code changes, Gambit will compile it to XXX.o2, and XXX.o3, etc. Each time it's recompiled, the new object file is loaded, and the new functions replace the old ones---no need to stop the REPL process and restart. It feels much more like Common Lisp, but it will satisfy my advisor's need to have something which he can link into a pre-existing C library---the best of both worlds.

04 August 2005

More Bigloo Macros

Regarding my previous post about Bigloo Macros and the REPL, I have realized that it's vital to export the macro-expander function from the module in which the macro is defined. Not doing this prevents recognizing the expander when (eval '(define-expander ....)) is evaluated. (Maybe this was obvious to everyone else, but I didn't realize it for a while.)

01 July 2005

MzScheme and GSL Random Numbers

I just used the foreign interface to PLT Scheme to wrap up the GSL random number generators. I've posted the code here; it hasn't been extensively tested, but I think it'll work OK.

As an aside, I really, really like the foreign interface to PLT Scheme---it's by far the best one I've ever dealt with. This paper explains the rationale. Definitely The Right Way to do it.

25 June 2005

Bigloo, Macros and the REPL

I've just spent quite a while trying to make Bigloo do what I want with macros. What I want:
  1. To be able to write macros within a module which I can use while compiling code in that module or other modules which import the macro-module.
  2. To be able to write macros which are imported into an repl which incorporates the module in which they are written.
Doing this has required a bit of trickery. (The difficulty is that I want the macros at 3 times: 1. When the compiler is processing the module in which they are defined. 2. When the compiler is processing other modules which use them. 3. When the repl is up and running.)

The solution is to write modules with macros like this:

(module with-gensyms
   (eval (export-exports))
   (export (with-gensyms-expander x e)))

(define (with-gensyms-expander x e)
   (match-case x
      ((with-gensyms (?sym) . ?body)
       (e `(let ((,sym (gensym)))
              ,@body) e))
      ((with-gensyms (?sym1 . ?rest) . ?body)
       (e `(let ((,sym1 (gensym)))
              (with-gensyms ,rest ,@body)) e))))

(eval '(define-expander with-gensyms with-gensyms-expander))
And use them in other modules like this:
(module f64vector
   (eval (export-all))
   (import (with-gensyms "with-gensyms.scm")
           (do-macros "do-macros.scm"))
   (load (with-gensyms "with-gensyms.scm")
         (do-macros "do-macros.scm"))
   (type (tvector f64vector (double)))
   (export (f64vector::f64vector . inits)
           (with-f64vectors-expander x e)))

(define (f64vector::f64vector . inits)
   (let* ((n (length inits))
          (v (make-f64vector n 0.0)))
      (let loop ((i 0) (list inits))
         (if (null? list)
                (f64vector-set! v i (car list))
                (loop (+fx i 1) (cdr list)))))))

(define (with-f64vectors-expander x e)
   (match-case x
      ((with-f64vectors ?vecs (<- . ?body))
       (with-gensyms (result n i)
          (e `(let* ((,n (f64vector-length ,(car vecs)))
                     (,result (make-f64vector ,n 0.0)))
                 (do-times (,i ,n)
                    (let ,(map (lambda (sym)
                                  `(,sym (f64vector-ref ,sym ,i)))
                       (f64vector-set! ,result ,i
                                       (begin ,@body))))
                 ,result) e)))
      ((with-f64vectors ?vecs . ?body)
       (with-gensyms (i n)
          (let* ((vector-syms (map (lambda (sym) (cons sym (gensym))) vecs))
                  (lambda (term)
                     (match-case term
                        ((?result <- . ?body)
                         `(f64vector-set! ,(cdr (assoc result vector-syms))
                                          (begin ,@body)))
                        (?- term)))))
             (e `(let ((,n (f64vector-length ,(car vecs))))
                    (let ,(map (lambda (sym)
                                  `(,(cdr (assoc sym vector-syms)) ,sym))
                       (do-times (,i ,n)
                          (let ,(map (lambda (sym)
                                        `(,sym (f64vector-ref ,sym ,i)))
                             ,@(map process-body-term body))))) e))))))

(eval '(define-expander with-f64vectors with-f64vectors-expander))
(Note that this module uses two macro-modules---one of which is not listed above---to define a third. I'm sorry for the confusing example, but it's what I have handy.)

How does this work? Here's my understanding:

  1. The compiler begins processing the with-gensyms.scm file. The compiler compiles it into with-gensyms.o which contains a module initialization routine that evaluates all the top-level commands in the module whenever the module is initialized---this ensures that the (eval '(define-expander with-gensyms ...)) runs whenever the module is initialized in compiled code (i.e. when running a custom repl).
  2. The compiler processes f64vector.scm. It sees the (load (with-gensyms ...)) command in the module header, and interprets the file with-gensyms.scm, installing the with-gensyms macro before processing the code in f64vector.scm. The f64vector.scm file compiles into f64vector.o which contains a module initialization routine which will initialize the with-gensyms module first (because bigloo sees that it is required by a (import (with-gensyms ...)) in the f64vector module header) whenever the f64vector module is required from compiled code (i.e. within the main routine which implements the repl).
Thus, when compiling the code, all the macros are interpreted, but when running the code (i.e. within the repl), the macro-expanders are actually compiled! Any module which needs the macros at compile time should use the (load ...) form in the module header, and any module which wishes to install these macros at run time should use the (import ...) form. Whew! The repl in question can be implemented by
(module repl
   (import (with-gensyms "with-gensyms.scm")
    (do-macros "do-macros.scm")
    (f64vector "f64vector.scm"))
   (main main))

(define (main argv)

It took me a long enough time to figure this out (and I wouldn't have figured it out without some suggestions from jg malecki (see this message and its antecedents---thanks jg) that I thought I should post the explanation here so that other people don't have to go through the same difficulties that I have.

23 June 2005

Making Scheme behave like Fortran 95

I just coded up a macro which makes Bigloo scheme behave more like Fortran 95 with respect to vectors. Usage:
(with-vectors (a b c)
   (a <- (+ b c)))
(with-vectors (a b c)
   (<- (+ a b c)))
(with-vectors (a b c)
   (a <- (+ b 2))
   (print "I can insert statements, too.")
   (c <- (+ a b)))

The second form above returns a fresh vector whose elements are the sums of the corresponding elements in a b and c. The length of the new vector is the same as the length of a. In all cases, the number of iterations of the body is the length of a (this isn't quite the right thing to do---that would be to make the number of iterations be the minimum of the lengths of a b and c or to throw an exception if a b and c are not the same length). The macro follows:

(define-expander with-vectors
   (lambda (x e)
      (match-case x
         ((with-vectors ?vecs (<- . ?body))
          (with-gensyms (result n i)
             (e `(let* ((,n (vector-length ,(car vecs)))
                        (,result (make-vector ,n)))
                    (do-times (,i ,n)
                       (let ,(map (lambda (sym) `(,sym (vector-ref ,sym ,i)))
                          (vector-set! ,result ,i
                                       (begin ,@body))))
                    ,result) e)))
         ((with-vectors ?vecs . ?body)
          (with-gensyms (i n)
             (let* ((vector-syms (map (lambda (sym) (cons sym (gensym))) vecs))
                     (lambda (term)
                        (match-case term
                           ((?result <- . ?body)
                            `(vector-set! ,(cdr (assoc result vector-syms))
                                          (begin ,@body)))
                           (?- term)))))
                (e `(let ((,n (vector-length ,(car vecs))))
                       (let ,(map (lambda (sym)
                                     `(,(cdr (assoc sym vector-syms)) ,sym))
                          (do-times (,i ,n)
                             (let ,(map (lambda (sym)
                                           `(,sym (vector-ref ,sym ,i)))
                                ,@(map process-body-term body))))) e)))))))
It requires a completely obvious (I hope) do-times macro, and also with-gensyms (a lisp favorite):
(define-expander do-times
   (lambda (x e)
      (match-case x
         ((do-times (?i ?n) . ?body)
          (with-gensyms (nn nn-1)
             (e `(let ((,nn ,n))
                    (let ((,nn-1 (-fx ,nn 1)))
                       (do ((,i 0 (+fx ,i 1)))
                           ((=fx ,i ,nn-1) ,@body)
                           ,@body))) e))))))

(define-expander with-gensyms
   (lambda (x e)
      (match-case x
         ((with-gensyms (?sym) . ?body)
          (e `(let ((,sym (gensym)))
                 ,@body) e))
         ((with-gensyms (?sym . ?rest-syms) . ?body)
          (e `(let ((,sym (gensym)))
                 (with-gensyms ,rest-syms ,@body)) e)))))
Nifty, huh? I also have one for my f64vectors (as described in my last post one can make vectors of doubles---the natural behavior from bigloo approximates SRFI 4):
(define-expander with-f64vectors
   (lambda (x e)
      (match-case x
         ((with-f64vectors ?vecs (<- . ?body))
          (with-gensyms (result n i)
             (e `(let* ((,n (f64vector-length ,(car vecs)))
                        (,result (make-f64vector ,n 0.0)))
                    (do-times (,i ,n)
                       (let ,(map (lambda (sym)
                                     `(,sym (f64vector-ref ,sym ,i)))
                          (f64vector-set! ,result ,i
                                       (begin ,@body))))
                    ,result) e)))
         ((with-f64vectors ?vecs . ?body)
          (with-gensyms (i n)
             (let* ((vector-syms (map (lambda (sym) (cons sym (gensym))) vecs))
                     (lambda (term)
                        (match-case term
                           ((?result <- . ?body)
                            `(f64vector-set! ,(cdr (assoc result vector-syms))
                                          (begin ,@body)))
                           (?- term)))))
                (e `(let ((,n (f64vector-length ,(car vecs))))
                       (let ,(map (lambda (sym)
                                     `(,(cdr (assoc sym vector-syms)) ,sym))
                          (do-times (,i ,n)
                             (let ,(map (lambda (sym)
                                           `(,sym (f64vector-ref ,sym ,i)))
                                ,@(map process-body-term body))))) e)))))))

22 June 2005

Typed Vectors in Bigloo

I learned at the beginning of the month how to make a typed vector (also called monomorphic, I think) in Bigloo scheme. I had known that Bigloo would automatically unbox vectors of doubles or integers if it could infer the type of them, but you can't guarantee this. This kept me from using Bigloo for my numerical projects---it's essential that such vectors be unboxed, and I didn't want to leave it up to the type-inferencing engine. However, at the beginning of the month a message came out on the Bigloo mailing list (it's no longer in the archive on gmane, and it came around before the inria archive, so this may be the only place you can find it): the way to make a typed-vector is demonstrated in the following example from Manuel Serrano:
(module foo
   (type (tvector array-of-int (int)))
   (export (foo::array-of-int ::int ::int))
   (main main))

(define (foo len init)
   (make-array-of-int len init))

(define (main x)
   (let ((v::array-of-int (foo 10 20)))
      (print v)
      (array-of-int-set! v 5 6)
      (print (array-of-int-ref v 5))))

Apparently you can also find more examples of this in the recette/vector.scm module in the source for Bigloo.

01 June 2005

All the President's Men...

I still remember reading All the President's Men when I was maybe in middle school. My parents were coming of age during that era, and reading the book made me feel like I was equally involved in the scandal. Learning yesterday that Deep Throat has finally come forward brought all that to mind again. I hope someone writes a book about him---it sounds like he is maybe a bit conflicted about his role, and that would make another interesting story.

31 May 2005

Almost beaten to the punch!

The shark-profile package has only been out for a few days, and I've been scooped! I'm glad that someone with more stamina and skill than I have has fixed the interrupt/floating-point interaction for SBCL on OS X. I've been using SB-SPROF a bit tonight, and it seems to be stable now. In consolation, I suppose there may still be people who wish to use shark instead of SB-SPROF, and I got several good weeks use out of SHARK-PROFILE, so all is not lost :).

27 May 2005

Profiling lisp with SBCL and Mac OS X

I just posted some code for profiling with shark (part of the CHUD toolkit on Mac OS X) and SBCL. I've been writing some fancy n-body integrators using common lisp for a research project, and wanted to profile the code. SBCL comes with a profiler (sb-sprof), but it doesn't work well---at all---on Mac OS X (see the reply to this message of mine for a description of the problem). So, I cooked up this code (with some help from Juho Snellman) which lets me use shark for profiling. It's quite a hack, but it does the job.

22 May 2005

Go New York Times!

Check this out:
While the proposal to close the case was ultimately rejected by senior officials, documents show that the inquiry was at a virtual standstill when an article in The New York Times on March 4, 2003, reported that at least one of the prisoner's deaths had been ruled a homicide, contradicting the military's earlier assertions that both had died of natural causes. Activity in the case quickly resumed.
I'm getting tired of our side being the bad guy so often. Why is the army not more thorough about disciplining those responsible for this kind of crap? At best, I would say it's apathy, or a desire to protect your own---however flawed they may be---and at worst, I would say it's because those in charge actually think this sort of thing is OK. Either way, it's not making me very happy. This also shows the media at its best---the army wasn't doing anything, and the NY Times got them moving. Let's hear it for the fourth estate!

21 May 2005

First Post

I've finally broken down and started a blog. Typically for me, it came impulsively as I was searching for information on the best way to rip books on tape (CD) in iTunes, and stumbled on http://aldoblog.com/blog/471. Since we're flying to California, Oregon and then the Netherlands this summer, we've got some 30 hours of time on planes, and thought we'd get some books on tape for Rachel's iPod. The above link gives the details, but the short version:
  • Set the import file format to AAC, 64kbit Stereo Rate, Mono Sampling
  • Use the Advanced menu option to consolidate the tracks into one (so you don't have 50-60 tracks for one book on the iPod).
  • Once the file is ripped, change its type from m4a to m4b so that iPod will automatically maintain your listening position rather than resetting to the beginning of each track. You can use the MakeBookmarkable script, or (I think) just change the extension of the imported file to m4b.
I intend to record such tibits here in the future so that they're easy to find.