There isn’t anything exciting below but minor tweaks are good for the soul.

• metabang-bind goes to 0.6.2

  • Found and fixed a glitch in the destructuring code (don’t call package-name unless you’re sure you have a symbol :)).

• CL-Markdown goes to 0.9.2

  • Improved docstring printing in the HTML
  • Added a bit more documentation

• Cl-Graph goes to 0.8.4

  • Brought files up to whatever my current sense of stylistic conventions is
  • Added a few more tests (there are still some that fail).

Here’s what I listened to last week while doing a lot of driving…

• Technometria: Exploiting Online Games - Gary McGraw (co-author of Exploiting Software) talks about MMORPGs, cheating, social situations, etc. Fun and intriguing; not too deep.

• The Singularity: A Period Not An Event - Rod Brooks (of iRobot — did you know that they make a gutter cleaning robot now…) speaks sensibly about the putative singularity. Finally.

• David Heinemeier Hansson gives the Rails Keynote - As a Lisper, I find the success of things like Ruby and Ruby on Rails really maddening (smile). They’ve done a great job tuning nice ideas into successful tools and getting those tools out into the world. From where I stand, Lisp continues to do wonderfully at building successful tools but we’re far less skilled (as a community) in getting these out into the world!

Happy hearing.

Jeff Darcy has interesting (and opinionated) things to say about High-Performance Server Architectures. He focuses on four nasties that cause the most pain:

• Data copies
• Context switches
• Memory allocation
• Lock contention

I enjoyed this and learned a bunch (but will I remember it? I hope so; that’s why I made myself take notes!). The only strikingly Lispy bit is about Memory Allocation and he isn’t really thinking about gc.

Data Copies

Aim for zero-copy but don’t be crazy; remember that even hashing is a sort of copy. Use indirection.

Combing through code and then making it twice as complicated just to get rid of that last few data copies is usually a waste of time that could be better spent in other ways.

Context switches

The #1 cause of context switches is having more active threads than you have processors

Duh! The C10K paper by Dan Kegel is a good resource.

The simplest conceptual model of a multi-threaded event-driven server has a queue at its center; requests are read by one or more “listener” threads and put on queues, from which one or more “worker” threads will remove and process them.

But if you implement this model too faithfully, you’ll asking for mucho context-switches so you need to be more clever. The general idea of separating a complex task into multiple stages is as old as C.A.R. Hoare’s Communicating Sequential Processes and Matt Welsh has another more modern (though not necessarily ideal) approach in SEDA.

Memory allocation

Dealing with memory is a very common thing for applications to do and the generality of allocators supplied by the OS or VM is generally less efficient than they could be. Thus there are three suggestions:

1. pre-allocaiton
2. lookaside lists
3. reduce contention by using multiple private lookaside lists (e.g., per-thread or pre-processor).

Lock contention

Locking is hard because you need to wend your way between the Charybdis of fine-grained locking and the Scylla of locking that is too coarse. To manage, you need a “mental map of tour program’s locking” consisting of

• one axis representing code (e.g., into stages)
• and another representing data (e.g., into separate resources)

This forms a grid delineating (ideally) separate data in separate stages where they ought to be no contention unless two requests share the data in the same stage. Given this, you should be able to spread your locks around in a reasonably good way and then tune things by collecting contention statistics, etc.

Other stuff

Lots!

and lastly:

Figure out what the important high-level operations in your code are, and time them under different conditions. This is not quite the same as traditional profiling; it’s about measuring design elements, not actual implementations. Low-level optimization is generally the last resort of someone who screwed up the design.

Tamas Papp suggested that bind should be able to handle arrays nicely. For example:

(bind ((#(a b) #(1 2)))
  (+ a b))
==> 3

I liked the idea and also wanted to see how extensible I had managed to make bind so I added some tests and this method:

(defmethod bind-generate-bindings
    ((kind array) variable-form value-form
     body declarations remaining-bindings)
  (let ((array-size (array-total-size variable-form))
    (gvalue (gensym "value")))
    `((let* ((,gvalue ,value-form)
        ,@(loop for i below array-size
         for var = (row-major-aref variable-form i)
         unless (eq var nil) collect
           `(,var (row-major-aref ,gvalue ,i))))
      ,@(bind-macro-helper
         remaining-bindings declarations body)))))    

A binding is a pair of variable-form and value-form. Skipping some details, the bind-generate-bindings method gets called and dispatches on the kind of binding. In this case, we need to pull apart the array and generate a let* method to hold each inner variable. The (unless (eq var nil) ...) form lets us easily skip over bits of the array (think of using nil in loop destructuring). In practice, a use of it might be

(bind ((#2a((a nil)
            (nil b)) my-array))
 (format t "~%Sum of the diagonal is: ~a" (+ a b)))

I think it’s handy. Thanks Tamas (and let me know if you have any more ideas!).

I recently listened to Rich Hickey’s LispNYC talk on Clojure. WoW. I’m going to let the talk and some other blogger’s comments stand in for anything I might try to add but I will say that this is fun and inspiring to hear. I’m excited for Clojure, for JVM / .NET targeting, for Lisp and for the computer geekiness in all of us.

• Stewart Sierra heard the talk in person and has a great overview on his weblog.
• Debasish Ghosh also reacts very positively and has some nice additional things to say.

So if you have an MP3 player and a few hours to spend, do yourself a favor ad start listening!

Thanks to Dan Muller, Andy Cristina and Attila Lendvai, ASDF-Install should now play more nicely on Windows systems. The change log has the details but here is the summary:

Summarizing: On Windows we will now use *GNU-TAR-PROGRAM* and *SHELL-SEARCH-PATHS* just like on UNIX. We still assume Cygwin and still run cygpath on the tar arguments. We no longer use *CYGWIN-BASH-PROGRAM*, but we still honor *CYGWIN-BIN-DIRECTORY*. Are these changes 100% backwards compatible? No, some failure cases will fail differently. But out-of-the-box success, using Cygwin, and without customization, is more likely.

Because of the magnitude of these changes, I’ve setup an unstable darcs repository. You can get these using

darcs get http://common-lisp.net/project/asdf-install/asdf-install-unstable

there is also a tarball at

http://common-lisp.net/project/asdf-install/asdf-install-unstable/asdf-install.tar.gz

the tarball is signed with my GPG key.

Please enjoy and let me know if anything seems amiss (let me here is it all works for you too so that I can move these changes into the stable repository…).

Thanks,

Does anyone still use the Common Lisp “advise” functionality? If you’re not familiar with it (which probably means you’re not using it!) you can think of advise as a pre-CLOS way of attaching additional code to your functions dynamically so that bar will run before foo (or after foo or even around foo). As an example of what you could do with advice, imagine if the Common Lisp trace facility didn’t exist… If you had advice, you could easily create your own trace!

I’ve got some code for a portable advice hanging around from long ago and I keep wondering whether or not I should dust it off and make a project out of it… (I hate to add work for the common-lisp.net maintainers unless I have too).

(if a weblog doesn’t have any out-links, does it exist?)

What is the advice facility? The LispWork’s Glossary says that

Advice Facility A facility that allows you to modify the behavior of an existing function by attaching named pieces of advice.

This is somewhat like telling you that iambic pentameter is “having or relating to five iambs”. (update: To be fair, LispWorks has a nice section on their advice facility elsewhere on their website. It’s not, however, linked from the glossary! Go figure).

The Franz documentation says that

The advice facility allows you to affect the action of a function by adding code before, around or after the evaluation of forms defining a function. You may do this with interpreted or compiled functions without changing the source code or having to recompile. This facility is particularly useful in cases where you have compiled code but do not have access to the source code (for example, in a commercial application).

which is a pretty good summary. Franz actually has two implementations, an old one called ‘advice’ and a new one called function wrapping. The big advantage of the later over the former is that with advice “the act of encapsulationg a function produces a new, non-eq function object that is installed at the point of the function spec.” What does this mean?

This means that for example if #’foo were encapsulated, but had been previously captured and stored in a variable to be later funcalled, the funcalled function does not execute the encapsulating functionality. In practical terms, it means that it is impossible to trace a function object by anything other than its name, and it is thus also impossible to trace a function which cannot be accessed via a function spec (usually a symbol, but possibly a list; see def-function-spec-handler). Also, an internal function such as #’(flet foo bar) which is traced does not retain its trace status when the parent function (i.e. #’foo, in this case) is redefined either by defun, compile, or loading a compiled file.

There’s been a lot more interest in a portable advice facility than I would have guessed so I’ll try to start dusting sooner rather than later. I’ll also write up a few of the things that I’ve used it for (or seen it used for) to better whet people’s appetites.

Thanks to KrewinK for sending in a patch to let moptilities’s copy-template function correctly handle class allocation slots. Thanks to the patch and a few fixes in the actual tests, moptilities now passes all of its tests!

I’ve also updated ASDF-Binary-Locations to version 0.2.5. The only code change is that I’m exporting implementation-specific-directory-name. I’ve also added a user’s guide.

I was reading Justin James TechRepublic weblog and came across this statement:

… a few rock stars might be able to blast out some insanely awesome system in three months using Lisp, but it will typically take a dozen average programmers 18 months to trudge through it in Java, VB.NET, or C#.

This feels like a familiar sentiment to me (I’ve been out of the “trenches” for a while but that was certainly the sort of thing I used to hear bruited about). It’s part of the “we’d like to do it in Lisp (or Scala or whatever) but we need to be sure that we can hire people…” meme.

But really, is Lisp hard? Is it harder than doing OO right? (Interestingly, James has another blog post complaining that most people are doing OO wrong). Lisp does require different skills than Java or C#. At least it does if you want to take advantage of what makes Lisp Lisp and use macros to their fullest. It’s also true that the lack of tools and libraries can be problematic (yes, there are libraries but nothing like what you can find in some other languages (and yes, many of those other libraries are junk but … that’s not my point so I’m going to ignore it) — at least I’m honest :) ).

I think the “Lisp is hard” idea is general bunk. It’s different and looks funny (but, hey, if you can stomach C++, what can’t you take!) but getting started in Lisp is no harder that any other language. In fact, its easier to get started in Lisp than it is in programs that require a separate compilation step.

> (defun hi-there () (print "hello world"))
==> #<function hi-there>
> (hi-there)
"hello world"
==> "hello world"

Still, it’s going to take a lot of work of some sort to start convincing people otherwise. Perhaps part of the problem is that Lispers like hearing the meme because we can pat ourselves on the back and say “that’s right, I Lisp, I smart. Me super-coder.”