Tim Berners Lee talks about the Giant Global Graph (a term that I hope doesn’t have lets… GGG, ugh!).

TBL gets a nice comeback from Anne Zelenka on GigaOM (which sounds like some sort of massive meditation site :)):

The semantic web has always been about computers taking on more processing for us, not about computers allowing us to be more human, which is where the social graph might more naturally aim.

But just like social intelligence is not a subset of academic intelligence, social knowledge and understanding isn’t a subset of the semantic meaning that semantic web technologies like RDF and SPARQL are aimed at representing.

The Semantic Web is a good step towards representing more of what we need to know to help computers help us, but it’s only a step.

Here’s what I listened to last week while doing a lot of driving…

• Technometria: Exploiting Online Games - Gary McGraw (co-author of Exploiting Software) talks about MMORPGs, cheating, social situations, etc. Fun and intriguing; not too deep.

• The Singularity: A Period Not An Event - Rod Brooks (of iRobot — did you know that they make a gutter cleaning robot now…) speaks sensibly about the putative singularity. Finally.

• David Heinemeier Hansson gives the Rails Keynote - As a Lisper, I find the success of things like Ruby and Ruby on Rails really maddening (smile). They’ve done a great job tuning nice ideas into successful tools and getting those tools out into the world. From where I stand, Lisp continues to do wonderfully at building successful tools but we’re far less skilled (as a community) in getting these out into the world!

Happy hearing.

I finally upgraded to Leopard last night… everything is snazzy and nice. I’m happy! (Though I wise I had a Core Duo 2 for 64-bit goodness).

Jeff Darcy has interesting (and opinionated) things to say about High-Performance Server Architectures. He focuses on four nasties that cause the most pain:

• Data copies
• Context switches
• Memory allocation
• Lock contention

I enjoyed this and learned a bunch (but will I remember it? I hope so; that’s why I made myself take notes!). The only strikingly Lispy bit is about Memory Allocation and he isn’t really thinking about gc.

Data Copies

Aim for zero-copy but don’t be crazy; remember that even hashing is a sort of copy. Use indirection.

Combing through code and then making it twice as complicated just to get rid of that last few data copies is usually a waste of time that could be better spent in other ways.

Context switches

The #1 cause of context switches is having more active threads than you have processors

Duh! The C10K paper by Dan Kegel is a good resource.

The simplest conceptual model of a multi-threaded event-driven server has a queue at its center; requests are read by one or more “listener” threads and put on queues, from which one or more “worker” threads will remove and process them.

But if you implement this model too faithfully, you’ll asking for mucho context-switches so you need to be more clever. The general idea of separating a complex task into multiple stages is as old as C.A.R. Hoare’s Communicating Sequential Processes and Matt Welsh has another more modern (though not necessarily ideal) approach in SEDA.

Memory allocation

Dealing with memory is a very common thing for applications to do and the generality of allocators supplied by the OS or VM is generally less efficient than they could be. Thus there are three suggestions:

1. pre-allocaiton
2. lookaside lists
3. reduce contention by using multiple private lookaside lists (e.g., per-thread or pre-processor).

Lock contention

Locking is hard because you need to wend your way between the Charybdis of fine-grained locking and the Scylla of locking that is too coarse. To manage, you need a “mental map of tour program’s locking” consisting of

• one axis representing code (e.g., into stages)
• and another representing data (e.g., into separate resources)

This forms a grid delineating (ideally) separate data in separate stages where they ought to be no contention unless two requests share the data in the same stage. Given this, you should be able to spread your locks around in a reasonably good way and then tune things by collecting contention statistics, etc.

Other stuff

Lots!

and lastly:

Figure out what the important high-level operations in your code are, and time them under different conditions. This is not quite the same as traditional profiling; it’s about measuring design elements, not actual implementations. Low-level optimization is generally the last resort of someone who screwed up the design.

I recently listened to Rich Hickey’s LispNYC talk on Clojure. WoW. I’m going to let the talk and some other blogger’s comments stand in for anything I might try to add but I will say that this is fun and inspiring to hear. I’m excited for Clojure, for JVM / .NET targeting, for Lisp and for the computer geekiness in all of us.

• Stewart Sierra heard the talk in person and has a great overview on his weblog.
• Debasish Ghosh also reacts very positively and has some nice additional things to say.

So if you have an MP3 player and a few hours to spend, do yourself a favor ad start listening!

Nature, Nurture, naturally… From TechRepublic:

According to a new workplace poll from Harris Interactive and Adecco (surveying 1,455 workers), men 18-34 are more likely than women to need to extra thanks from the boss in order to be productive. Apparently, guys older than that don’t need outside praise that much.

It’s impossible to interpret a poll unless you see the questions but it’s fun to poke fun at stuff regardless.

I was trying to do this earlier today and quick Googling didn’t help so I’m adding a blog page in the hopes of increasing the (easily) available knowledge of the planet — wow, sounds important.

To find the OS X software version using the command line (with nothing up your sleeves), use sw_vers. For example

> sw_vers
ProductName:    Mac OS X
ProductVersion: 10.5.1
BuildVersion:   9B18

Is that cool, or what. Probably what but I wasn’t really looking for an answer. This may look like a conversation but it’s really all about me. I’m glad I could be honest. Now remember. Software Version Command Line OS X How to find. Discovering. Getting the software version with the command line in OS X. Is this a hint? A how-to? Just a a bunch of words strung together. How do you put a needle into a word, anyways? Oh, and thanks to Ahmon.

What is the advice facility? The LispWork’s Glossary says that

Advice Facility A facility that allows you to modify the behavior of an existing function by attaching named pieces of advice.

This is somewhat like telling you that iambic pentameter is “having or relating to five iambs”. (update: To be fair, LispWorks has a nice section on their advice facility elsewhere on their website. It’s not, however, linked from the glossary! Go figure).

The Franz documentation says that

The advice facility allows you to affect the action of a function by adding code before, around or after the evaluation of forms defining a function. You may do this with interpreted or compiled functions without changing the source code or having to recompile. This facility is particularly useful in cases where you have compiled code but do not have access to the source code (for example, in a commercial application).

which is a pretty good summary. Franz actually has two implementations, an old one called ‘advice’ and a new one called function wrapping. The big advantage of the later over the former is that with advice “the act of encapsulationg a function produces a new, non-eq function object that is installed at the point of the function spec.” What does this mean?

This means that for example if #’foo were encapsulated, but had been previously captured and stored in a variable to be later funcalled, the funcalled function does not execute the encapsulating functionality. In practical terms, it means that it is impossible to trace a function object by anything other than its name, and it is thus also impossible to trace a function which cannot be accessed via a function spec (usually a symbol, but possibly a list; see def-function-spec-handler). Also, an internal function such as #’(flet foo bar) which is traced does not retain its trace status when the parent function (i.e. #’foo, in this case) is redefined either by defun, compile, or loading a compiled file.

There’s been a lot more interest in a portable advice facility than I would have guessed so I’ll try to start dusting sooner rather than later. I’ll also write up a few of the things that I’ve used it for (or seen it used for) to better whet people’s appetites.

IT Conversations and the Conversations Network keep my brain happy…

• Moira Gunn talks with Sandra Blakeslee about her book the Body Has a Mind of Its Own: How Body Maps in Your Brain Help You Do (Almost) Everything Better. It’s a wonderful conversations about neuroscience and cognitive maps. Blakeslee (a science writer) does a wonderful job conveying the excitement and joy of discovery. Recommended.

• Moira Gunn also has a nice conversation with John Thackera, a designer and writer. His most recent book, In the Bubble, discusses how we can try to break out of our self-imposed design limitations and see a bigger picture. I’m reading the book now and its a fun read with a very European slant on tools and technology. Its inspiring to hear of the many things that people are doing right (there and here in my somewhat benighted USofA). Aside from an occasionally too informal riff, Thackera’s prose flows nicely with a excellent mix of insight, quotations, examples and analysis. Both the interview and the book are recommended.

• I also recently heard a far less interesting interview with Morgan Gillis, the executive director of the LiMo Foundation. I guess the excitement of Mobile Linux and a new cell phone platform just goes through my head…

• Finally, I heard a wonderful interview between Tom Parish (the host) and Jake McKee. McKee used to work at Lego and has gone on to focus on social media and customer interaction: the creation of community in support of selling product. He has some great points to make about how to do this honestly and with integrity — indeed, you can’t succeed in this sort of endeavor unless you loads of both! He comes to the table with great background and experience and really makes his points well. Recommended.

People have often told me I could use some advice. I say “but I use it all the time”. Ba da bing.

Humor aside, the Common Lisp Advice Facility lets you wrap (and unwrap) new code around old code dynamically even if you don’t have the source for the old code. This is an incredible feature for doing all kinds of useful things. (It’s also an amazing feature for tying yourself up in a maze of little passages, all alike. In other words, “with great power comes great power, so be careful.”

It can be nice to know how many times a function has been called, that’s why profilers usually allow you to get call counts! Sometimes, however, going through the trouble of messing with a profiler and getting its information back out in a useful fashion is more trouble that seems necessary. Enter with-call-counts. The complete code is below (though it relies on Franz’s (excellent) fwrapper machinery, the idea is the same).

;; a bizarre hack in search of a dream
;; It's a lot of machinery to count a single functions calls... but it's cool!
(defvar *call-counter* (make-hash-table))

(excl:def-fwrapper call-counter (&rest args)
  (declare (ignorable args)
       (dynamic-extent args))
  (%count-call (%find-call-counting-spot excl::primary-function))
  (excl:call-next-fwrapper))

(defun start-call-counting (function &optional (reset? nil))
  (bind ((spot (%find-call-counting-spot function)))
    (setf (second spot) t)
    (when reset? (reset-call-counting function))
    (excl:fwrap function :call-counter 'call-counter)))

(defun reset-call-counting (function)
  (setf (first (%find-call-counting-spot function)) 0))

(defun stop-call-counting (function)
  (setf (second (%find-call-counting-spot function)) nil)
  (excl:funwrap function :call-counter))

(defun call-counts (function)
  (values-list (%find-call-counting-spot function)))

(defun %find-call-counting-spot (function)
  (setf function (coerce function 'function))
  (bind (((values spot found?) (gethash function *call-counter*)))
    (if found? 
    spot
    (setf (gethash function *call-counter*) (list 0 nil)))))

(defun %count-call (spot)
  (incf (car spot)))

(defmacro with-call-count ((function &key (resetp t)) &body body)
  (with-gensyms (gfunction gresetp)
    `(let ((,gfunction ,function)
       (,gresetp ,resetp))
       (unwind-protect
        (progn
          (start-call-counting ,gfunction ,gresetp)
          (progn ,@body)
          (list (intern ,gfunction :keyword) (call-counts ,gfunction)))
     (stop-call-counting ,gfunction)))))

(defmacro with-call-counts (((&rest functions) &key (resetp t)) &body body)
  (with-gensyms (gresetp)
    (cond ((null functions) `(list :result (progn ,@body)))
      (t `(let ((,gresetp ,resetp)
            (this nil)
            (rest nil))
        (setf this
              (with-call-count (,(first functions) :resetp ,gresetp)
            (setf rest 
                  (with-call-counts (,(rest functions) 
                          :resetp ,gresetp)
                ,@body))))
        (nconc this rest))))))

The with-call-count macro is the heart of the above. Like most macros of this sort, it expands into an unwind-protect and some function calls to turn call counting on and off. The start-call-counting function adds an fwrapper and stop-call-counting removes it. The wrapper itself (created with the excl:def-fwrapper) is simple: it run before the wrapped function and just increments a counter. Finally, the with-call-counts macro expands into multiple applications of with-call-count.

I’m not completely happen with the syntax of the machinery above but I’m happy enough that I’ve used it many times in debugging and testing. It’s nice to be able to:

• ensure that a cache is actually working by counting the number of calls to some inner function with the cache off and on.
• check that some function isn’t being called or that it is!
• check that a binary search is really calling its comparison function the expected number of times.

Because advice makes writing macros like the above easy and because macro like the above make it easy to measure whats happening in your code (without, notice, modifying your code), it’s a big win!