mult.ifario.us

I cut over to perpubplat about a week ago, so it's worth a quick check to see if I met my goal of getting a few more nines.

On the left, we have an every-30-minutes response time chart for the Typo version of this site, and on the right, the perpubplat version. (The charts are from the free monitoring service from mon.itor.us.) The Typo configuration used Apache2 with mod_fcgid (see my earlier post on the subject) configured for seven processes with a maximum 60-minute lifetime; the perpubplat configuration uses Apache2 with mod_fastcgi configured for a single process that runs on 50 lightweight Haskell threads internally. (N.B.: The graphs use different scales on the y-axis.)

The graph hints that things are better, but some quick text processing on server logs make the difference more explicit. Here's an analysis of the response codes from the most recent log file for perpubplat:

$ head -n 1 multifarious-combined.log | awk '{print $4}'
[07/Jan/2008:00:32:50
$ ^head^tail
[11/Jan/2008:15:03:50
$ awk '{print $9}' multifarious-combined.log | sort | uniq -c | sort
      1 400
      2 206
     73 304
   1572 302
   2976 404
   5574 301
  14772 200

No 500s. (Most of the 404's are comment spammers trying to hit old URLs for comments.) Here's the same analysis for Typo from a week back in December of last year (2007):

$ zcat mult.ifario.us-access.log.3.gz | head -n 1 | awk '{print $4}'
[10/Dec/2007:19:49:06
$ ^head^tail
[19/Dec/2007:14:28:17
$ zcat mult.ifario.us-access.log.3.gz | awk '{print $9}' | sort | uniq -c | sort
      5 206
      5 400
     23 503
     30 302
    237 404
   1098 301
   2259 500
   9319 304
  20885 200

A change from one-ish nines (-log₁₀ (33861 - 2282)/33861 ~ 1.17) to 100% uptime is a positive change, and the CPU trace for the virtual server suggests that the perpubplat configuration uses a tiny fraction of the machine resources of the Typo configuration.

Io has been getting some nice buzz lately for its reflection capabilities and minimality. Steve Dekorte's blog is also a good read, although software wisdom is vastly outnumbered by links to cool architecture and design.

My favorite thing about the language is still its concurrency annotations. Way back in 2004, when I was surveying Actor languages, I posted about Io's support for making methods non-blocking via annotating the invocation. (This is similar to Haskell's par annotation.) Other than activeCoroCount being replaced by activeActorCount, it looks like the annotations remain the same:

• foo bar calls the bar method on the object foo synchronously, returning a value.
• foo @bar calls the method asynchronously, immediately returning a future.
• foo @@bar calls the method asynchronously, immediately returning nil

I haven't had a chance to try out the deadlock detection, but coming up with an example should be an interesting exercise.

Somehow, one of the bifold closet doors in the house got broken. The door has a hollow construction with pivota at the top and bottom set into 3/8" holes in relatively small pieces of wood (maybe 2x2s?). Whatever happened broke a chunk of wood out of the piece of wood on the bottom of the door in the worst possible place, taking half the hole with it and pushing the front and back surface of the door out. To make matters worse, we discovered this when the door fell out and attacked my wife when she tried to open it...

Replacing the door wasn't an option, since it is part of a bank of four such doors that all match and also match other doors in the house, and the doors lack any sort of identifying marks as to the manufacturer. My best guess was Jeld Wen, but there is a huge variety of different dimensions, proportions, and styles. So I set out to repair it as a weekend project.

Here was the process:

1. Craft a piece of wood to roughly match the missing tetrahedron.
2. Reglue the front/back/side of the door onto the remaining wood with carpenter's glue. Clamp to shape and wipe off excess; let dry.
3. Glue the custom-made hunk of wood into place with some Gorilla Glue. It's the right glue for the job because it expands as it dries, ensuring a tight fit. The only odd thing is that you need to moisten the wood before gluing. I clamped the door into the right shape with some scrap 1x4 to ensure that it didn't force the door apart. Let cure.
4. Use a razor blade to ensure that the bottom surface of the door is flat and smooth.
5. Construct a guide for the 3/8" hole by drilling through a piece of scrap lumber. This is important, as there's no way to hand-hold the drill steady enough to drill the half a hole that you need.
6. Use a pipe clamp (cheap and useful!) to clamp the guide in place on the bottom of the door and a piece of scrap lumber on the top of the door.
7. Drill the hole using the guide, remove the clamp, and tap in the sleeve for the bottom pivot with a mallet.
8. Install the door.

That was fun.

Setting up perpubplat to talk to a web server takes a bit of work, but once you've got the code, bootstrapping a post can be done with just ghci.

The perpubplat configuration in the Darcs repository expects to find posts and comments stored under /tmp/content and drafts stored under /tmp/drafts. (Clearly, a different configuration would be desirable for production use...) Create those directories and then a file /tmp/drafts/hello-world.draft with the following contents:

title: Hello, World!
tags: one two three

--- START BODY ---

<p>Hello, World!</p>

--- END BODY ---

Next, load up the code in ghci:

$ cd perpubplat/perpubplat/src
$ ghci Blog.BackEnd.IoOperations Blog.Model.Entry \
  Blog.BackEnd.ModelTransformations Blog.FrontEnd.Presentation \
  Blog.FrontEnd.Views
GHCi, version 6.8.2: http://www.haskell.org/ghc/  :? for help
Loading package base ... linking ... done.
[ 1 of 13] Compiling Blog.Widgets.FlickrBadge ( Blog/Widgets/FlickrBadge.hs, interpreted )
[...]
Ok, modules loaded: Utilities, Blog.Constants, Blog.Model.Entry, [...]
*Blog.BackEnd.IoOperations> :set prompt "> "
> :m + Blog.BackEnd.ModelTransformations Blog.FrontEnd.Views Blog.FrontEnd.Presentation
> m <- boot
> (s,m') <- ingest_draft m "hello-world.draft"

The ingest_draft function does the work of adding the post parsed from the draft to the internal data structure (some maps and lists bundled together) and storing the new post in the content directory. You can find it in /tmp/content/hello-world/content.ppp.

For a little more exercise, here's how to render the "all posts" view containing the new post:

> assemble_view (All Nothing) m'
Loading package xhtml-3000.0.2.1 ... linking ... done.
"<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" [...]

Which is the XHTML that would be displayed in the browser. As to the content of the internal data model for the post we just created:

> :m + Blog.Model.Entry
> s
"hello-world"
> post_by_permatitle m' s
Item {internal_id = 0, kind = Post, title = "Hello, World!" [...]

And that's all there is to it.

I succumbed to some mild peer pressure, posted a Darcs repository, cleaned-up the Cabal files for perpubplat, and fixed a few minor bugs. If you have darcs installed, you can pull down a copy of the source code — a grand total of less than 1500 lines according to SLOCCount — with:

darcs get http://datapr0n.com/repos/perpubplat

You can also browse the repository at the same URL, but Apache will serve those files to you as text/x-haskell, i.e., your browser probably won't display them as plain text in the browser. The Darcs-maintained revision history is here.

The README file contains instructions on building, but if you've already got GHC 6.8.2 and FastCGI libraries installed, it's as easy as:

$ cd perpubplat
$ cd perpubplat
$ chmod +x Setup.lhs
$ ./Setup.lhs configure && ./Setup.lhs build && sudo ./Setup.lhs install
$ cd ../perpubplat_servlet
$ ./Setup.lhs configure && ./Setup.lhs build

The FastCGI handler is dist/build/perpubplat.fcgi/perpubplat.fcgi.

My first startup's offices were in a classic Chicago loft with minimal build-out, and by minimal I mean that no wall went more than half-way to the fifteen-foot ceilings. It was a great arrangement in many ways (see, e.g., Dick Costolo on the subject), but it was challenging in situations where privacy was necessary, like hiring or compensation discussions, sales calls (when you don't necessarily want your potential customer listening to a heated foosball match), or Board business. There have been many times when I needed or wanted to take a private call while in a public space or on-site visiting a customer.

And this brings me to the idea of a cellphone booth. Why not have low-tech (in fact, no-tech) booths for cellphone users that can be dropped into airports, building lobbies, cafes, and offices like a "portapotty" on a construction site? It probably isn't that expensive to construct a booth (plywood, some acoustical foam), and the obvious twist is to brand it and place print advertising on the inside and outside to create a revenue stream. I can't claim that the idea is original, as I used a cellphone booth in the United lounge in Copenhagen once upon a time, but I haven't seen one anywhere else. (Also, a similar idea was suggested back in the 1960s...)

I started thinking about replacing Typo back in October of 2006, and my home-brew project to do so is at the point where it's usable as a replacement. In fact, I cut over this morning, so you're looking at it right now.

The current implementation represents an investment of around 60 hours of learning and hacking time spent on Apache, FastCGI, Atom, XHTML, and Haskell.

Why Not X?

It is reasonable to ask "Why not use X?" where reasonable values for X include default blogging tools like Wordpress or Roller, a more recent version of Typo, some language other than Haskell (like Erlang or OCaml or Scala), or an existing Haskell framework like HApps or Hope. The short answer is that I rolled my own because I wanted to roll my own.

Tooling and Methodology

I used the default tooling stack of Emacs (the Aquamacs flavor) with haskell-mode, GHC, and Darcs for revision control. My workflow loop was equivalently simple: I worked in Emacs until I thought that something might work and then loaded a module into ghci to experiment with.

Basic Architecture

The basic architecture is straightforward; here are the highlights:

1. Container: FastCGI handler implemented in Haskell that parses request URIs to determine how to respond. Apache with mod_fastcgi is configured as described in an earlier post.
2. Storage: Plain text, simple file storage of entities (posts, comments, drafts, etc.) in a human editable, human readable format.
3. Concurrency: An event loop that listens on a Haskell Chan and manages a single in-memory instance of the data. The event loop runs on a lightweight background thread that gets forked when Apache spins-up the FastCGI handler, and the approach is equivalent to the one I used for the sequence number generator experiment.
4. Browser Views: XHTML rendering via the Text.XHtml combinator package.
5. Atom Feeds: A modified version of the lightweight Atom library that I posted on previously.

Early Returns and Open Items

My expectations are met so far. The implementation is deployed on my virtual server at Linode. Some ad hoc benchmarking shows that it will support sustained 50 req/sec loads with 10 or 20 concurrent clients without issue, and that's a stark contrast to Typo's performance of four requests per second (downhill and with the wind). (Benchmarking traffic may well be saturating the network between here and there, so it might even do a bit more.) Turning on profiling shows that most of the time is taken in string concatenation for a page view or a feed. One option would be to set up conditional GET and some basic caching, and switching to Data.ByteString would be another.

There are two open items I'm still thinking about: dynamic content in the browser and comments.

For dynamic content in the browser, i.e., integration via Javascript clients to HTTP APIs, I experimented a bit with in-page widgets that do a bit of DOM rewriting to display fancy badges from del.icio.us or Reddit or shared items from Google Reader, but pages load more slowly, it requires clients to have Javascript enabled, and it's incompatible with XHTML. (The one compromise for the moment is the Flickr montage, since it adds a splash of color.) Instead, I'm planning to add additional background threads to the application to poll del.icio.us, Flickr, and other services via HTTP APIs and then vend cached data to sidebar widgets.

The second item is support for comments, trackbacks, referrers, and the like, and that's just a matter of me deciding how I want to manage the workflow and ensure a good experience for repeat commenters or correspondents (i.e., people who link here).

Open Source?

I will make source code available as a Darcs repository at some point in the near future, but it's not a priority for me. The milestone I'd like to achieve before releasing the source is to have the cabal builds all squeaky clean, and that's not far off. (Right now, things are just built with ghc --make.) Even then, I still wouldn't call it "open source". Calling something "open source", to me, should carry with it an implicit promise of usefulness and fitness for purpose, but as the name might suggest, I intend this to be a personal publishing platform.