mult.ifario.us

I started thinking about replacing Typo back in October of 2006, and my home-brew project to do so is at the point where it's usable as a replacement. In fact, I cut over this morning, so you're looking at it right now.

The current implementation represents an investment of around 60 hours of learning and hacking time spent on Apache, FastCGI, Atom, XHTML, and Haskell.

Why Not X?

It is reasonable to ask "Why not use X?" where reasonable values for X include default blogging tools like Wordpress or Roller, a more recent version of Typo, some language other than Haskell (like Erlang or OCaml or Scala), or an existing Haskell framework like HApps or Hope. The short answer is that I rolled my own because I wanted to roll my own.

Tooling and Methodology

I used the default tooling stack of Emacs (the Aquamacs flavor) with haskell-mode, GHC, and Darcs for revision control. My workflow loop was equivalently simple: I worked in Emacs until I thought that something might work and then loaded a module into ghci to experiment with.

Basic Architecture

The basic architecture is straightforward; here are the highlights:

1. Container: FastCGI handler implemented in Haskell that parses request URIs to determine how to respond. Apache with mod_fastcgi is configured as described in an earlier post.
2. Storage: Plain text, simple file storage of entities (posts, comments, drafts, etc.) in a human editable, human readable format.
3. Concurrency: An event loop that listens on a Haskell Chan and manages a single in-memory instance of the data. The event loop runs on a lightweight background thread that gets forked when Apache spins-up the FastCGI handler, and the approach is equivalent to the one I used for the sequence number generator experiment.
4. Browser Views: XHTML rendering via the Text.XHtml combinator package.
5. Atom Feeds: A modified version of the lightweight Atom library that I posted on previously.

Early Returns and Open Items

My expectations are met so far. The implementation is deployed on my virtual server at Linode. Some ad hoc benchmarking shows that it will support sustained 50 req/sec loads with 10 or 20 concurrent clients without issue, and that's a stark contrast to Typo's performance of four requests per second (downhill and with the wind). (Benchmarking traffic may well be saturating the network between here and there, so it might even do a bit more.) Turning on profiling shows that most of the time is taken in string concatenation for a page view or a feed. One option would be to set up conditional GET and some basic caching, and switching to Data.ByteString would be another.

There are two open items I'm still thinking about: dynamic content in the browser and comments.

For dynamic content in the browser, i.e., integration via Javascript clients to HTTP APIs, I experimented a bit with in-page widgets that do a bit of DOM rewriting to display fancy badges from del.icio.us or Reddit or shared items from Google Reader, but pages load more slowly, it requires clients to have Javascript enabled, and it's incompatible with XHTML. (The one compromise for the moment is the Flickr montage, since it adds a splash of color.) Instead, I'm planning to add additional background threads to the application to poll del.icio.us, Flickr, and other services via HTTP APIs and then vend cached data to sidebar widgets.

The second item is support for comments, trackbacks, referrers, and the like, and that's just a matter of me deciding how I want to manage the workflow and ensure a good experience for repeat commenters or correspondents (i.e., people who link here).

Open Source?

I will make source code available as a Darcs repository at some point in the near future, but it's not a priority for me. The milestone I'd like to achieve before releasing the source is to have the cabal builds all squeaky clean, and that's not far off. (Right now, things are just built with ghc --make.) Even then, I still wouldn't call it "open source". Calling something "open source", to me, should carry with it an implicit promise of usefulness and fitness for purpose, but as the name might suggest, I intend this to be a personal publishing platform.

Herein part six in my hobby project to rewrite my personal publishing software in Haskell. In part five (and its addendum), I roughed-out a persistence and concurrency model for the back-end. The next two pieces are rendering content (which will be done programmatically using the Text.XHtml.Strict module; that's a separate post) and integrating with a web server via FastCGI. This post covers FastCGI integration for Lighttpd and Apache2 in the form of smoke-testing a simple FastCGI handler.

Units of Concurrency

For the concurrency model that I plan to use in the actual application, a single OS process is critically important, as multiple processes wouldn't be aware of who was doing what within the other processes. Multiple active threads within that one process are fine. Most web-based systems use a single process as a concurrency pinch point, but that process is usually the database as opposed to the web application.

Haskell in the form of GHC provides two flavors of concurrency, which I'll refer to as forkIO and forkOS (after the functions forkIO and forkOS, respectively). The forkIO flavor uses Haskell's internally managed, lightweight threads, and the forkOS flavor uses threads from the underlying operating system. (For some perspective on what an OS thread really means, take a look at my post on SMP Erlang on Mac OS.) The FastCGI binding library provides a mechanism to use forkIO, forkOS, or some other mechanism to assign a worker thread to a request, and I want to compare the two fork flavors for performance and stability.

It's worth reading the fine print in the Control.Concurrent documentation. For the present application, every thread does make foreign calls as part of handling a FastCGI request and every request is likely to complete in less than Haskell scheduler's default granularity of 20ms. I'm less interested in performance and more interested in looking for leaks, deadlocks, or other bad behavior.

Building the Right Network.FastCGI

Both the 1.0 and 3000.0.0 versions of the Network.FastCGI appear in the Hackage directory, but the darcs head version (3001.0.0 as of this posting) is the one with the multi-threaded bindings exposed. (For the uninitiated, darcs is a distributed source code management system implemented in Haskell. The darcs codebase is in literate Haskell, so it's an interesting read for that if nothing else.) Darcs is available from the usual package managers; I use MacPorts on the Mac.

Get the latest Network.FastCGI from the darcs repository:

darcs get http://darcs.haskell.org/fastcgi/

The fastcgi.cabal file documents the dependencies, but a GHC 6.6.1 install is sufficient. Then build and install it the usual Cabal way:

cd fastcgi
runghc Setup.hs configure
runghc Setup.hs build
sudo runghc Setup.hs install

One more step is needed to register the package with the compiler:

sudo runghc Setup.hs register

And then to make sure that it worked:

$ ghc-pkg list 
/opt/local/lib/ghc-6.6.1/package.conf:
    Cabal-1.1.6.2, GLUT-2.1.1, HGL-3.1.1, HUnit-1.1.1, OpenGL-2.2.1,
    QuickCheck-1.0.1, X11-1.2.1, base-2.1.1, cgi-3001.1.1,
    fastcgi-3000.0.0, fastcgi-3001.0.0, fgl-5.4.1, filepath-1.0,
    (ghc-6.6.1), haskell-src-1.0.1, haskell98-1.0, html-1.0.1,
    mtl-1.0.1, network-2.0.1, parsec-2.0, readline-1.0,
    regex-base-0.72, regex-compat-0.71, regex-posix-0.71, rts-1.0,
    stm-2.0, template-haskell-2.1, time-1.1.1, unix-2.1, xhtml-3000.0.2

It takes a bit more doing to get it going with the GHC tip (to-be 6.8) because the Data.ByteString modules have been promoted into core GHC packages and rearranged a bit, but no meaningful code changes beyond some of the import statements and the fastcgi.cabal file are required. (I've sent a patch to the maintainer.)

A Simple FastCGI Handler for Process/Thread Information

The following short Haskell program (test_IO.hs) sends back a plain text response with process and thread information:

import Control.Concurrent
import System.Posix.Process (getProcessID)

import Network.FastCGI

test :: CGI CGIResult
test = do setHeader "Content-type" "text/plain"
          pid <- liftIO getProcessID
          threadId <- liftIO myThreadId
          let tid = concat $ drop 1 $ words $ show threadId
          output $ unlines [ "Process ID: " ++ show pid,
                             "Thread ID:  " ++ tid]

main = runFastCGIConcurrent' forkIO 10 test

(This is an adaptation of the printinput.hs example that uses the multi-threaded API.) To build it:

$ ghc -threaded -package fastcgi --make -o test_IO.fcgi test_IO.hs
[1 of 1] Compiling Main             ( test_IO.hs, test_IO.o )
Linking test_IO.fcgi ...

The equivalent program (test_OS.hs) with forkOS in place of forkIO does the job for OS threads.

I can use these two FastCGI handlers with different possible compiler version, web server, and FastCGI module combinations and see how things do under some simulated loads. The only gotcha with this approach is that some HTTP benchmarking tools use response byte counts as an assertion of a correct response, and they will complain as the thread ID goes from one digit to two to three, etc. My current favorite is Jef Pozkanzer's simple http_load with a tiny tweak to show the response code if a byte count comes out off. Using a different tool, e.g., ab or httperf, produces similar results.

The Web Servers: Apache2 and Lighttpd

There are probably other alternatives that I'm overlooking, but I'm going to try the two web servers that I'm familiar with, Lighttpd 1.4.15 and Apache 2.2.4, both on Mac OS X.

Configuring Lighttpd

A Lighttpd configuration file fragment for a FastCGI handler with a single process would be:

fastcgi.server = ( ".fcgi" =>
                   ( "localhost" =>
                     (
                       "socket" => "/tmp/test.sock",
                       "bin-path" => "/path/to/test_OS.fcgi",
                       "min-procs" => 1,
                       "max-procs" => 1
                     )
                   )
                 )

See the Lighttpd FastCGI documentation for the full rundown on parameters.

Also, at least as of Lighttpd 1.4.15, which is the version that MacPorts installed for me, the following configuration change is necessary to avoid a bug:

server.event-handler = "poll"

(The default value is freebsd-kqueue; see the Lighttpd performance documentation.)

After copying the file into place, we can spin-up Lighttpd and hit the URL:

$ lighttpd -f lighttpd.conf
lighttpd -f lighttpd.conf 
$ curl http://localhost:8181/test.fcgi
Process ID: 21139
Thread ID:  4
$ curl http://localhost:8181/test.fcgi
Process ID: 21139
Thread ID:  5

The thread ID changes and the process ID doesn't, so things are good. For a bigger kick:

$ echo 'http://127.0.0.1:8181/test_OS.fcgi' > /tmp/lighttpd_OS
$ http_load -parallel 20 -fetches 1000 /tmp/lighttpd_OS 2>&1 | grep -v 8181
1000 fetches, 20 max parallel, 33908 bytes, in 0.375528 seconds
33.908 mean bytes/connection
2662.92 fetches/sec, 90294.2 bytes/sec
msecs/connect: 0.19518 mean, 1.036 max, 0.09 min
msecs/first-response: 7.26042 mean, 25.558 max, 4.31 min
996 bad byte counts
HTTP response codes:
  code 200 -- 1000

The 996 bad byte count errors are expected, since the responses for thread IDs 10 through 1005 have a different number of bytes than those for thread IDs 6,7,8, and 9. In any case, so far, so good:

$ curl http://localhost:8181/test_OS.fcgi
Process ID: 21139
Thread ID:  1006

Configuring Apache2 with mod_fastcgi

The single-process configuration file fragment for Apache2 with mod_fastcgi is:

LoadModule fastcgi_module modules/mod_fastcgi.so

FastCgiConfig -maxClassProcesses 1 -processSlack 1

<Location /fastcgi>
        SetHandler fastcgi-script
        Options ExecCGI
        allow from all
</Location>

This configuration passes the basic smoke test with no issues. Under load, the forkIO version burns about half the CPU and the same amount of memory as the forkOS version. Both versions use three OS threads most of the time, and as expected based on the comments above about the way that Haskell handles scheduling, the forkOS version never uses more than four OS threads no matter how hard the server is hit.

Configuring Apache2 with mod_fcgid

The configuration fragment for Apache2 with mod_fcgid is:

LoadModule fcgid_module modules/mod_fcgid.so

MaxProcessCount 1

<Location /fcgid>
  SetHandler fcgid-script
  Options ExecCGI
  allow from all
</Location>

With the same smoke testing approach as above (with a redirect to silence the byte count complaints):

$ echo 'http://127.0.0.1:7007/fcgid/test_OS.fcgi' > /tmp/fcgid_OS
$ curl http://127.0.0.1:7007/fcgid/test_OS.fcgi
Process ID: 16854
Thread ID:  4
$ http_load -parallel 20 -fetches 1000 /tmp/fcgid_OS 2>&1 | grep -v fcgid
1000 fetches, 20 max parallel, 34704 bytes, in 1.2484 seconds
34.704 mean bytes/connection
801.028 fetches/sec, 27798.9 bytes/sec
msecs/connect: 0.294162 mean, 2.339 max, 0.051 min
msecs/first-response: 12.8977 mean, 1009.92 max, 2.758 min
986 bad byte counts
HTTP response codes: 
  code 200 -- 998
  code 500 -- 2
$ curl http://127.0.0.1:7007/fcgid/test_OS.fcgi
Process ID: 16869
Thread ID:  7

Fail, since 16854 /= 16869, and based on the mod_fcgid's stated goals of keeping FastCGI handlers "fresh" by killing them at the first sign of an issue, not that surprising.

Aggregated Results and Additional Observations

The results in these tables were generated using http_load. For the "6000/min" test:

$ http_load -rate 100 -seconds 60 url_file 2>&1 | grep -v port

For the "60000/min" test:

$ http_load -rate 1000 -seconds 60 url_file 2>&1 | grep -v port

For the fixed rate tests, the number of nines is determined by the proportion of 200 responses out of the total number of responses (the others being 500 and 503). For the requests per second mark:

$ http_load -parallel 20 -seconds 10 url_file 2>&1 | grep -v port

First, with the current GHC version:

None of these really cause the machine to break a sweat, with the web server doing most of the work and the FastCGI handler never consuming more than 60% of a core and a couple megabytes of resident memory. An overnight run showed the mod_fastcgi and forkOS combination to perform flawlessly under moderate load for over 10⁸ requests.

With the latest GHC release candidate used to compile both the FastCGI package and the handlers:

Looks like GHC 6.6.1 and Apache2/mod_fastcgi is the winning combination.

Addendum

I got GHC 6.6.1 installed and configured the forkIO and forkOS handlers on the User Mode Linux server where I have this blog hosted, and it looks like forkIO is a winner there, with process stability and around 100 requests per second sustained throughput. With the forkOS variant, the process IDs do tick up with each hit, but that's a property of fork() on the kernel where one process corresponds to one thread rather than being a result of a restarted FastCGI handler.

Herein post 4 of n on my hobby project to rewrite my personal publishing software in Haskell. Herein, I create a economically-driven approach to Atom syndication format for entries and comments. By citing economics as the prime motivator, I mean that I'm aiming neither for the most complete nor the most elegant implementation except where those two overlap with exigency.

Required Reading

To make sure that I knew enough about Atom, I read through the Atom Syndication Format RFC (I prefer the plain text to the pretty version), the introduction at AtomEnabled.org, and Mark Pilgrim's note on How to make a good ID in Atom. After reading so many specifications that either use flabby XML Schema (like WSDL) or ad hoc XML (like RSS 2.0), the use of RELAX NG compact syntax in the RFC was a breath of fresh air.

Really Simple Atom Data Model

The first set of design decisions I made were what to throw out from Atom. I decided to omit the atom:contributor construct entirely as well as atom:author is sufficient for my purposes, and atom:source and attributes on atom:link other than @rel and @href weren't going to be any use to me, either. I decided to omit the @scheme and @label attributes on atom:category since all of my @term values will be human readable and don't plan on using any scheme other than my own. I decided to omit any specific constraints on components that might otherwise be URIs, RFC3339-formatted date/time, or other — String will do for now, and I'll make sure that properly formatted data (including escaping as necessary) is used in the first place. I also decided to leave any of the sequencing and quantity constraints out of the model, as this will be an internal model only.

Here's the way it looks, and if you squint at it just right, it doesn't look that different from the RELAX NG compact schema:

data AtomElement = Feed [AtomElement]
                 | Entry [AtomElement]
                 | Content AtomContent
                 | Author { author_name :: String,
                            author_uri :: Maybe String,
                            author_email :: Maybe String }
                 | Category String 
                 | Generator { gen_name :: String,
                               gen_uri :: String,
                               gen_version :: String }
                 | Id String
                 | Icon String
                 | Link { rel :: String,
                          href :: String }
                 | Logo String
                 | Published String
                 | Rights AtomContent
                 | Subtitle AtomContent
                 | Summary AtomContent
                 | Title AtomContent
                 | Updated String
                   deriving (Show)

The Maybe String for the author_uri and author_email components of the Author representation are intended to allow for comments where the author may omit an email address or link. (Of course, I may just omit their comment under those circumstances...) Next, one more type for AtomContent, where I elected to eliminate the possibility of HTML content:

data ContentType = XHTML | TEXT
                 deriving (Eq,Show,Enum)

data AtomContent = AtomContent { contentType :: ContentType,
                                 body :: String }
                 deriving (Show)

XML Output

With a few (non-limiting) assumptions, getting XML out is simple. First up, the Atom URI, my choice to bind it to the atom prefix and my assumption that XHTML will always in the default namespace:

_prefix :: String
_prefix = "atom"

_uri :: String
_uri = "http://www.w3.org/2005/Atom"

start_div :: String
start_div = "<div xmlns=\"http://www.w3.org/1999/xhtml\">"

end_div :: String
end_div =  "</div>"

It's worth noting that the XHTML specification (via either the transitional DTD or the strict DTD) requires that the XHTML namespace be the default namespace, but there is no requirement that an XHTML fragment in an Atom feed use the default namespace.

Next, some really simple functions to wrap content in elements:

-- Format a clopen element with a list of attributes.
clopen :: String -> [(String,String)] -> String
clopen s [] = "<" ++ (prefix s) ++ "/>"
clopen s xs = "<" ++ (prefix s) ++ (nv_to_s "" xs) ++ "/>"

-- Wrap a string in an element.
wrap :: String -> String -> String
wrap s t = "<" ++ (prefix s) ++ ">" ++ t ++ "</" ++ (prefix s) ++ ">"

-- If a value is present (i.e., not Nothing), wrap it in an element.
wrap_m :: String -> Maybe String -> String
wrap_m _ Nothing = ""
wrap_m s (Just t) = wrap s t

-- Wrap an element with attributes around a string.
wrap_ :: String -> [(String,String)] -> String -> String
wrap_ s [] t = wrap s t
wrap_ s xs t = "<" ++ (prefix s) ++ (nv_to_s "" xs) ++ ('>':t)
               ++ "</" ++ (prefix s) ++ ">"

wrap_ns :: String -> String -> String
wrap_ns s t = wrap_ s [(_prefix,_uri)] t

-- Format a list of name-value pairs as attributes.
nv_to_s :: String -> [(String,String)] -> String
nv_to_s = foldl att

att :: String -> (String,String) -> String
att s (n,v) = s ++ (' ':(n ++ "=\"" ++ v ++ "\""))

And then just map the various shades of AtomElement onto the functions:

toXml :: AtomElement -> String
toXml (Feed xs) = wrap_ns "feed" (content_ xs)
toXml (Entry xs) = wrap_ns "entry" (content_ xs)

toXml' :: AtomElement -> String
toXml' (Entry xs) = wrap "entry" (content_ xs)
toXml' (Category s) = clopen "category" [("term",s)]
toXml' (Id s) = wrap "id" s
toXml' (Icon s) = wrap "icon" s
toXml' (Link r h) = clopen "link" [("rel",r),("href",h)]
toXml' (Logo s) = wrap "logo" s
toXml' (Published s) = wrap "published" s
toXml' (Updated s) = wrap "updated" s
toXml' (Author s u e) = wrap "author" ((wrap "name" s)
                                       ++ (wrap_m "uri" u)
                                       ++ (wrap_m "email" e))
toXml' (Generator n u v) = wrap_ "generator" [("uri",u),("version",v)] n
toXml' (Content a) = atom_text "content" a
toXml' (Rights a) = atom_text "rights" a
toXml' (Subtitle a) = atom_text "subtitle" a
toXml' (Summary a) = atom_text "summary" a
toXml' (Title a) = atom_text "title" a

content_ :: [AtomElement] -> String
content_ = concat.(map toXml')

-- Render an Atom text construct as XML.
atom_text :: String -> AtomContent -> String
atom_text s (AtomContent XHTML t) = wrap_ s [("type","xhtml")] (start_div ++ t ++ end_div)
atom_text s (AtomContent TEXT t) = wrap s t

(The Atom spec allows @type="text" to be omitted.) The toXml function and the AtomElement, AtomContent, and ContentType types are all that would be exported from the module.

A quick check with ghci shows that this does the right thing:

[...]
*Text.Atom> let entry = Entry [Title (AtomContent TEXT "Atom-Powered Robots Run Amok"), Id "urn:uuid:foo", Updated "2003-12-12T18:30Z", Author John Doe" Nothing Nothing, Content (AtomContent XHTML "
Some text.
")] 
*Text.Atom> toXml entry
"<atom:entry atom=\"http://www.w3.org/2005/Atom\"><atom:title type=\"text\">Atom-Powered Robots Run Amok</atom:title><atom:id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</atom:id><atom:updated>2003-12-12T18:30Z</atom:updated><atom:author><atom:name>John Doe</atom:name></atom:author><atom:content type=\"xhtml\"><div xmlns=\"http://www.w3.org/1999/xhtml\"><p>Some text.</p></div></atom:content></atom:entry>"

The let entry=... line makes more sense with some whitespace thrown in:

let entry = Entry [
  Title (AtomContent TEXT "Atom-Powered Robots Run Amok"),
  Id "urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a",
  Updated "2003-12-12T18:30Z",
  Author "John Doe" Nothing Nothing,
  Content (AtomContent XHTML "<p>Some text.</p>")
]

Other Available XML Wheels

While the above is a small wheel, it is a wheel nonetheless, and I looked at three Haskell XML libraries before reinventing it:

• Haskell XML Toolbox, a.k.a., HXT, (link) appears to be under active development and supports my basic requirements of XML output and namespace support. The API looks agreeable, and there is RSS aggregator in 50 lines as an example. If I choose to implement the Atom Publishing Protocol, HXT is the way I'll go to get atom:entry turned into the right kind of internal structure.
• HaXml (link) appears to lack namespace support, so I dismissed it without looking deeply at it.
• HXML (link) lacks namespace support, so I dismissed it without looking deeply at it. That said, the validation concept in HXML has the same heritage as the one used in the RELAX NG validator Jing.

What's Left?

There's enough real work left for at least three more blog entries: storage/state management for entries (probably STM with persistence via the filesystem), a commenting facility, human-facing content display and navigation (probably Haskell via the Text.XHtml package), and making sure that the FCGI wrapper works well multi-threaded. (I want a multi-threaded FCGI handler so that STM can serve as the concurrency control for the application; otherwise, the persistence layer will need to provide that functionality.)

State's the next one I'll tackle.

Herein, part two of n, where I think about some basic design decisions and some actual code gets written.

From GET to Function Currying

First up, some thoughts about how to map a request to a response.

A weblog is a list of entries grouped and displayed in different formats according to parameters supplied by client applications, so serving pages is limited to figuring out what format to display (single-entry HTML, multi-entry HTML, Atom) and which entries to include (include/exclude by tag, include/exclude by category, date range, maximum number of entries). Ideally, I'd like a reader to be able to, e.g., exclude only the posts I make about my kid or include only posts about BPEL and Java.

The simple design is to turn the combination of a URL and parameters into a list of functions that are applied to the list of all entries, loosely:

(format url) (filter (filters url) entries)

And that's the whole program, up to details. In mildly abused Haskell notation:

format :: URL -> [Entry] -> String
filters :: URL -> [Entry -> Boolean]

In Haskell, the notation

f :: A -> B

is very much as it is in mathematics with f being a function with domain (things of type) A and codomain (things of type) B. Unraveling expressions with multiple arrows is accomplished by adding parentheses from the right. That is,

f :: A -> B -> C  =  f :: A -> (B -> C)

is essentially a function from A×B to C as in "f is a function that maps (things of type) A to a function that maps (things of type) B to (things of type) C". So, the function format above maps a URL and a list of Entry to a String. This is one of the things that I like about Haskell — a function looks like a function.

Haskell supports currying of functions, which is only appropriate, as the language was named for Haskell Curry. The filters can be implemented by currying arguments onto functions that take two arguments. For example, consider the functions:

include_by_tag :: String -> Entry -> Boolean
include_by_tag s e = s `elem` (tags e)

exclude_by_tag :: String -> Entry -> Boolean
exclude_by_tag s e = s `notElem` (tags e)

(tags is a function that returns the list of tags applied to an entry.) In addition to showing off that Haskell lets you flip back and forth between prefix notation (foo 1 2) and infix notation (1 `foo` 2), the idea would be to map a query string atom like tag=foo to the function (include_by_tag "foo") or a query string atom like tag=-foo to the function (exclude_by_tag "foo").

Playing with the concept in ghci is straightforward. For example, drop the following two lines of Haskell into a text file called curryplus.hs:

plus :: Integer -> Integer -> Integer
plus x y = x + y

And then fire-up ghci:

$ ghci
   ___         ___ _
  / _ \ /\  /\/ __(_)
 / /_\// /_/ / /  | |      GHC Interactive, version 6.6, for Haskell 98.
/ /_\\/ __  / /___| |      http://www.haskell.org/ghc/
\____/\/ /_/\____/|_|      Type :? for help.

Loading package base ... linking ... done.
Prelude> :load curryplus.hs
[1 of 1] Compiling Main             ( curryplus.hs, interpreted )
Ok, modules loaded: Main.
*Main> :t plus
plus :: Integer -> Integer -> Integer
*Main> :t (plus 1)
(plus 1) :: Integer -> Integer
*Main> let f = (plus 1)
*Main> f 2
3

The :t command in ghci interrogates the type of the expression passed to it.

Now, where does the list of entries come from?

Storing Entries — Database or Filesystem or...?

Databases are lovely places to store data where reads and writes may overlap, and the filesystem is a good place to store information that either doesn't fit or isn't needed in memory. For a weblog, read/write contention should be light (frequent reads, infrequent writes), with writes limited to posts and comments, and optimistic concurrency is entirely acceptable. (It's of no consequence if someone gets slightly stale content.) However, the total amount of content in my weblog, counting from 2002, is in the hundreds of kilobytes, so there is no reason not to hold the whole thing in memory.

Haskell (and specifically GHC) has a couple of shiny objects that I'm tempted by. The shiniest one is STM or Software Transactional Memory, and like the three-line quicksort implementation is one of the teasers for Haskell, the four-line AtomicInteger.getAndIncrement() implementation is the teaser for STM:

get_and_increment :: TVar Integer -> IO Integer
get_and_increment i = atomically ( do j <- readTVar i
                                      writeTVar i (j+1)
                                      return j )

The whitespace in the above definition is critically important, as it tells Haskell that the lines are all part of the do. I'll come back to the left arrow ("<-") and do notation below.

Experimenting with this in ghci isn't much more complicated than the plus example above. Put the four lines above in a text file called get_and_inc.hs following the import statement:

import Control.Concurrent.STM

And fire up ghci with an extra directive to get it to load the STM package:

$ ghci -package stm
   ___         ___ _
  / _ \ /\  /\/ __(_)
 / /_\// /_/ / /  | |      GHC Interactive, version 6.6, for Haskell 98.
/ /_\\/ __  / /___| |      http://www.haskell.org/ghc/
\____/\/ /_/\____/|_|      Type :? for help.

Loading package base ... linking ... done.
Loading package stm-2.0 ... linking ... done.
Prelude> :load get_and_inc.hs
[1 of 1] Compiling Main             ( get_and_inc.hs, interpreted )
Ok, modules loaded: Main.
*Main> x <- atomically ( newTVar 1 )
*Main> :t x
x :: TVar Integer
*Main> get_and_increment x
1
*Main> get_and_increment x
2
*Main> get_and_increment x
3

The original paper on STM is from Microsoft Research back in 2005. The follow-on paper Lock Free Data Structures using STM in Haskell is a good read, wherein the authors construct two implementations of ArrayBlockingQueue in Haskell, one using locks and one using STM, and then compare their performance. (Spoiler: The larger the number of processors, the better STM performs.)

As promised, a quick word about <- and do. Haskell is a purely functional language, and that means that it is side-effect-free. On the one hand it's great to have referential transparency, since it lets the compiler or runtime VM do things like replace common pieces of a complicated expression with values, but on the other hand, something imperatively trivial like console output is a side-effect. The elegant workaround is to allow a function to return an action as a value, where the action is to be performed by an external observer at some point. The construct is called a monad, and <- and do are some of the notation that comes with it. (Defining the nature of the action and the observer is the art of constructing a monad.) The "do" chains up actions, and the "<-" captures a value created by an action. Yet another good paper from Microsoft Research has a thorough tutorial on monads in Haskell.

This brings me to the design for storing and accessing the entries:

• Use a single multi-threaded FastCGI handler to serve all requests.
• Maintain an in-memory copy of all content (entries and approved comments), with access and updates managed via STM.
• Store entry and approved comment content as separate files on disk, to be loaded at start-up time and written when an entry is posted or comment approved.

Data structures and a lo-fi approach to outputting Atom are up next.