mult.ifario.us

My odyssey with Maven continues. This entry is spurred by having a WAR built with Maven come out to be three times the size of the one built with the original Ant build. JUnit, JMock, a couple of different Log4J's, and other assorted goodies. With multiple modules and liberal use of open source components, the question is not whether someone did but who peed in which POM?

Open source reminds me of college. I had the opportunity to enjoy some eclectic people during my education at Reed and Berkeley. Rent a room and then sublet the closet? That's cool. Eat what others would otherwise throw away in the dining commons? That's cool. (Off topic, at least one former "scrounger" has done just fine...) These sort of situations came with their own etiquette, e.g., tell a "scrounger" if you have a cold when you drop off your tray and leave items intact and relatively unmolested. The bohemian analogy cuts both ways with open source — you can probably find whatever you are looking for, but it may not be in quite the state that you'd like.

Some shell scripting (find, grep, xargs, and friends) identified commons-httpclient as the likely culprit, and sure enough, it's there plain as day:

<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <version>3.8.1</version>
</dependency>

There should be a <scope>test</scope>, but there isn't. Since he helps steward the commons, I pinged Henri about it, and it looks like the issue was already fixed for versions 3.1 and on. This was only part of the battle, however, because commons-httpclient wasn't an explicit dependency; it was only getting included as a transitive dependency of some other dependency of one of the modules that the web application used, and the module hierarchy was already four levels deep. Surely someone else has already experienced issues with dependencies of unknown provenance and come up with a way to navigate the graph, and it turns out that there are (at least) two solutions.

First up, for playing Heracles to my Odysseus or Anchises to my Aeneas or Virgil to my Dante or Laurel to my Hardy or whatever in JAR hell, Henri gets a hat-tip for pointing me at the pomtools plugin, which provides an interactive interface for navigating the graph of dependencies and can alter and serialize the underlying model of the project to fix version conflicts. I didn't end up trying it, but I will, since I have a soft spot for anything with a terminal interface.

Instead, since I also have a soft spot for GraphViz, I used the depgraph plugin from the EL4J project, which I found via Philipp Oser's blog. In my case, the plugin produced the following graph:

The graph showed commons-httpclient referenced by a variety of XFire components, and some exclusions got me out of JAR purgatory for the moment. (I ate a couple of whole pomegranates down there, so I'm sure I'll be headed back sometime soon...) This isn't just a Jakarta Commons issue. XFire has a little of the same kind of POM-rot as of 1.2.3, but that will disappear in the forthcoming 1.2.5. For those keeping score at home, AXIS2 has some (xmlunit should be <scope>'d to test), too. This makes me wish for a Maven "lint" that would flag common errors like test libraries listed as runtime dependencies or dependencies not referenced from runtime source code.

Getting the depgraph plugin wired-up was straightforward. I just added a plugin repository to the master POM:

<pluginRepositories>
  <pluginRepository>
    <id>elca-services</id>
    <url>http://el4.elca-services.ch/el4j/maven2repository</url>
    <releases>
      <enabled>true</enabled>
    </releases>
  </pluginRepository>
</pluginRepositories>

Then the plugin to the build:

<build>
[...]
  <plugins>
    [...]
    <plugin>
      <groupId>ch.elca.el4j.maven.plugins</groupId>
      <artifactId>maven-depgraph-plugin</artifactId>
      <configuration>
        <outDir>target/site/images</outDir>
        <outFile>${pom.artifactId}.png</outFile>
      </configuration>
    </plugin>
  </plugins>
</build>

And then it's just a mvn depgraph:depgraph to get a view of the dependency graph. The real lesson here is to aggressively scope your dependencies as a service to the community.

Herein post 5 of n on my hobby project to rewrite my personal publishing software in Haskell. (In strict terms, the project is to write it, since I didn't write the current system.) This post covers persistence and concurrency using the filesystem and Haskell's software transactional memory implementation.

Exploiting Commutativity and Choosing Locking Granularity

As I imagine things working, the basic operations that I want to be able to perform against the persistent form of the blog are something like:

• Create an entry (and by extension, a comment).
• Change the metadata on an entry, e.g., publish/unpublish or add/remove tags.
• Add a comment to an existing entry.

From an end-user perspective, these all commute with each other — it doesn't matter whether a comment is added before or after a tag is changed — so it's reasonable to let the system take care of ordering the operations to be performed. Moreover, because creation commutes with linking, locking granularity can be limited to an individual entry. (There is no reason to lock both the newly created comment and its parent entry simultaneously.)

Without further ado, here's a locking scheme implemented at the granularity of an entry. This would be used only for writes. First, a wrapper type to hold the lock status for an entry:

data Holder = Holder { entry :: Entry,
                       locked :: Bool }
            deriving (Show)

And then the lock/unlock code:

lock :: Holder -> Holder
lock (Holder e False) = Holder e True
lock (Holder e True) = error "Already locked."

unlock :: Holder -> Holder
unlock (Holder e True) = Holder e False
unlock (Holder e False) = error "Already unlocked."

It's worth stopping to observe a common construct in functional programming. A lock function that locks a Holder can't exist because all values are immutable. Instead, lock creates a new Holder that has locked set to True but is otherwise identical to the original, and we can use the STM mechanics to create actions to be applied to a TVar:

check_out :: TVar Holder -> STM ()
check_out h = do { h' < readTVar h 
                 ; if locked h'
                   then retry
                   else writeTVar h (lock h') }

check_in :: TVar Holder -> STM ()
check_in h = do { h' < readTVar h
                ; if locked h'
                  then writeTVar h (unlock h')
                  else error "Already unlocked." }

The retry above will cause check_out to block until the entry is checked back in, while check_in signals an error if it is asked to release an already free entry.

s_show :: Show a => TVar a -> IO String
s_show = atomically.(liftM show).readTVar

Operating on Entries

To integrate operations on entries, I'm going to take the minimal use case of publishing and unpublishing, so my Entry data structure is almost trivial:

data Entry = Entry { entry_id :: String,
                     published :: Bool }
             deriving (Show)

publish :: Entry -> Entry
publish (Entry i _) = Entry i True

unpublish :: Entry -> Entry
unpublish (Entry i _) = Entry i False

Add in a function to convert an Entry -> Entry function to a Holder -> Holder function:

liftH :: (Entry -> Entry) -> (TVar Holder -> STM ())
liftH f = \ h -> do { h' < readTVar h
                    ; writeTVar h ((holderize f) h')
                    ; return () }
          where holderize f = \ (Holder e l) -> Holder (f e) l

Combining a publish with check_in/check_out is straightforward in the STM monoid. Here's some scratch work in ghci that shows this in action:

$ ghci -package stm
   ___         ___ _
  / _ \ /\  /\/ __(_)
 / /_\// /_/ / /  | |      GHC Interactive, version 6.6, for Haskell 98.
/ /_\\/ __  / /___| |      http://www.haskell.org/ghc/
\____/\/ /_/\____/|_|      Type :? for help.

Loading package base ... linking ... done.
Loading package stm-2.0 ... linking ... done.
:Prelude> :load experiment.hs
[1 of 1] Compiling Main             ( experiment.hs, interpreted )
Ok, modules loaded: Main.
*Main> let h = Holder (Entry "foo" False) False
*Main> th < atomically ( newTVar h )
*Main> s_show th
"Holder {entry = Entry {entry_id = \"foo\", published = False}, locked = False}"
*Main> let co = check_out th
*Main> let pub = (liftH publish) th
*Main> let ci = check_in th
*Main> atomically ( co >> pub >> ci)
*Main> s_show th
"Holder {entry = Entry {entry_id = \"foo\", published = True}, locked = False}"

Integrating Persistence via IO

One of my working design assumptions is that the data for the system will reside entirely in memory, being updated as changes are made and reloaded (lazily) in the event of a system crash or system start-up. (As I commented previously, four years of blogging has produced around 500kb of content, mark-up included, so this isn't an unreasonable assumption.) Comments from spammers could produce a lot more data, but I plan to save every item but only load published items into memory. (So spammers are just going to burn disk space.) I'm going to aim for one file per entry, for the sake of the current discussion, named by the entry_id of the Entry. Conveniently, STM includes the unsafeIOToSTM function for composing STM actions and IO actions. (The other way around is not permitted by design.)

Attention: I've gotten some public and private comments that unsafeIOToSTM is not the right thing to use in this scenario, so I've written a revision to this entry.

Writing an entry to a file is straightforward:

store :: TVar Holder -> STM ()
store h = do { h' < readTVar h
             ; let e = entry h'
             ; unsafeIOToSTM (writeFile (fname e) ((show e) ++ "\n")) }

Continuing the same ghci session from above:

*Main> let out = store th
*Main> :! cat entry-foo.hb
cat: entry-foo.hb: No such file or directory
*Main> atomically (  co >> pub >> out >> ci )
*Main> :! cat entry-foo.hb
Entry {entry_id = "foo", published = True}
*Main> let unpub = (liftH unpublish) th
*Main> atomically (  co >> unpub >> out >> ci )
*Main> :! cat entry-foo.hb
Entry {entry_id = "foo", published = False}

This could (and probably should) be made a bit fancier with regard to recovering from errors while writing the file, but I'm happy with the basic ergonomics so far.

With the default Erlang book now over a decade old, a new one was sorely needed, and it looks like Joe Armstrong has stepped up (again). (Hat-tip to Bill de Hóra for the pointer to the book, since I haven't been tracking the Erlang space lately.) It's not in the beta PDF yet, but I'm looking forward to what is currently slated for Chapter 19:

How to structure applications for programming multi-core CPUs. Increasing parallelism. Deciding the granularity of concurrency.

Good stuff.

I ran across Alan Warms's response to a question from Ross Mayfield to Dick Costolo:

How does an Internet entrepreneur overcome not being in the Silicon Valley?

(Dick, by the way, has been publishing lots of good stuff on his Ask the Wizard blog.) The answer is: it depends on how far you are from your customer.

Before moving West to wetter pastures, I lived in Chicago for 8 years, five of which I spent as an entrepreneur, and Chicago is a great place to build a business. My experience matches Alan's:

1. You can make a late morning meeting anywhere in the country — other than the East Coast — without staying overnight. For the East Coast (e.g., NY, Boston, Atlanta), early afternoon works better, and you can still be home for a late dinner. Chicago is a hub for United and Southwest, and commuting to major cities in the Midwest (like Nashville, St. Louis, Minneapolis, Detroit, and Milwaukee) is as convenient as commuting to the Chicago suburbs. (Unfortunately, that doesn't say all that much...)
2. There are plenty of smart young people around. Within Chicago or close-in suburbs, there are: University of Chicago, Northwestern, and University of Illinois Chicago. Within a reasonable distance, there are University of Illinois Urbana-Champaign, Purdue, University of Michigan,...
3. There are plenty of experienced industry practitioners in the area, thanks to the shippers/haulers, banks, financial services companies, pharmaceutical companies, healthcare companies, insurance companies, manufacturers, heavy industries, and other companies in the immediate area. The first few that come to mind are Motorola, State Farm, Citadel, Allstate, Baxter Healthcare, Caterpillar, UBS, McDonalds, BankOne, Wrigley, and Boeing. If you cast a slightly wider net, say a ninety-minute plane flight, you pick up Harley Davidson, GM, Ford, Chrysler, Schneider National, Anheuser-Busch, Smurfit-Stone, AG Edwards, and Northwestern Mutual.
4. Corporate necessities are affordable. For example, we rented industrial loft space for $10 per square foot (per year). During the Bubble.

If you're looking to raise venture capital, there are some top-shelf people in Chicago. (Based on personal experience, I have good things to say about Apex, JK&B, and OCA.) Valley VCs may be reluctant to sign up for years of plane flights for Board meetings when they have plenty of action close to home, and I can't really blame them for that.

For the first four years of its life, FiveSight sold traditionally-licensed middleware to end-users (like those companies that I listed above), and all of the advantages that I've just enumerated worked in our favor. As our model shifted to one where our customers, partners, and potential acquirers were software companies, not being in the Valley left us at a disadvantage competing with companies that had a presence there. Flying out to get customer face-time became both a necessity and a distraction for the core team both because of the travel time and because of the temporary loss of the close communication and collaboration that made the team work best. I can recall one acquisition opportunity where our competitor could literally walk to the offices of the acquirer. Even in the most objective head-to-head competition, the local candidate is going to be able to more effectively gather and apply information to influence the deal. Also, being unknown is a disadvantage. The environment in the Valley doesn't actively exclude outsiders, but it tends to include insiders via "X worked with Y at company Z funded by W" relationships.

(This reminds me that I need to write an entry about when to go "all in".)

Open source, blogging, and professional/social use of the Internet are helping information to travel faster and farther than it did previously, but these things only make a difference when they bring you into closer and better communication with your customer.

Herein post 4 of n on my hobby project to rewrite my personal publishing software in Haskell. Herein, I create a economically-driven approach to Atom syndication format for entries and comments. By citing economics as the prime motivator, I mean that I'm aiming neither for the most complete nor the most elegant implementation except where those two overlap with exigency.

Required Reading

To make sure that I knew enough about Atom, I read through the Atom Syndication Format RFC (I prefer the plain text to the pretty version), the introduction at AtomEnabled.org, and Mark Pilgrim's note on How to make a good ID in Atom. After reading so many specifications that either use flabby XML Schema (like WSDL) or ad hoc XML (like RSS 2.0), the use of RELAX NG compact syntax in the RFC was a breath of fresh air.

Really Simple Atom Data Model

The first set of design decisions I made were what to throw out from Atom. I decided to omit the atom:contributor construct entirely as well as atom:author is sufficient for my purposes, and atom:source and attributes on atom:link other than @rel and @href weren't going to be any use to me, either. I decided to omit the @scheme and @label attributes on atom:category since all of my @term values will be human readable and don't plan on using any scheme other than my own. I decided to omit any specific constraints on components that might otherwise be URIs, RFC3339-formatted date/time, or other — String will do for now, and I'll make sure that properly formatted data (including escaping as necessary) is used in the first place. I also decided to leave any of the sequencing and quantity constraints out of the model, as this will be an internal model only.

Here's the way it looks, and if you squint at it just right, it doesn't look that different from the RELAX NG compact schema:

data AtomElement = Feed [AtomElement]
                 | Entry [AtomElement]
                 | Content AtomContent
                 | Author { author_name :: String,
                            author_uri :: Maybe String,
                            author_email :: Maybe String }
                 | Category String 
                 | Generator { gen_name :: String,
                               gen_uri :: String,
                               gen_version :: String }
                 | Id String
                 | Icon String
                 | Link { rel :: String,
                          href :: String }
                 | Logo String
                 | Published String
                 | Rights AtomContent
                 | Subtitle AtomContent
                 | Summary AtomContent
                 | Title AtomContent
                 | Updated String
                   deriving (Show)

The Maybe String for the author_uri and author_email components of the Author representation are intended to allow for comments where the author may omit an email address or link. (Of course, I may just omit their comment under those circumstances...) Next, one more type for AtomContent, where I elected to eliminate the possibility of HTML content:

data ContentType = XHTML | TEXT
                 deriving (Eq,Show,Enum)

data AtomContent = AtomContent { contentType :: ContentType,
                                 body :: String }
                 deriving (Show)

XML Output

With a few (non-limiting) assumptions, getting XML out is simple. First up, the Atom URI, my choice to bind it to the atom prefix and my assumption that XHTML will always in the default namespace:

_prefix :: String
_prefix = "atom"

_uri :: String
_uri = "http://www.w3.org/2005/Atom"

start_div :: String
start_div = "<div xmlns=\"http://www.w3.org/1999/xhtml\">"

end_div :: String
end_div =  "</div>"

It's worth noting that the XHTML specification (via either the transitional DTD or the strict DTD) requires that the XHTML namespace be the default namespace, but there is no requirement that an XHTML fragment in an Atom feed use the default namespace.

Next, some really simple functions to wrap content in elements:

-- Format a clopen element with a list of attributes.
clopen :: String -> [(String,String)] -> String
clopen s [] = "<" ++ (prefix s) ++ "/>"
clopen s xs = "<" ++ (prefix s) ++ (nv_to_s "" xs) ++ "/>"

-- Wrap a string in an element.
wrap :: String -> String -> String
wrap s t = "<" ++ (prefix s) ++ ">" ++ t ++ "</" ++ (prefix s) ++ ">"

-- If a value is present (i.e., not Nothing), wrap it in an element.
wrap_m :: String -> Maybe String -> String
wrap_m _ Nothing = ""
wrap_m s (Just t) = wrap s t

-- Wrap an element with attributes around a string.
wrap_ :: String -> [(String,String)] -> String -> String
wrap_ s [] t = wrap s t
wrap_ s xs t = "<" ++ (prefix s) ++ (nv_to_s "" xs) ++ ('>':t)
               ++ "</" ++ (prefix s) ++ ">"

wrap_ns :: String -> String -> String
wrap_ns s t = wrap_ s [(_prefix,_uri)] t

-- Format a list of name-value pairs as attributes.
nv_to_s :: String -> [(String,String)] -> String
nv_to_s = foldl att

att :: String -> (String,String) -> String
att s (n,v) = s ++ (' ':(n ++ "=\"" ++ v ++ "\""))

And then just map the various shades of AtomElement onto the functions:

toXml :: AtomElement -> String
toXml (Feed xs) = wrap_ns "feed" (content_ xs)
toXml (Entry xs) = wrap_ns "entry" (content_ xs)

toXml' :: AtomElement -> String
toXml' (Entry xs) = wrap "entry" (content_ xs)
toXml' (Category s) = clopen "category" [("term",s)]
toXml' (Id s) = wrap "id" s
toXml' (Icon s) = wrap "icon" s
toXml' (Link r h) = clopen "link" [("rel",r),("href",h)]
toXml' (Logo s) = wrap "logo" s
toXml' (Published s) = wrap "published" s
toXml' (Updated s) = wrap "updated" s
toXml' (Author s u e) = wrap "author" ((wrap "name" s)
                                       ++ (wrap_m "uri" u)
                                       ++ (wrap_m "email" e))
toXml' (Generator n u v) = wrap_ "generator" [("uri",u),("version",v)] n
toXml' (Content a) = atom_text "content" a
toXml' (Rights a) = atom_text "rights" a
toXml' (Subtitle a) = atom_text "subtitle" a
toXml' (Summary a) = atom_text "summary" a
toXml' (Title a) = atom_text "title" a

content_ :: [AtomElement] -> String
content_ = concat.(map toXml')

-- Render an Atom text construct as XML.
atom_text :: String -> AtomContent -> String
atom_text s (AtomContent XHTML t) = wrap_ s [("type","xhtml")] (start_div ++ t ++ end_div)
atom_text s (AtomContent TEXT t) = wrap s t

(The Atom spec allows @type="text" to be omitted.) The toXml function and the AtomElement, AtomContent, and ContentType types are all that would be exported from the module.

A quick check with ghci shows that this does the right thing:

[...]
*Text.Atom> let entry = Entry [Title (AtomContent TEXT "Atom-Powered Robots Run Amok"), Id "urn:uuid:foo", Updated "2003-12-12T18:30Z", Author John Doe" Nothing Nothing, Content (AtomContent XHTML "
Some text.
")] 
*Text.Atom> toXml entry
"<atom:entry atom=\"http://www.w3.org/2005/Atom\"><atom:title type=\"text\">Atom-Powered Robots Run Amok</atom:title><atom:id>urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a</atom:id><atom:updated>2003-12-12T18:30Z</atom:updated><atom:author><atom:name>John Doe</atom:name></atom:author><atom:content type=\"xhtml\"><div xmlns=\"http://www.w3.org/1999/xhtml\"><p>Some text.</p></div></atom:content></atom:entry>"

The let entry=... line makes more sense with some whitespace thrown in:

let entry = Entry [
  Title (AtomContent TEXT "Atom-Powered Robots Run Amok"),
  Id "urn:uuid:1225c695-cfb8-4ebb-aaaa-80da344efa6a",
  Updated "2003-12-12T18:30Z",
  Author "John Doe" Nothing Nothing,
  Content (AtomContent XHTML "<p>Some text.</p>")
]

Other Available XML Wheels

While the above is a small wheel, it is a wheel nonetheless, and I looked at three Haskell XML libraries before reinventing it:

• Haskell XML Toolbox, a.k.a., HXT, (link) appears to be under active development and supports my basic requirements of XML output and namespace support. The API looks agreeable, and there is RSS aggregator in 50 lines as an example. If I choose to implement the Atom Publishing Protocol, HXT is the way I'll go to get atom:entry turned into the right kind of internal structure.
• HaXml (link) appears to lack namespace support, so I dismissed it without looking deeply at it.
• HXML (link) lacks namespace support, so I dismissed it without looking deeply at it. That said, the validation concept in HXML has the same heritage as the one used in the RELAX NG validator Jing.

What's Left?

There's enough real work left for at least three more blog entries: storage/state management for entries (probably STM with persistence via the filesystem), a commenting facility, human-facing content display and navigation (probably Haskell via the Text.XHtml package), and making sure that the FCGI wrapper works well multi-threaded. (I want a multi-threaded FCGI handler so that STM can serve as the concurrency control for the application; otherwise, the persistence layer will need to provide that functionality.)

State's the next one I'll tackle.

As our monthly attempt to get a good meal, the wife and I went to Nishino for a "deluxe" eight-course omakase on Saturday, and other than the dessert, it did not disappoint. The dessert could be described as a tempura banana split, and while that sounds like it could be good, it wasn't. The rest of the meal was so good that we didn't care about dessert. Most of the courses had a Japanese-French mash-up feel to it with amazing sashimi paired with good sauces (red wine reduction with tuna, foie gras, and shitake mushroom) or simple garnishes (amberjack with jalapeno, ginger salsa, and a garlic chip). The straight-ahead nigiri course was excellent, with literally the best o-toro that I've ever had and some sawara that was even smoother and butterier. (I'm a little unclear on the sawara (spanish mackerel) versus aji (horse mackerel), since our server said both "aji" and "spanish mackerel" — either way, those were really amazing pieces of fish, mercury risk aside.) Speaking as someone who used to regularly travel to Japan, this was great sushi; we're definitely going back.

When you first experience Maven, it's like the cinematic cliché of two lovers running across a grassy field to embrace: It takes care of getting the right JARs for you, keeps test and build dependencies separate, and generates configuration artifacts for your favorite IDE, too. Even though it's XML, it's still a sweet five-minute user experience, but then you start stepping on the rakes in the grass:

"Hmmm. Doesn't look like Emma is supported by Maven2, so I'll give Cobertura a shot." THWAP! The current version of the plugin is broken, but downgrading to an earlier version appears to work.

"Looks like TestNG support only extends to version 4.7, but maybe my tests written against 5.5 will still work." THWAP! It turns out that up to date support for TestNG is a bit of a can of worms. (I would have tried the 2.8-SNAPSHOT version, but the pom was broken.)

"Oh well. I can switch to JUnit4 and do without data providers, make @BeforeClass methods static, and replace dependency-driven order of execution by some hand-coded magic." THWAP! The documentation doesn't explicitly state that the current version of the Surefire plugin is for JUnit 3.8 only, but it is. Adding the Apache plugin snapshot repository and using the 2.3-SNAPSHOT version of the plugin gets things working. (Of course, then it also turns out that IDEA 6 doesn't support JUnit 4.2.)

This is no worse than dealing with dependencies and working directories in Ant, so I'm happy enough with Maven in the relatively green-field environment that I'm working in. I can see where trying to bend an existing project, especially a large one, to Maven could quickly become a fool's errand.

I have to admit that I looked closely at Maven in its version 1 incarnation, but it flunked the five minute test. (Among other things, I was never able to grok how the "reactor" worked for multi-module builds.) Maven 2, thanks to the Apache site and the book and examples from Mergere, gets over that hump. (FiveSight had a home-grown multi-module build system built on top of Ant, but it didn't do an especially good job of rolling-up reporting or of managing conflicts in transitive dependencies (lib A version C requires lib B version D, but lib E requires lib B version F).)

I do like where Matthieu is headed with raven or Russel Winder is headed with gant, but I'd really like the two together: non-XML syntax, dependency management, fully-stocked and up-to-date repositories, well-supported by continuous integration systems, and works with everything that Ant works with. Maybe gant plus the Maven2 Ant tasks or Ivy is the most silver-ish bullet for the time being, but seeing as I don't need to shoot any werewolves, Maven fits the bill for the moment.