mult.ifario.us

On a lark, I decided to try the Code Analysis Plugin for Eclipse, but I immediately ran into an error:

Exception in thread "main" java.lang.InternalError: Can't start the AWT because
Java was started on the first thread.  Make sure StartOnFirstThread is not
specified in your application's Info.plist or on the command line
	at java.lang.ClassLoader$NativeLibrary.load(Native Method)
[etc.]

It turns out that this is the result of a fundamental disconnect between SWT and AWT on the Mac! A quick Google for StartOnFirstThread gives a snapshot of some of the afflicted plugins, none of which are on my must-have list, so it won't send me running for Netbeans or Linux. Nonetheless, it does raise some doubts in my mind about both SWT and Mac OS X.

Via Charles Miller, a link to Fred Brooks's classic essay, "No Silver Bullet". (You can also get it at the IEEE Computer Society, here.)

As a small advertisement, here's a quote:

I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems.

If this is true, building software will always be hard. There is inherently no silver bullet.

The essay is also included in the most recent edition of Brooks's The Mythical Man Month, which is also required reading. (I've blogged elsewhere about this previously.)

Discovery occurs only through experience. Expecting to get it wrong the first time -- which you will -- and then being ready to throw it away are the first steps to getting it (more) right the next time. Ignorance of this fact is damaging to everyone who builds and uses software, whether the project in question is an internal application, a product, or a specification. (There are plenty of examples of crimes perpetrated on the community by committees, and there are more coming.)

Bill Burnham ranks BPEL at a healthy #7 of the top 10 technology trends for 2005. His comments about the market are interesting:

On the opportunity front, while the opportunity to fund core BPEL servers has passed, there may be opportunities to fund both high level and low level BPEL deals. At a high level, BPEL servers that are customized to meet the needs of a particular industry, such as insurance or manufacturing, may present interesting investment opportunities, while at a low level BPEL gateways or routers that quickly process and transform messages may also be attractive.

My vision of the present and the future, albeit from knee-deep in the mud of the trenches, is both the same and different.

First, with regard to the market, the core BPEL server market hasn't solidified yet, no more so than the relational database market did in the first year after the inception of SQL, and it won't solidify until 12 months or more from now. WS-BPEL 2.0 is coming, and it entails enough changes over BPEL4WS 1.1 that most implementations will take a while to adapt after the specification is finalized. (BPEL4WS 1.1, while available in various forms from a number of vendors (including us), is not a standard and has enough inconsistencies in it that make it unlikely for two different servers to have similar behavior even if given the same process.) The release of the WS-BPEL 2.0 specification by OASIS later this year will be the real starting gun on the BPEL race.

Nonetheless, I agree with Bill's assessment of the VC opportunity, just for different reasons. From my view, there is no VC-compatible investment opportunity in the core BPEL server space now for the same reason that there was no VC-compatible investment opportunity in the BPEL server space 12 months ago — the structure of the market is too unclear for a VC investment in the current economic climate. This only matters if VC investment is required to build a market-contender BPEL core, and it's not.

I also agree that the embeddable BPEL functionality market is more interesting, as the more messaging and concurrency become a part of everyday applications, the more important orchestration becomes a core functional requirement for building those applications. The question for the embeddable BPEL (or orchestration) space (if you ask me) is who will be the Hibernate or Spring — well-designed, well-packaged, minimal but functionally complete, and refined but open to extension. Of course, I have my opinion...

Automated black box testing of commandline tools tends to be (but should not be) a bit of a black art. There is a tendency among Java developers to ignore otherwise useful tools and ideas that aren't prepackaged for either Ant or JUnit, and even if I'm working with commandline tools implemented in Java, my preference is for expect, a tool specifically for automating interactive commandline applications. The language is based on Tcl, which isn't too unreasonable to pick up given a few examples, and expect comes with most *nixes (including Mac OS). (It is also available for Windows.) The point of expect is that you can use it to spawn one or more processes, send them text, and tell the tool what to expect (Get it...?) in the way of output. Conveniences like expected response timeouts, regular expressions, and basic control flow (via Tcl) are all natively available in expect.

There are plenty of more or less instructive examples available, but for a concrete use case, an expect script could download and unpack a fresh Tomcat, drop and refresh a database, launch the Tomcat, deploy a web application by copying a WAR into place, and watch the Tomcat logs to ensure that nothing unexpected happens while the same script runs a test client against the web application from another machine. It's also reasonable to write expect scripts that simulate the examples provided in user documentation as part of an integration test suite. (Essentially, expect can do whatever you might normally do with a bunch of shell windows open on your desktop, including logging into other servers and interacting with the remote shell.)

Cedric has some good comments about the "call super antipattern". For background, some designs require that a method m() on a class C call-back to the super at either the top of bottom of the subclass method:

class SubC extends C {
  void m() {
    // do some stuff
    // and then don't forget to call the super
    super.m();
  }

This might seem like a small incremental sin after using extends in the first place, but call super is insidious because it's like a secret handshake. As that software ages, someone will add code or extend the module and forget about calling super, causing all hell to break loose in some subtle way. Even worse, someone may alter the base class without knowing about the dependencies. (Extension is often used as a substitute for a macro facility in Java, but that's another topic.)

The call super antipattern is an example of favoring recursion over iteration because the class extension mechanism for Java classes makes it convenient. Any number of the usual techniques for replacing recursion with iteration (remember Towers of Hanoi from CS 101?) can be used to unravel a design that uses the call super antipattern and will make for a better design now and down the road.

Dion has called for a little restraint on performance comparisons between Rails and J2EE. I don't have any special place in my heart for either one, but restraint is certainly warranted, as there are four kinds of lies -- lies, damn lies, statstics, and benchmarks... A thorough and careful design, not any particular language and certainly not any particular framework, is the shortest path to elegance. (IMHO.)

On this subject, one study concluded that the most significant overall risk factor in a project is not using a particular methodology but rather using the wrong one. Knowing the right one requires luck, an experienced mentor, or proximity to a sufficient number of acknowledged, dissected failures.

Like most things, this brings to mind an anecdote, and this one is even relevant. Once upon a time (ages ago -- 2001), we needed to meet a performance benchmark (10^x end-to-end transactions per eight hours) for a customer before we closed a license agreement. (They paid for the proof of concept and benchmark on their hardware, which is the fair thing to do.) We configured a dual-CPU test system on a cheap beater box with a fibre channel array built from pieces that we bought off eBay, some random FC cabling provider, and a liquidator who specialized in cheap FC drives. The total cost was something like $700. We used an empty PC case to hold the array with a paperclip to short the on/off jumper for the power supply. We sat down with a profiler (the final source of Truth when performance tuning) and tuned things to beat the required performance benchmark by a factor of two on our FrankenPC. Feeling good, we set everything up on a couple of the customer's HP N-class servers, and... the system fell short of the benchmark by a factor of three, i.e., it underperformed the gimpy PC by a factor of six.

So, what gives? Why would a couple of $70,000 server-class machines with a staff of trained professionals to take care of them be slower than a $700 FrankenPC? After a good number of possible causes were eliminated, it turned out that the disk array for the SAN that supported the two servers was configured in the worst possible way, and after that was corrected, the system sailed past the required performance mark as expected.

Stefan is asking what the big deal is with process languages:

Whether it's workflow, orchestration, business process management, or a similar concept under some local name familiar to you, one always seems to end up looking for a flexible, accessible way to define how data flows from one processing node to another. In an XML-centric world, you'll think about XML documents that get transformed, separated into parts, aggregated again; [...]

After endless debates, you end up with some open source or commercial solution based on some half-baked standard that you just can't seem to really like. [...]

Call me a heretic, but I believe that every time somebody tries to sell you some XML-based, pseudo-declarative language it is either a) Turing-complete or b) sucks. And I have not yet seen convincing arguments that this would be different for BPEL.

Messaging Merge as Motivation

The simple "merge" operation in a messaging environment crops up every time a parallel branch of processing needs to occur asynchronously (e.g., user interaction or batch processing by an external system) -- it is the way that the response rejoins the original "thread" of processing. It is also one of the first steps toward orchestration. Writing an efficient merge widget requires reliably dehydrating processing state, correlating inbound messages with dehydrated states, and then managing the consumption of inbound messages and rehydration of suspended processing state intelligently. (In fact, it was the implementation and re-implementation of different messaging merge scenarios that led to our first orchestration design way back in early 2001.)

It Seemed Easy Enough at First Glance...

The three core aspects of the merge are transparent dehydration and rehydration, correlation of messages with instances, and "chunky" concurrency that ties it all together reliably and scalably. (Concurrency != threads; threads are just an abstraction.) These three core aspects are the answer to one of Stefan's other questions:

Now let's just say you pick the JVM as your engine, and Java as your process definition language — what exactly do you lose? Not dynamic enough? When was the last time you changed a process already in deployment without serious consideration? Even so, pick Jython, or Groovy — now what is the difference? How about using some other scripting language?

BPEL takes some heat for its syntax, and that isn't undeserved. The XML-based syntax of BPEL is unfortunate, to be sure, but it's a consequence of the fact that related specifications (WSDL, SOAP, etc.) use XML as a language and that most folks find parsing XML easier than writing a real parser for an old-fashioned language. I wouldn't necessarily go as far as Phil Windley to call it "syntactic arsenic", and I've written a good amount of BPEL by hand. (Emacs works fine for me and is more likely to produce correct BPEL than any of the GUI editors I've tried.)

BPEL exists because Java, Python, Jython, and Ruby -- as well as more interesting and exotic stuff like Join Java and TyCO -- simply don't support those three core aspects of the merge scenario, and doing a good implementation from scratch is complex.

In its domain, BPEL is a powerful tool, and as with many things, if it hurts, you're doing it wrong... The correct strategy for working with an orchestration engine, at least if you ask me, is to treat it as a component that factors specific functional aspects out of the system or product being implemented. An orchestration engine belongs at the bottom of the architecture as a supporting component, not at the top as an application language or a design. Those functional aspects that remain after factoring out orchestration, e.g., fine-grained XML handling, should be encapsulated and attacked with the best tool for the respective job, and that typically is not BPEL or another orchestration language.