mult.ifario.us

Bill Burnham ranks BPEL at a healthy #7 of the top 10 technology trends for 2005. His comments about the market are interesting:

On the opportunity front, while the opportunity to fund core BPEL servers has passed, there may be opportunities to fund both high level and low level BPEL deals. At a high level, BPEL servers that are customized to meet the needs of a particular industry, such as insurance or manufacturing, may present interesting investment opportunities, while at a low level BPEL gateways or routers that quickly process and transform messages may also be attractive.

My vision of the present and the future, albeit from knee-deep in the mud of the trenches, is both the same and different.

First, with regard to the market, the core BPEL server market hasn't solidified yet, no more so than the relational database market did in the first year after the inception of SQL, and it won't solidify until 12 months or more from now. WS-BPEL 2.0 is coming, and it entails enough changes over BPEL4WS 1.1 that most implementations will take a while to adapt after the specification is finalized. (BPEL4WS 1.1, while available in various forms from a number of vendors (including us), is not a standard and has enough inconsistencies in it that make it unlikely for two different servers to have similar behavior even if given the same process.) The release of the WS-BPEL 2.0 specification by OASIS later this year will be the real starting gun on the BPEL race.

Nonetheless, I agree with Bill's assessment of the VC opportunity, just for different reasons. From my view, there is no VC-compatible investment opportunity in the core BPEL server space now for the same reason that there was no VC-compatible investment opportunity in the BPEL server space 12 months ago — the structure of the market is too unclear for a VC investment in the current economic climate. This only matters if VC investment is required to build a market-contender BPEL core, and it's not.

I also agree that the embeddable BPEL functionality market is more interesting, as the more messaging and concurrency become a part of everyday applications, the more important orchestration becomes a core functional requirement for building those applications. The question for the embeddable BPEL (or orchestration) space (if you ask me) is who will be the Hibernate or Spring — well-designed, well-packaged, minimal but functionally complete, and refined but open to extension. Of course, I have my opinion...

Automated black box testing of commandline tools tends to be (but should not be) a bit of a black art. There is a tendency among Java developers to ignore otherwise useful tools and ideas that aren't prepackaged for either Ant or JUnit, and even if I'm working with commandline tools implemented in Java, my preference is for expect, a tool specifically for automating interactive commandline applications. The language is based on Tcl, which isn't too unreasonable to pick up given a few examples, and expect comes with most *nixes (including Mac OS). (It is also available for Windows.) The point of expect is that you can use it to spawn one or more processes, send them text, and tell the tool what to expect (Get it...?) in the way of output. Conveniences like expected response timeouts, regular expressions, and basic control flow (via Tcl) are all natively available in expect.

There are plenty of more or less instructive examples available, but for a concrete use case, an expect script could download and unpack a fresh Tomcat, drop and refresh a database, launch the Tomcat, deploy a web application by copying a WAR into place, and watch the Tomcat logs to ensure that nothing unexpected happens while the same script runs a test client against the web application from another machine. It's also reasonable to write expect scripts that simulate the examples provided in user documentation as part of an integration test suite. (Essentially, expect can do whatever you might normally do with a bunch of shell windows open on your desktop, including logging into other servers and interacting with the remote shell.)

Cedric has some good comments about the "call super antipattern". For background, some designs require that a method m() on a class C call-back to the super at either the top of bottom of the subclass method:

class SubC extends C {
  void m() {
    // do some stuff
    // and then don't forget to call the super
    super.m();
  }

This might seem like a small incremental sin after using extends in the first place, but call super is insidious because it's like a secret handshake. As that software ages, someone will add code or extend the module and forget about calling super, causing all hell to break loose in some subtle way. Even worse, someone may alter the base class without knowing about the dependencies. (Extension is often used as a substitute for a macro facility in Java, but that's another topic.)

The call super antipattern is an example of favoring recursion over iteration because the class extension mechanism for Java classes makes it convenient. Any number of the usual techniques for replacing recursion with iteration (remember Towers of Hanoi from CS 101?) can be used to unravel a design that uses the call super antipattern and will make for a better design now and down the road.

Dion has called for a little restraint on performance comparisons between Rails and J2EE. I don't have any special place in my heart for either one, but restraint is certainly warranted, as there are four kinds of lies -- lies, damn lies, statstics, and benchmarks... A thorough and careful design, not any particular language and certainly not any particular framework, is the shortest path to elegance. (IMHO.)

On this subject, one study concluded that the most significant overall risk factor in a project is not using a particular methodology but rather using the wrong one. Knowing the right one requires luck, an experienced mentor, or proximity to a sufficient number of acknowledged, dissected failures.

Like most things, this brings to mind an anecdote, and this one is even relevant. Once upon a time (ages ago -- 2001), we needed to meet a performance benchmark (10^x end-to-end transactions per eight hours) for a customer before we closed a license agreement. (They paid for the proof of concept and benchmark on their hardware, which is the fair thing to do.) We configured a dual-CPU test system on a cheap beater box with a fibre channel array built from pieces that we bought off eBay, some random FC cabling provider, and a liquidator who specialized in cheap FC drives. The total cost was something like $700. We used an empty PC case to hold the array with a paperclip to short the on/off jumper for the power supply. We sat down with a profiler (the final source of Truth when performance tuning) and tuned things to beat the required performance benchmark by a factor of two on our FrankenPC. Feeling good, we set everything up on a couple of the customer's HP N-class servers, and... the system fell short of the benchmark by a factor of three, i.e., it underperformed the gimpy PC by a factor of six.

So, what gives? Why would a couple of $70,000 server-class machines with a staff of trained professionals to take care of them be slower than a $700 FrankenPC? After a good number of possible causes were eliminated, it turned out that the disk array for the SAN that supported the two servers was configured in the worst possible way, and after that was corrected, the system sailed past the required performance mark as expected.

Stefan is asking what the big deal is with process languages:

Whether it's workflow, orchestration, business process management, or a similar concept under some local name familiar to you, one always seems to end up looking for a flexible, accessible way to define how data flows from one processing node to another. In an XML-centric world, you'll think about XML documents that get transformed, separated into parts, aggregated again; [...]

After endless debates, you end up with some open source or commercial solution based on some half-baked standard that you just can't seem to really like. [...]

Call me a heretic, but I believe that every time somebody tries to sell you some XML-based, pseudo-declarative language it is either a) Turing-complete or b) sucks. And I have not yet seen convincing arguments that this would be different for BPEL.

Messaging Merge as Motivation

The simple "merge" operation in a messaging environment crops up every time a parallel branch of processing needs to occur asynchronously (e.g., user interaction or batch processing by an external system) -- it is the way that the response rejoins the original "thread" of processing. It is also one of the first steps toward orchestration. Writing an efficient merge widget requires reliably dehydrating processing state, correlating inbound messages with dehydrated states, and then managing the consumption of inbound messages and rehydration of suspended processing state intelligently. (In fact, it was the implementation and re-implementation of different messaging merge scenarios that led to our first orchestration design way back in early 2001.)

It Seemed Easy Enough at First Glance...

The three core aspects of the merge are transparent dehydration and rehydration, correlation of messages with instances, and "chunky" concurrency that ties it all together reliably and scalably. (Concurrency != threads; threads are just an abstraction.) These three core aspects are the answer to one of Stefan's other questions:

Now let's just say you pick the JVM as your engine, and Java as your process definition language — what exactly do you lose? Not dynamic enough? When was the last time you changed a process already in deployment without serious consideration? Even so, pick Jython, or Groovy — now what is the difference? How about using some other scripting language?

BPEL takes some heat for its syntax, and that isn't undeserved. The XML-based syntax of BPEL is unfortunate, to be sure, but it's a consequence of the fact that related specifications (WSDL, SOAP, etc.) use XML as a language and that most folks find parsing XML easier than writing a real parser for an old-fashioned language. I wouldn't necessarily go as far as Phil Windley to call it "syntactic arsenic", and I've written a good amount of BPEL by hand. (Emacs works fine for me and is more likely to produce correct BPEL than any of the GUI editors I've tried.)

BPEL exists because Java, Python, Jython, and Ruby -- as well as more interesting and exotic stuff like Join Java and TyCO -- simply don't support those three core aspects of the merge scenario, and doing a good implementation from scratch is complex.

In its domain, BPEL is a powerful tool, and as with many things, if it hurts, you're doing it wrong... The correct strategy for working with an orchestration engine, at least if you ask me, is to treat it as a component that factors specific functional aspects out of the system or product being implemented. An orchestration engine belongs at the bottom of the architecture as a supporting component, not at the top as an application language or a design. Those functional aspects that remain after factoring out orchestration, e.g., fine-grained XML handling, should be encapsulated and attacked with the best tool for the respective job, and that typically is not BPEL or another orchestration language.

After experiments with Microsoft Word and DocBook (both of which turned out to be poor choices but for different reasons), we're using the Confluence wiki for the documentation for PXE, and I've been enjoying TimTam instead of a browser-based client.

Named for the tasty cookie from Oz that can be used as a straw, TimTam is an Eclipse-based thick client (a "thiki", as I would call it) for Confluence with a useful interface -- left-hand tree for wiki pages and a normal-sized window with reasonable key bindings, content-assist for macros, and a source/preview dual view for editing content. TimTam's still pre-1.0, but the tip builds and works well on Mac OS X with Eclipse 3.1M5a.

I've added a tag to my del.icio.us bookmarks for academic work about BPEL. The list is non-exhaustive, and I'll add items to it as I come across them. (If you know of something substantial that I'm missing, let me know.) Thanks to del.icio.us, you can also get the items as an RSS feed.