mult.ifario.us

The classic parable of the blind men and the elephant applies to some recent proclamations about BPEL and BPMN (and about BPM and SOA, for that matter) being at odds, and there is long line of blind men waiting to take their crack at sizing-up the BPM elephant. I started out believe that the elephant is defined by its back end, the execution aspects, and that's where FiveSight developed a specialty. Nonetheless, customers (who wanted to ride on the elephant) educated me about its other properties, and combined with the expectation that standards would help define the parts of the BPM animal, the breadth of customer expectations is one of the things that drove our specialization. In 2003, we set out to build the “best back-end in the business”, gave it the code name “JLo”, and took an OEM stance. (Note to any entrepreneurs in the audience — Either you are an elephant vendor or your customers are the elephant vendors. The middle ground is not populated by successful businesses.)

I still recall a customer encounter (probably from 2002) for FiveSight's BPM-ish product that predated our BPEL effort. A customer was interested in an email widget to mass-mail customers. (Never mind that mass emailing requires more than a bit of art in dealing uniformly with bounces, proper MIME enveloping for correct viewing by different clients, delivery throttling by large email account providers, etc.) We built one more or less blind that wrapped up some of the usual JavaMail functionality and took a string as input for the body. We showed it to the customer, including a sidebar about how easy it was to implement the widget complete with a code sample (“Even your IT staff could do it, Ms. Customer!” <wink wink>). The business analyst spoke up and asked “Can it do RTF editing? We'd like to have consistent branding with a logo and specific fonts.” Ummm, yeah...

Aside: Once upon a time, on a trip to Japan, I met someone whose title was actually “Business Analyst”. I was tempted to frame his card.

The point of the story is that there are modeling aspects and execution aspects to a process, and they are closely related but not equivalent. BPMN is a modeling notation for describing processes in business terms. BPEL is a service orchestration language for combining services that was designed specifically for a WSDL-based SOA. It is possible, by design but not without effort, to generate orchestration language artifacts from modeling notation artifacts. One could observe that if nothing below the model mattered, then the orchestration language is irrelevant, and that would be true if the processes lived in a closed, black box. But that's so BPM 0.9 — vendors should know better than to imagine that they can shoehorn their customers into an architectural stovepipe when there are other alternatives available.

Agility requires degrees of freedom, and the combination of BPMN and BPEL to bind the higher-level modeling aspects of BPM onto SOA provides two important degrees of freedom. First, the modeling tool is decoupled from the execution environment. (Note that a good backmapping from the BPEL to BPMN is important, but that's a property of the mapping used.) Second, the processes are first-class citizens of a service-oriented architecture, fueling use and reuse outside of the context of the modeling environment.

Stefan is asking what the big deal is with process languages:

Whether it's workflow, orchestration, business process management, or a similar concept under some local name familiar to you, one always seems to end up looking for a flexible, accessible way to define how data flows from one processing node to another. In an XML-centric world, you'll think about XML documents that get transformed, separated into parts, aggregated again; [...]

After endless debates, you end up with some open source or commercial solution based on some half-baked standard that you just can't seem to really like. [...]

Call me a heretic, but I believe that every time somebody tries to sell you some XML-based, pseudo-declarative language it is either a) Turing-complete or b) sucks. And I have not yet seen convincing arguments that this would be different for BPEL.

Messaging Merge as Motivation

The simple "merge" operation in a messaging environment crops up every time a parallel branch of processing needs to occur asynchronously (e.g., user interaction or batch processing by an external system) -- it is the way that the response rejoins the original "thread" of processing. It is also one of the first steps toward orchestration. Writing an efficient merge widget requires reliably dehydrating processing state, correlating inbound messages with dehydrated states, and then managing the consumption of inbound messages and rehydration of suspended processing state intelligently. (In fact, it was the implementation and re-implementation of different messaging merge scenarios that led to our first orchestration design way back in early 2001.)

It Seemed Easy Enough at First Glance...

The three core aspects of the merge are transparent dehydration and rehydration, correlation of messages with instances, and "chunky" concurrency that ties it all together reliably and scalably. (Concurrency != threads; threads are just an abstraction.) These three core aspects are the answer to one of Stefan's other questions:

Now let's just say you pick the JVM as your engine, and Java as your process definition language — what exactly do you lose? Not dynamic enough? When was the last time you changed a process already in deployment without serious consideration? Even so, pick Jython, or Groovy — now what is the difference? How about using some other scripting language?

BPEL takes some heat for its syntax, and that isn't undeserved. The XML-based syntax of BPEL is unfortunate, to be sure, but it's a consequence of the fact that related specifications (WSDL, SOAP, etc.) use XML as a language and that most folks find parsing XML easier than writing a real parser for an old-fashioned language. I wouldn't necessarily go as far as Phil Windley to call it "syntactic arsenic", and I've written a good amount of BPEL by hand. (Emacs works fine for me and is more likely to produce correct BPEL than any of the GUI editors I've tried.)

BPEL exists because Java, Python, Jython, and Ruby -- as well as more interesting and exotic stuff like Join Java and TyCO -- simply don't support those three core aspects of the merge scenario, and doing a good implementation from scratch is complex.

In its domain, BPEL is a powerful tool, and as with many things, if it hurts, you're doing it wrong... The correct strategy for working with an orchestration engine, at least if you ask me, is to treat it as a component that factors specific functional aspects out of the system or product being implemented. An orchestration engine belongs at the bottom of the architecture as a supporting component, not at the top as an application language or a design. Those functional aspects that remain after factoring out orchestration, e.g., fine-grained XML handling, should be encapsulated and attacked with the best tool for the respective job, and that typically is not BPEL or another orchestration language.

Dragos Manolescu's orchestration patterns site now has draft content on-line and an RSS feed. Dragos is taking a high-level pattern-oriented view of orchestration, including runtime, design-time, management, and philosophical considerations. Orchestration, as services from services, will pick up speed as a prerequisite for reuse and recombination in service-oriented architecture, so I'm looking forward to seeing this body of work evolve and expand. (I might even find the time to contribute a pattern or two.)

Note that Dragos's work on orchestration patterns is not to be confused with the work of Wil Van Der Aalst on so-called "workflow patterns" (which would be more correctly called control-flow patterns). Wil's work takes an internal view of control-flow in process description languages with an eye to classifying existing languages (like BPEL) and finding a triple point where the expressiveness and efficiency of process description languages is simultaneously maximized.

This is a post that I've been meaning to write anyway, but it's also in response to part of JJ's comment on an earlier post.

BPEL is not a programming language. Perhaps more to the point, BPEL is not an application programming language, i.e., BPEL can't be used to write an application. Instead, a BPEL engine is a point solution to the challenges of service orchestration in the same way that a database is a point solution to the challenges of data management. The statement that "SQL is not an application programming language" is analogous.

This summarizes my expectations of and aspirations for BPEL. BPEL holds promise for reducing the software engineering cost and complexity of building applications with services. BPEL will be a substantial part of B2B applications, integration applications, and all manner of other software engineering endeavors, but it's only a part of the overall picture.

Chris Peltz (from HP) has written an overview of web services orchestration that was cited by Floyd Marinescu on TheServerSide.com:

Review of Web Services Orchestration Tools and Standards. Today's web services platforms allow exposing application logic as web services, but there also needs to be a standard way to combine web services together to form business processes. In 2002, a number of new standards were introduced to address this including BPEL4WS and WSCI. A recent whitepaper from HP provides a review [of] standards, products, and intiatives related to ws orchestration.

Other than missing a couple of important vendors in the space (e.g., us), the report focuses on BPEL4WS, WSCI, and BPML.

The missing pieces of the report, in my view, are highlighted in the section called "early work" and the bibliography. The following sentence introduces the idea that any web services orchestration specification is fundamentally dependent on a notion of long-running transactions to support its execution:

Traditional transactional systems that are ACID-based are typically not sufficient for long-running, distributed transactions.

(BTP is comparable to WS-T/WS-C, but it surprising to see it given no coverage, considering that HP spinout Arjuna has been heavily involved in work on the BTP specification.) Nonetheless, the "early work" section picks up with WSFL (which is really just MQWorkflow done with web services), and the bibliography doesn't contain anything before 2000.

The fact is, there is a significant body of work starting from the late 1970's (i.e., Gray, "Notes on Data Base Operating Systems") and extending through the 1980's (e.g., J. E. B. Moss, "Nested transactions: an approach to reliable distributed computing", which you can buy on Amazon.com) and 1990's (e.g., Reuter/WÒ¤chter, The ConTract Model) on extended transaction models that doesn't even get a nod but is very much worthy of consideration. That isn't to say that there isn't interesting work ongoing, e.g., The CORBA Activity Service Framework for Supporting Extended Transactions from 2001.

Maybe this seems like heady or esoteric stuff at first glance, but if we honestly believe that service-oriented architecture and orchestration are the future of software, then any talk of "standards" or "specifications" with a less than fully rigorous consideration of the literature and previous implementations would be irresponsible.