mult.ifario.us

A couple of weeks back, I wrote a simple but well-instrumented Java framework to handle SEDA-like use cases (thread pools linked by queues) for a consulting customer. The java.util.concurrent package and friends makes this sort of thing much easier than it used to be, and it was surprisingly easy to crank it out. As a smoke test, I set up a torture test for a simple configuration and left it running over night, and it appeared solid — no memory or thread leaks, no lock-ups.

Someone working on a different problem needed something similar and took the framework for a quick test drive, ending up with an out of memory error after a night of doing nothing! It turns out that there was a bug that meant that the poll(long,TimeUnit) on an empty LinkedBlockingQueue leaks, and I'd never run across it in testing because I hadn't tested what happens when the system has no load for an extended period.

The lesson is that no load doesn't mean that the system is actually doing nothing, and it's an important scenario to add to a test plan.

An integration environment for a complex system is challenging to maintain, and even a relatively small number of active development teams (e.g., two) can generate a good number of concurrent updates to the target system. Here are some thoughts about how to stop worrying and love a little chaos.

My perspective on software testing is that it is intended to verify a set of assertions about a piece of software in a given environment. Engineering process ensures that the artifacts placed in a given environment have a tight relationship with the source code and configuration for the software, and this is an absolute. It is tempting to strive for a similar level of control over the environments for different stages of testing, but this sort of perfection can actually be the enemy of building good software. (“Good software” is software that does the work it is supposed to do, ideally making money for a business.)

For the sake of argument, suppose that the system under development, SystemX, has a dependency on SystemY. If SystemY is down or degraded in the integration environment, two things should happen. First, SystemX should quickly detect the issue and exhibit an expected, graceful degradation of functionality because that's what it's supposed to do in production under the same circumstances. Second, SystemX should be notifying the SystemY team that the system is degraded. Ideally this is an automated thing, but at a bare minimum it is the lead for SystemX getting in synchronous communication with the lead for SystemY and finding out what's up. Third, the SystemX team needs to sit down and exercise some judgement about how SystemX would behave if SystemY were operating normally. That judgement can be based solely on the knowledge of how SystemX has changed with respect to the last integration release where SystemY was operating normally, but careful and detailed management of changes is critically important. (What revision was known to be working correctly with SystemY? What changes have been made since then?) Note that none of this requires any action on the part of the SystemY team. (It's neither the business nor the responsibility of the SystemX team how SystemY operates and vice versa.)

As bullets:

• Test what you can test.
• Own your outcomes.
• Ensure that your own house is in order and reason from what you know.

I needed an algorithm to traverse a graph of relationships and test whether it was a connected tree or not, so I coded-up a quick (and relatively efficient) search. No matter how many times I code graph searches, I know myself well enough to expect a couple of mistakes, so I went to write some tests after I'd finished with the algorithm. The reason that I'd put off writing the tests was that the signature for my search was the following:

static FancyIface findRoot(Map<FancyIface,FancyIface[]> graph)

FancyIface is an interface with a bazillion methods and where the API doesn't expose any implementations. Rather than mocking instances of FancyIface for testing purposes, it makes sense to genericize the method:

static <T> T findRoot(Map<T,T[]> graph)

and then code the tests with something nice and simple, like integers:

public void testFindRoot() {
  Map<Integer,Integer[]> m = new HashMap<Integer,Integer[]>();
  m.put(1,new Integer[] {2,3});
  m.put(2,new Integer[] {4});
  // should return 1, as this is a tree rooted at 1.
  assertTrue(Foo.findRoot(m) == 1);
  m.put(3,new Integer[] {4});
  // should return null because this is no longer a tree.
  assertFalse(Foo.findRoot(m) == null);
}

And that's it.