mult.ifario.us

Kids absorb and imitate. Two recent events reminded me about just how well they absorb and how strong the desire to imitate can be.

Television is the lead plumbing of our age, and we give the kid only small doses. She'll get more than enough of it through friends and relatives anyway. Only PBS/Sprout, only shows with slow-paced visuals. She's seen a couple episodes of Sesame Street, at most. Nonetheless, she knows who Elmo is. A couple of weeks ago at Freddy's, she selected an Elmo-branded baby toothbrush for herself. It went back on the rack; we bought her an unbranded one. Then, at Whole Foods, she spontaneously snagged some Elmo-branded oatmeal off the shelf. That went back on the shelf, too, since oats have never been her favorite, and I'd rather give her the good stuff.

We took the kid out to a reasonably nice restaurant recently, ordered escargot as an appetizer and gave her a few slices to try. Since she was about six months old and would lunge for food on the table, she's wanted to at least try anything that we're eating, and usually she likes it. We manage to keep the things from her that she shouldn't have (unpasteurized cheeses, less than fully cooked eggs, etc.), but once she catches you eating something, she wants to try it. It's very likely that she'll remember the appearance or packaging and try to sneak up to the table, climb onto a chair, and snag some while the table is being set. She has also started insisting on having the same utensils that we have for a meal. This doesn't work out well for chopsticks, but it works OK for forks. Her version of using a fork is to spear something with the fork and then take it off with her hand and pop it in her mouth. (She does does know how to use a fork properly.)

For anyone who's been wondering why this blog has been up and down over the past week, it's a slow-motion battle between the memory police at TextDrive killing Typo instance that hosts the blog and either a FastCGI dispatcher or a nanny cron job starting it back up. The onus is clearly on me to figure out what's burning memory, and my first inclination was to naively google for Ruby profilers. Here's a rambling account of what I did to conclude that I'm probably out of luck as far as a quick cure for the issues and then to address them.

There are a couple of speed-oriented Ruby performance profilers, the built-in one and ruby-prof, but there are no space-oriented profilers. There was a brute-force approach based on ObjectSpace.each_object in an old mailing list post from Michael Garniss that looked suitable, so I integrated it into the main controller in Typo as an after_filter and fired-up several concurrent wget commands to walk around on a production configuration on my development box at home:

while true; \
do wget -nv -r --delete-after http://localhost:3000; \
done

(There is no reason to try to set it on fire with something like ab.) That won't catch any issues with the vanilla two dispatcher lighttpd/FastCGI configuration that I use on Textdrive, but it should catch any issues with Typo internals, badly behaved sidebars, etc.

With the profiling code integrated, a request that includes the dump takes several seconds to complete, and there are several hits per page; so I added a class variable (@@no_sooner_than) and a little logic so that profiling requests would only run once a minute or so. With several wget walkers working, top reports that the server runs along at a happy 80-90Mb, and eyeballing the profiling output shows memory usage oscillating between <7Mb and ~20Mb without any perceptible upward trend over the course of an hour and a half. (That said, that's all the data I captured, as WEBrick locked up completely after that hour and a half.)

Armed with the information that there wasn't an easy fix for the memory issues, I switched the FastGCI configuration for the production instance to a single dispatcher from the previous two, pointed a couple of wget walkers at it, and tracked memory usage and process id at the commandline, like so:

while true; \
do ps mux | grep ruby | grep -v grep; \
read -t 30; done

I also changed the wget walker command to provide more useful information:

wget -S -r -b -l 4 --delete-after http://mult.ifario.us \
-a /tmp/log_id

where id is a unique number per walker, and so far, so good. Crunching the wget output through shell commands (awk, grep, cut, sort, uniq -c, etc.), e.g.:

cat log* | grep HTTP/1.1 | cut -f 4 -d ' ' | sort | uniq -c

says that mult.ifario.us is consistently returning snappy HTTP/1.1 200 responses about two nines (99.x%) of the time, which isn't great but isn't awful. (Really it's more like 2.5 nines, i.e., −log₁₀(0.003), but who's counting?)

This is one time when I've missed some of the Java runtime environment's capabilities (i.e., the JVMTI) in other language runtimes, but no rocket science was required to get Typo under control.

Tim Bray on ALT-TAB:

[...] It’s only taken me a day or two to decide that the Windows/Gnome view that a window is a window is a window, and alt-tab works at a window at a time, and you don’t have to bear in mind the difference between an app an a window, is right, and OS X is wrong about this. [...]

Funny. As I've been bouncing between Linuxes, Windows, and Mac OS X, I've come to the opposite conclusion because, e.g., I can't tell from the ALT-TAB display which terminal window I'll get, and if I then get it wrong, I have to go through the stack of windows again and make another guess.

Fragile casts are a pet peeve of mine. A cast is fragile if it would be unexpected by the user, e.g., they pass in a class that implements an interface but the method casts it to a specific implementation class. For example:

public void method(Iface x) {
  some(code);
  ((IfaceImpl)x).foo(); // Doh!
  more(code);
}

A more egregious version:

public void setFoo(Iface x) {
  this.foo = x;
}

public void method() {
  some(code);
  ((IfaceImpl)foo).func(); // Doh!!
  more(code);
}

The user's experience of the fragile cast is like stepping on a rake in tall grass and getting hit in the nose with the handle: a sudden and unexpected ClassCastException hidden away in some code that they didn't write. Of course, ClassCastException doesn't tell you what cast failed, so if the method is buried deep within a closed-source API, it may take the user a long time to figure out what's wrong. (Or they might just get annoyed enough to run it all through Jad and see what's what.)

The prescription is simple: Don't do that! If the user is only supposed to use subclasses of some class, then give them a base class instead of an interface. Or, as another alternative, alter your design so that functionality in the base class is externalized and the method can rely only on the interface.

As a consumer/producer of open source, it was interesting to see Google Code hosting launch today. It also explains why people from Google were trolling with questions about how Google could help open source... (I should have asked for a search appliance for the Haus now that I'm thinking about it.) It's pleasantly minimal in that it fits cleanly into the spectrum of available infrastructure below the level of a SourceForge, RubyForge, or Java.net: no mailing lists, no forums, no releases, and no project entry criteria — just fill-in a form to get 100Mb of subversion space and an ultralight bugtracker.

From a producer perspective, no strings attached, free subversion hosting is a great offering, although I'd be more likely to use something private and something cooler than subversion — like darcs or Bazaar or another system that supports both push and pull branching semantics — until and even after a project is ready for public consumption. (Subversion+svk would count.) As a consumer, my question is how I'm going to filter out the crap. Nothing is worse than seeing a promising-sounding project and then finding out that there's nothing of value there. On SourceForge, there's plenty of deadwood, but I can look for releases, at activity levels on mailing lists, or at project statistics. On Java.net, I can look at releases, mailing lists, or at rankings. A good start on a crap filter would be a simple voting model for projects, e.g., allow a user to leave a star on a project that they found actually useful. (Of course, something like the Google Finance plotting widget applied to subversion activity and branches/tags wouldn't hurt, either.)

My answer to the “How could Google help open source?” at the time was straightforward: contribute effort and even sponsorship to the projects that Google developers find useful. This is what everyone does (or at least what everyone is supposed to do). Barriers to entry have never been an issue for open source projects, but drawing together a community and channeling combined energy into defining and creating good software have always been and remains a challenge.

An integration environment for a complex system is challenging to maintain, and even a relatively small number of active development teams (e.g., two) can generate a good number of concurrent updates to the target system. Here are some thoughts about how to stop worrying and love a little chaos.

My perspective on software testing is that it is intended to verify a set of assertions about a piece of software in a given environment. Engineering process ensures that the artifacts placed in a given environment have a tight relationship with the source code and configuration for the software, and this is an absolute. It is tempting to strive for a similar level of control over the environments for different stages of testing, but this sort of perfection can actually be the enemy of building good software. (“Good software” is software that does the work it is supposed to do, ideally making money for a business.)

For the sake of argument, suppose that the system under development, SystemX, has a dependency on SystemY. If SystemY is down or degraded in the integration environment, two things should happen. First, SystemX should quickly detect the issue and exhibit an expected, graceful degradation of functionality because that's what it's supposed to do in production under the same circumstances. Second, SystemX should be notifying the SystemY team that the system is degraded. Ideally this is an automated thing, but at a bare minimum it is the lead for SystemX getting in synchronous communication with the lead for SystemY and finding out what's up. Third, the SystemX team needs to sit down and exercise some judgement about how SystemX would behave if SystemY were operating normally. That judgement can be based solely on the knowledge of how SystemX has changed with respect to the last integration release where SystemY was operating normally, but careful and detailed management of changes is critically important. (What revision was known to be working correctly with SystemY? What changes have been made since then?) Note that none of this requires any action on the part of the SystemY team. (It's neither the business nor the responsibility of the SystemX team how SystemY operates and vice versa.)

As bullets:

• Test what you can test.
• Own your outcomes.
• Ensure that your own house is in order and reason from what you know.

This one is only of moderate difficulty, but it's still worth a mention (since I stumbled across it today debugging someone else's code). What could go wrong with the following Java code?

try {
  [code]
} catch (Exception  e) {
  throw new RuntimeException(e);
} finally {
  [code]
}