Never use the same communications channel to both handle load and send management information or control.
Here are two instances of the Nagging Nanny Management Antipattern in action, one with MQSeries and one with Ruby on Rails.
What do you get when you cross queues and topics? In the olden days of FiveSight (like 2000), we had an integration server product that made it relatively easy to hook JMS queues (and other sources of "messages" like directories, sockets, etc.) to business logic. (Think of it as an early approximation to something like Mule or ServiceMix.) The first incarnation of product the used JMS topics to transmit management messages (like "pause", "stop", "deploy") from a management console to one or more remote servers. This was all well and good until our first MQSeries deployment (5.1, I think; more ancient history), and at the time, MQSeries simulated topics on top of its queues. The result was that a heavily-loaded server would be unable to respond to management messages buried amongst thousands of other messages running through the same queue manager. We refactored the design to use a separate communication pathway for management messages, and things worked as intended.
Are you busy? Some folks were making a similar mistake managing web server instances at TextDrive recently. A good number of users were using an innocuous looking nanny script to ensure that their instances of lighttpd/RoR were up and running. However, note that the nanny script asks the server how it's doing by sending a request:
/usr/local/bin/curl -I http://YOURDOMAIN.com:YOURPORT
Thus, if the request times out because the server's loaded down and unresponsive, the nanny script will bounce it, making the problem worse. Some further complications with not cleaning up resources spawned by the killed lighttpd instances made it even more fun.










