The Many Disguises of Incompetence

Wednesday, September 21, 2005
or How Luck Makes Heroes of Goats

I have an expectation, and optimism, that information will emerge from New Orleans and be placed into sufficient context to grow into something like an updated, and useful, Conventional Wisdom.

I suffered a minor setback today.

Consider for a moment this WaPo article Backups Enabled Systems to Survive: Data Protection Paid Off After Katrina

Our Hero in this little vignette is:
"...interim technology manager Rajeev Jain [who] entered the building not knowing what to expect. The ground floor of the New Orleans school system headquarters was under three inches of water, and when he headed upstairs, he saw that the fourth-floor ceiling was damaged and leaking."
So far so, so good. Rajeev Jain, undoubtedly doing his job, was surveying the damage to the school system HQ. He then
"...hailed police to sledgehammer through a locked door."
and therein lies the fickle finger of fate pointed by Lady Luck who, on that day at least, was smiling favorably on Rajeev Jain.
"He found what he was looking for in a storage closet: 170 dry and apparently undamaged computer backup tapes storing recently updated payroll records and other critical financial information."
What possible "incompetence" could this happy scenario demonstrate?

Rajeev Jain should never have gone looking, nor needed to summon police to sledgehammer through a locked storeroom door, in the flooded, water dripping from the-damaged 4th-floor ceiling, school system HQ. Those computer backup tapes storing recently updated payroll records and other critical financial information should never have been in a storage closet in the school HQ. They should have been stored well outside the range of any disaster that might reasonably be expected to befall the school system HQ. Had this been a simple fire, or had the roof collapsed and wiped out Our Little Storeroom, Rajeev Jain would be sweating bullets and wondering how he was going to answer some very pointed questions from some irate people.

Lest you doubt my word for this:
"Government institutions and large companies generally had adequate backup systems in place and data-recovery contracts with firms such as IBM to help rescue damaged data tapes and rebuild software systems. The best-prepared had backup files stored on computers outside the hurricane zone." (emphasis mine)
Although storing backup files "on computer systems" is a function of how quickly one needs to achieve recovery. Storing backup files "in adequately protected facilities outside the hurricane zone" would be sufficient for some cases.

Perhaps I come across as a grump here, but I've lived through this and counting on the storage closet to protect irreplaceable assets, electronic or otherwise, is not a good idea.


MeaninglessHotAir said...

Backing up, cleaning up, moving tapes out of the office, all of these things take time and money. Meanwhile, people are under deadlines, have lots of day-to-day responsibilities, and budgets always need to be cut. This is why backing up is seldom done, even out of the office into another office. If my company's building had a fire we would be out of business. It's generally too low a probability event to justify the extra cost involved.

Knucklehead said...


There are no end of reason that you and I, and small business people, fail to do that which is sensible and potentially disastrous to fail to do.

There is no reason for an director of information technology, interim or otherwise, to not adequately protect essential data. That is, specifically, part of his job.

When the towers of the WTC collapsed there was nobody with even the most remote connection to professional IT who wasn't involved in various disaster recovery analysis exercises. Nobody.

For a while there was even a common acronym associated with DR - SMHIG (Smoking Hole in the Ground).

"So, Mr. Director of IT, what are you going to do if your data center is suddenly converted into a SMHIG?"

"Well, I'm gonna beg, borrow, steal, rent, lease and purchase computers real quick, and then upload my backup..."

"And where are your backups?"

"Ummm... they're in the SMHIG."

While I believe it is patently unfair to demand of a POTUS,

"Four long years after 9/11 how do you explain why the City of New Orleans didn't have a hardened emergency communications network!?!"

I think it is completely fair to ask that question of the NOLA PD. It is also completely fair to ask of an IT director why, four long years after 9/11, backups of critical files are sitll kept onsite rather than far, far, away.

Rick Ballard said...

Even for a small business, it really doesn't take much effort to get critical financial data onto a disc and into a purse or briefcase every night. 'Course, it better be a pretty well trusted employee.

MHA is right about how most businesses handle other than critical data. It takes a manager with a cattle prod to get most employees to back up. And then somebody has to turn the cattle prod on the manager.

What data recovery companies are likely to do well from this, Knuck? (he asked wistfully as he looked at his portfolio value)

Knucklehead said...

Darned good question, Rick. There are a number of ways that companies react to this sort of thing.

Some scurry around, wave their hands, make plans, issue RFIs, then do little or nothing.

Those who do take action chose from a pretty wide variety of ways to deal with this sort of thing. You can just up and outsource/offsite your entire operation to a corporate application hoster (IBM is the Big Kahuna there). I'd expect them to make some money on this. I'd expect HP and SunGard to get some activity.

The two or three biggest network guys (like Verizon) will get additional business out of it.

The big-honkin-storage guys (IBM, Hitachi, EMC, STK) will sell a lot of iron for a little while. The big SW houses like Oracle and SAP will also likely get a bump.

The standard consulting houses will get some contracts (the EDS, CSC kinda guys).

Somebody, somewhere, is going to figure out how to get through to all the small shops. The guys who do things like backup their business computers and take the CDs home and suddenly had both business and home wiped out. I have no idea who plays in that space. This is where any big gains would be, IMO.

The big outfits will see additional business but it isn't market moving kind of gains for them.

The big problem is that its hard work and difficult to get the little guys to spend money hardening their IT. The deep pockets have learned their lessons and done it already. The shallower pockets are the hardest to tap and keep their attention. They are also the ones who can least afford to be out of business for a few days or weeks.