Wednesday, September 21, 2005

The Many Disguises of Incompetence

or How Luck Makes Heroes of Goats

I have an expectation, and optimism, that information will emerge from New Orleans and be placed into sufficient context to grow into something like an updated, and useful, Conventional Wisdom.

I suffered a minor setback today.

Consider for a moment this WaPo article Backups Enabled Systems to Survive: Data Protection Paid Off After Katrina

Our Hero in this little vignette is:
"...interim technology manager Rajeev Jain [who] entered the building not knowing what to expect. The ground floor of the New Orleans school system headquarters was under three inches of water, and when he headed upstairs, he saw that the fourth-floor ceiling was damaged and leaking."
So far so, so good. Rajeev Jain, undoubtedly doing his job, was surveying the damage to the school system HQ. He then
"...hailed police to sledgehammer through a locked door."
and therein lies the fickle finger of fate pointed by Lady Luck who, on that day at least, was smiling favorably on Rajeev Jain.
"He found what he was looking for in a storage closet: 170 dry and apparently undamaged computer backup tapes storing recently updated payroll records and other critical financial information."
What possible "incompetence" could this happy scenario demonstrate?

Rajeev Jain should never have gone looking, nor needed to summon police to sledgehammer through a locked storeroom door, in the flooded, water dripping from the-damaged 4th-floor ceiling, school system HQ. Those computer backup tapes storing recently updated payroll records and other critical financial information should never have been in a storage closet in the school HQ. They should have been stored well outside the range of any disaster that might reasonably be expected to befall the school system HQ. Had this been a simple fire, or had the roof collapsed and wiped out Our Little Storeroom, Rajeev Jain would be sweating bullets and wondering how he was going to answer some very pointed questions from some irate people.

Lest you doubt my word for this:
"Government institutions and large companies generally had adequate backup systems in place and data-recovery contracts with firms such as IBM to help rescue damaged data tapes and rebuild software systems. The best-prepared had backup files stored on computers outside the hurricane zone." (emphasis mine)
Although storing backup files "on computer systems" is a function of how quickly one needs to achieve recovery. Storing backup files "in adequately protected facilities outside the hurricane zone" would be sufficient for some cases.

Perhaps I come across as a grump here, but I've lived through this and counting on the storage closet to protect irreplaceable assets, electronic or otherwise, is not a good idea.

2 comments:

MeaninglessHotAir said...

Backing up, cleaning up, moving tapes out of the office, all of these things take time and money. Meanwhile, people are under deadlines, have lots of day-to-day responsibilities, and budgets always need to be cut. This is why backing up is seldom done, even out of the office into another office. If my company's building had a fire we would be out of business. It's generally too low a probability event to justify the extra cost involved.

Rick Ballard said...

Even for a small business, it really doesn't take much effort to get critical financial data onto a disc and into a purse or briefcase every night. 'Course, it better be a pretty well trusted employee.

MHA is right about how most businesses handle other than critical data. It takes a manager with a cattle prod to get most employees to back up. And then somebody has to turn the cattle prod on the manager.

What data recovery companies are likely to do well from this, Knuck? (he asked wistfully as he looked at his portfolio value)