![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
Getting religion (again)
2005-05-06 02:19 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
mdlbearThe agonized screams you heard this morning came from a Mandelbear who has just discovered that his primary home directory (at work) has disappeared. Totally blown away.
Things weren't as bad as they could have been: I do almost all my work on my desktop machine, so all of my recent software development and almost all of my config files were safe. But all of my email archives (which amount to several GB these days), all of my internally-web-accessible documents -- basically everything that wasn't a config file or software under development -- was gone. And there was something massively wonky about my Firefox config, too.
As it turns out, the RAID array had crashed. A restore is in progress, but the last useable backup appears to have been done last Friday, so I lose a week's worth of email, and I'm going to have an interesting time patching the CVS version history. Luckily, I was going to do some major refactoring Real Soon Now. Sounds like a good time.
The ironic thing, of course, is that I had just gotten a huge USB drive a couple of weeks ago, and making a complete backup of both home directories onto it (with regular nightly backups) was pretty high on my list of Important Things to Do. So was making the desktop machine my primary home, with the networked server the backup. Those tasks suddenly got a lot higher on the list.
Things weren't as bad as they could have been: I do almost all my work on my desktop machine, so all of my recent software development and almost all of my config files were safe. But all of my email archives (which amount to several GB these days), all of my internally-web-accessible documents -- basically everything that wasn't a config file or software under development -- was gone. And there was something massively wonky about my Firefox config, too.
As it turns out, the RAID array had crashed. A restore is in progress, but the last useable backup appears to have been done last Friday, so I lose a week's worth of email, and I'm going to have an interesting time patching the CVS version history. Luckily, I was going to do some major refactoring Real Soon Now. Sounds like a good time.
The ironic thing, of course, is that I had just gotten a huge USB drive a couple of weeks ago, and making a complete backup of both home directories onto it (with regular nightly backups) was pretty high on my list of Important Things to Do. So was making the desktop machine my primary home, with the networked server the backup. Those tasks suddenly got a lot higher on the list.
no subject
Date: 2005-05-06 04:19 pm (UTC)no subject
Date: 2005-05-06 04:39 pm (UTC)1. RAID controller fails.
2. Two disks fail (a RAID-5 can recover from one)
I actually once had a server that both #1 and #2 happened at the same time, on the same RAID array. If it didn't happen to me, I wouldn't believe it (nor do I expect anyone else to believe it ;-) ). Hindsight is 20/20: I should have bought a lottery ticket that day.
no subject
Date: 2005-05-09 12:34 pm (UTC)These reasons are why, for truly mission-critical stuff, I like the concept of mixing software and hardware RAID: you can have a hotspare drive on your array, and then you can RAID-1 the whole shooting match to a different controller. (Better have PCI-X or something like that, though.)
no subject
Date: 2005-05-09 09:07 pm (UTC)no subject
Date: 2005-05-06 05:25 pm (UTC)backups...
Date: 2005-05-06 04:40 pm (UTC)where I work, they are referred to as karma, as in:
"The idiot CEO deleted the TPS reports of the server again. How's your karma looking today?"