Backups (I love it when a plan comes together)

[mdlbear]

Did backups this morning using the new SATA backup drive and new scripts. Fast as a bat: 10 minutes for 273GB of data.

I still haven't done the rest of the associated reorganization; I just wanted to get a snapshot of the current state.

I used to back up a lot of separate directories from many different machines, including /etc from all the servers. It was mostly done with rsync over ssh, which made it slow and required me to be logged in with my ssh-agent running. Not usually a problem, but...

I now have it down to three directories: /home, /mm (where all the multimedia lives), and /local (where all the public websites live, among other things). They're all mounted via NFS on the same machine as the backup drive, so I can do unattended daily backups via a crontab if I want to. They're all snapshotted using cp -al, which makes a mirror tree using hard links, with a monthly snapshot kept around permanently.

The next step is to back up all of the various servers' state into those backed-up directories: /etc and other configuration files will go into a directory in /home that's only readable by root. This does require ssh, but fortunately doesn't have to be done very often. Once a week, as I've been doing it, will be sufficient.

There are also a few web directories, local home directories, and working storage on various specialized machines: the DSL line, the recording box, and so on. Eventually the very few of these that aren't already just mirrors of internal directories will also have to be backed up onto the main fileserver; I'm not going to worry about that right now.

The next step is the big one: offsite backups. For that, I'll have to shuffle stuff around so that all the really private stuff is in /home, and also ensure that anything that wants to have old versions archived offsite is under version control (which it mostly is now). Multimedia is already split pretty cleanly between my recording stuff and stuff ripped from CDs and DVDs. I'll also have to re-jigger /local a little to make a cleaner separation between the third-party stuff that normally lives in /usr/local and the locally-developed content that's currently in /local/starport.

At that point I'll be able to upload /home to an encrypted archive, and everything else to an archive I won't have to worry too much about. ("Everything else" may include work in progress, and a few things I don't have publication rights to like live recordings of other peoples' songs, but it won't include anything that will cause trouble if it accidentally gets out or if (when) the NSA gets around to snooping it.)