How our websites don't quite work yet

[mdlbear]

This post is an attempt to get my thoughts in order about how the household web space is evolving. Mostly cut-tagged, because it's mainly of interest only to household members working on websites (waves at chaoswolf) and to that subset of my readers who are working on websites and/or have a home fileserver.

Back in the old days things used to be pretty simple: all of my web space was under one directory hosted by my ISP, rahul.net. I wrote a bunch of makefiles to upload changes, first using FTP and later SSH. Eventually I got a DSL line and was able to host a few sites locally; all I had to do was put a configuration file in the top-level directory that said where things went.

Meanwhile, I have a home fileserver (originally central, as in "Grand Central Starport", and currently nova, as in "new"). Up until last year everything related to websites and system administration was under a single directory, /usr/local/starport.

I've always kept full weekly backups using rsync; currently those are done to a disk attached to the machine in the bedroom.

How things work now:

About a year ago I acquired 500GB of space on dreamhost.com, with an eye toward using it for backups and of course web space. About the same time I got a fast DSL connection with sonic.net, keeping the old DSL line because that's where mail gets delivered.

I also split up the space on the fileserver; it's now in four different directories.

• /share has the actual websites, as well the related makefiles and scripts.
• /users has working directories, mainly mine (/users/steve) and another one for recording (/users/record) where I've moved all of the track files, raw concert recordings, and so on. There's also /users/starport, which is where all of the configuration files and notes on system administration from /usr/local/starport ended up. /users/steve/Albums contains all of the data associated with CDs.
• /mm has shared "multimedia" files, in particular /mm/audio, which is for ripped CDs and downloaded music.
• /home/starport contains backups for every server's /etc directory, and other configuration files that need to be kept inside the firewall.

The general idea is that everything in /share is intended to be up on the web -- it might be hidden or password-protected, but only to protect third-party copyrights, not to protect privacy. Similarly, nothing in /users and /mm needs to be kept secret, although much of it is either owned by third parties or is work-in-progress that isn't yet ready for prime time.

Everything that needs protecting is in /home, and that's backed up locally. The rest is being backed up every couple of days (on an irregular basis) to my space at dreamhost, and downloaded from there to a USB drive that I keep at work. I used it to initialize the dreamhost site because work's T1 was considerably faster than my wimpy upload pipe.

How things need to work:

A couple of things need to happen, and I'm finally getting enough round tuits together to make them happen:

• I need to put together a mostly-automated procedure for shuttling a USB hard drive between home and work. We recently got a DS3 connection, meaning that it's about 75 times faster for uploads than my current DSL line. Wav files (especially concert recordings) and audacity projects are just too damned big to upload from home; ripped CDs are marginal but it looks as though a day's worth of ripping can be uploaded overnight.
• I need to get rid of my old DSL line, and move email (thestarport.org) to a server that somebody else maintains spam filtering on. Presumably one of the ISPs; rahul.net makes the most sense because that's where it is already.
• I need to move my account at rahul.net, currently a $45/month premium account, to a less expensive account or drop it altogether.
• I need to move all of the websites to dreamhost, unless I decide to keep one or two at rahul.net.
• I need to write a CGI for dreamhost that other users can use to trigger an rsync upload of a website, so that they don't need to use ssh. I need to modify the website makefiles to use it.
• I need to set up a procedure for making encrypted off-site backups of /home. (This is closely related to something I'm doing at work, which is a good thing. It's been hanging fire for a year now.)
• I need to fix the way stuff is published. Currently it gets copied from a working directory (under /users) to a web mirror directory (under /share/Web) and uploaded from there. Obviously, if everything's being uploaded to dreamhost anyway, I can just update in place and use symlinks.
• Some working directories may need customization on their way to multiple websites. That needs to be done as well.

I still have to work out how web-based collaboration will happen; that will be of interest to members of Tres Gique, as well as anyone else thinking of collaborating with me on a website, CD, cookbook, or any other project. Clearly, simple uploading isn't enough at that point. More on that further upwhen.