deandra.homeip.net

When I started working at the school district, in the summer of '04, backups were done using a Windows workstation running a program called PowerSync. Once a copy of the file server's data was replicated to the workstation, it was broken into manageable chunks, zipped up, and sent via FTP to the board office. There it was burned onto DVD-R and archived.

If I had to sum up this system in one word, it was "baroque". There were so many moving parts, I couldn't believe it. I was looking for better (read, simpler, more elegant,) ways to back up the district when I discovered that Netware 6.5 (which we were just starting to deploy,) included rsync as part of the Nterprise Branch Office (NBO) package. I thought "Woohoo! This is perfect!"

I began to devise a system where a single Linux box would pull copies of data, via Rsync, from the remote Netware machines, and dump it to a central datastore, where it could be archived for offsite storage. My first incarnation took advantage of the surplus DAT equipment, but that quickly proved unsuitable for two reasons: it was too slow, and far too unreliable. Even though I had written software to multiplex a tarball and write to five tapes simultaneously, it still took far too long. And even when it was finished, it usually failed due to shitty tapes or shitty drives. (I never did discover which).

The second incarnation was Procyon, a machine with about a terabyte of IDE storage for storing the local copies, and some cold-pluggable IDE devices for offsite backups. This worked well, until I ran up against the limits of the hardware. A workstation-class motherboard, and crappy IDE controllers brought the system to its knees. However, I had sold the money people on the system, well enough that they took me to the next step.

This week I received two (TWO!) Dell PowerEdge 2800 tower servers. Each machine has a Xeon 3.0GHz processor, and 1GB of RAM. What blew my mind, though, was the disk. In addition to the normal 1x8 hot-pluggable backplane, Dell added a 1x2 hot-pluggable bay into the 5.25" bays in the top of the tower. The 1x2 bay contained a pair of 36GB 15krpm SCSI drives, and the 1x8 bay in the bottom contained eight (yes, eight) 300GB 10k rpm U320 hot-pluggable SCSI drives. As an added bonus, the Dell PERC4 RAID controller, which manages both bays, is fully supported in Linux, by Dell, no less.

I configured a RAID 1 on the 36GB drives, and a RAID5 on the 1x8 backplane. With 1 drive reserved as hot spare, this gives a total of 1.8TB of storage. Per tower. This means that I have more storage in one single filesystem than any given server room I have ever seen.

Okay, enough drooling. The intention is for each server to back up half the district, and then the two servers will exchange data on the weekends, thereby preserving offsite copies of everything. All data transfers will be done using Rsync.

The whole thing is controlled with a Perl script that I wrote. Nothing too fancy; just process automation with logging to a MySQL database on another machine. This script is currently unreleased, and I currently have no plans to do so.

So far I have had very little difficulty with this new system. I have even been able to bring a couple of the older Netware servers into the loop by simply dropping the rsync NLM onto them and firing it up.

Update November 2006: I thought 1.8TB was lots, my how the times have changed. Currently sitting my office are a pair of PowerEdge 2900 servers. Each machine has a single Intel Xeon Woodcrest CPU. That's dual-core, 64-bit. There's room for a second IC. The machine's currently have 4GB of RAM, with room for more, especially since I'm running Debian AMD64 on it. When in place, the machines will only have 36GB of disk, but they'll be attached to a PowerVault MD1000, which is a SAS-connected disk enclosure that can hold 15 SATA hard drives. The two enclosures I have waiting for these machines each have eight 500GB drives in them, for a total of 4TB of space per machine.

Update February 2008: I just installed seven 750GB SATA drives in each of the MD1000 enclosures, filling them to capacity and brining my total space to about 9TB per machine. This allowed me to have enough space to start using snapshot LVs, so I can have regressive backups.