random neuron misfires

Thursday, June 1st, 2006 at 10:46 pm | 5,805 views | trackback url

I’ve been looking for a good snapshot tool for FreeBSD, and I think I’ve finally found it.

Today, I stumbled across a slick little tool called rsnapshot, written by Nathan Rosenquist. It was featured in the BSD Hacks book by O’Reilly Media.

rsnapshot is based on rsync, and allows quick snapshots of a filesystem in-time, using hardlinks to preserve space. Basically you configure it using the default config file, tweak a few options, and check the syntax.

rsnapshot, like the rsync tool it is built upon, has a wealth of options (including inheriting those from rsync), including ‘configtest’ to check the syntax of the config file, and ‘du’ to check the size of the snapshot tree.

Here’s an example from my own backups:

# rsnapshot du
771.7G    /usr/local/array/.snapshots/hourly.0/
771.7G    total

You can set up rsnapshot to run from cron (and you should), like this:

0 */4 * * *     /usr/local/bin/rsnapshot hourly
30 23 * * *     /usr/local/bin/rsnapshot daily

From here, you’ll see it create a tree like:

[22:28:20 Thu Jun 01]
[117] flea:/usr/local/array/.snapshots
# ls -l
drwxr-xr-x    7 root     wheel         4096 May 28 00:00 daily.0
drwxr-xr-x    7 root     wheel         4096 May 27 00:00 daily.1
drwxr-xr-x    7 root     wheel         4096 May 26 00:00 daily.2
drwxr-xr-x    7 root     wheel         4096 May 25 00:00 daily.3
drwxr-xr-x    7 root     wheel         4096 May 24 00:00 daily.4
drwxr-xr-x    7 root     wheel         4096 May 23 00:00 daily.5
drwxr-xr-x    7 root     wheel         4096 May 22 00:00 daily.6
drwxr-xr-x    7 root     wheel          512 May 29 00:00 hourly.0
drwxr-xr-x    7 root     wheel          512 May 28 20:00 hourly.1
drwxr-xr-x    7 root     wheel          512 May 28 16:00 hourly.2
drwxr-xr-x    7 root     wheel          512 May 28 12:00 hourly.3
drwxr-xr-x    7 root     wheel          512 May 28 08:00 hourly.4
drwxr-xr-x    7 root     wheel          512 May 28 04:00 hourly.5

Another great feature is how it intelligently rotates the hourlys into daily and folds them together.

When ‘rsnapshot daily’ is run, it will rotate all the daily.X directories and then copy the contents of hourly.5 into daily.0. hourly.0 will always contain the most recent snapshot, and daily.6 will always contain a snapshot from a week ago. Unless the files change between snapshots, the “full” backups are really just multiple hard links to the same files.

If a file changes at any point, the next backup will unlink the hard link in hourly.0, and replace it with a brand new file. This will now take double the disk space it did before, but it is still considerably less than it would be to have full unique copies of this file 13 times over.

Once I manage to get this all working, I’m going to be migrating the Windows login and profiles over to the Samba server (now acting as a PDC) and start doing backups of that data into the snapshots as well.

The volume that this data is stored on and backed up on is also protected by geli, which wraps around the cryptographic GEOM class available in the BSD kernel. I started with the GEOM encryption, but it was a bit slower than I needed.

‘geli’ improves upon GEOM by a little bit, in the following ways:

• Utilizes the crypto(9) framework — when cryptographic hardware is available, geli will use it automatically.
• Supports multiple cryptographic algorithms (currently AES, Blowfish, and 3DES).
• Allows the root partition to be encrypted. The passphrase used to access the encrypted root partition will be requested during the system boot.
• Allows the use of two independent keys (e.g. a “key” and a “company key”).
• geli is fast – performs simple sector-to-sector encryption.
• Allows backup and restore of Master Keys. When a user has to destroy his keys, it will be possible to get access to the data again by restoring keys from the backup.
• Allows to attach a disk with a random, one-time key — useful for swap partitions and temporary file systems.

And here’s the best part..

“Unlike cumbersome encryption methods that encrypt only individual files, gbde and geli transparently encrypt entire file systems. No cleartext ever touches the hard drive’s platter.”

The goal for this server, besides its many workhorse duties, is to be the backup and mirror server for several terabytes of projects, such as Project Gutenberg, CPAN, LDP, Wikipedia, and dozens of other projects.

Its also moving into its role as PDC with Samba using LDAP authentication over SSL/TLS for all clients, who will have their profiles and home directories transparently mapped and mounted on the server, backed up regularly with rsnapshot. It should all work out great when I’m done with it.

Last Modified: Saturday, March 5th, 2016 @ 21:31

This entry was posted on Thursday, June 1st, 2006 at 10:46 pm and is filed under Uncategorized. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.