Data Hoarding 101
I have digital backups of data, especially of my photos, going back almost a decade (I’m not that old) but it’s only recently that I got into creating time-snapshots of my backups, after I mistakenly deleted a bunch of (non-crucial) personal files with a stray
rsync --delete on a directory I had not backed-up.
My home server used be compatible with macOS’s Time Machine and when I kept most things on my Mac that was useful, but I changed my data hoarding habits and that solution started to not be the optimal one.
Time Based Snapshots Using rsync
rsync has got to be one of my favourite and most used utilities. I use it constantly to transfer files between machine on my local network (and remote ones) but I’ve discovered it’s very useful as a backup tool. The following is a short script that uses
rsync to create snapshots of the my data directory to another directory (they are mount points on two different drives):
#!/bin/sh TIMESTAMP=`date "+%Y-%m-%dT%H-%M-%S"` USER=user SOURCEDIR=/var/data/backups TARGETDIR=/var/redundancy # Create new backup using rsync and output to log rsync -avPh --delete --link-dest=$TARGETDIR/current $SOURCEDIR/$USER/ $TARGETDIR/$USER-$TIMESTAMP > /var/log/rsync/$TIMESTAMP.log 2>&1 # check exit status to see if backup failed if [ "$?" = 0 ]; then # Remove link to current backup rm -f $TARGETDIR/current # Create link to the newest backup ln -s $TARGETDIR/$USER-$TIMESTAMP $TARGETDIR/current else # Rename directory if failed mv $TARGETDIR/$USER-$TIMESTAMP $TARGETDIR/failed-$USER-$TIMESTAMP fi
Essentially this script creates incremental copies of backups for a given user on the system by comparing any changes to that user’s data directory (
$SOURCEDIR) with the current backup (
$TARGETDIR) and making a new snapshot of any changes with the date.
Now all that remains is to save this script somewhere and create a
cron job to run it on an automatic interval:
# daily at 5am 0 5 * * * bash /usr/local/bin/rsync-time-machine
This method uses rsync’s
--link-dest option to create hard-links so that isn’t duplicative of file data (the contents are only stored once, so you don’t use twice as much) and you can delete snapshots at will. (The links do still take up some space but it’s relatively insignificant.)
Be careful when deleting old snapshots. A new full backup of should be made before further incremental ones. I have to give a shoutout to Mike Rubel whose post was helpful in coming up with this solution.