March 20, 2009

Rsync and retrying until we get it right

Ok, this isn't all that special, but I scoured the first two or three pages of Google results and didn't come up with anything that solved my problem. So here it is, Internet - may the next person be luckier than me and not have to read any man pages.

Rsync is a cool utility, especially when I'm trying to plonk my 10Gb backup onto Dreamhost's flaky backup server. But I wish I could make it retry when things go south. There are various threads on doing this, but it would seem it's not built into rsync itself.

The obvious solution is to check the return value, and if rsync returns anything but success, run it again. Here was my first try:

while [ $? -ne 0 ]; do rsync -avz --progress --partial -e "ssh -i /home/youngian/my_ssh_key" /mnt/storage/duplicity_backups backupuser@backup.dreamhost.com:.; done
The problem with this is that if you want to halt the program, Ctrl-C only stops the current rsync process, and the loop helpfully starts another one immediately. Even worse, my connection kept breaking so hard that rsync would quit with the same "unkown" error code on connection problems as it did on a SIGINT, so I couldn't have my loop differentiate and break when needed. Here is my final script:



On a side note, duplicity is pretty neat. I only wish it would support resuming of interrupted backup sessions so that I didn't have to do this in two steps. My current backup workflow is

PASSPHRASE="backup" duplicity --encrypt-key 77XABAX7 /home/youngian --exclude "**/.VirtualBox" --exclude "**/.kde" --exclude /home/youngian/tmp/ --exclude /home/youngian/backup/ file:///mnt/storage/duplicity_backups/ --volsize 100

...and then the above rsync script.

3 comments:

DonThrash said...

Thank you! Your post did save me from spending another day reading man pages. This does just what I was trying to do.

Jason said...

I've extended your script so that you don't need to edit each time + a few extras... and created a fork on gist.git hub.
See here...

http://gist.github.com/366269

Hope you find it usefull!!

Robin said...

thank you! you rule