March 20, 2009

Rsync and retrying until we get it right

Ok, this isn't all that special, but I scoured the first two or three pages of Google results and didn't come up with anything that solved my problem. So here it is, Internet - may the next person be luckier than me and not have to read any man pages.

Rsync is a cool utility, especially when I'm trying to plonk my 10Gb backup onto Dreamhost's flaky backup server. But I wish I could make it retry when things go south. There are various threads on doing this, but it would seem it's not built into rsync itself.

The obvious solution is to check the return value, and if rsync returns anything but success, run it again. Here was my first try:

while [ $? -ne 0 ]; do rsync -avz --progress --partial -e "ssh -i /home/youngian/my_ssh_key" /mnt/storage/duplicity_backups backupuser@backup.dreamhost.com:.; done
The problem with this is that if you want to halt the program, Ctrl-C only stops the current rsync process, and the loop helpfully starts another one immediately. Even worse, my connection kept breaking so hard that rsync would quit with the same "unkown" error code on connection problems as it did on a SIGINT, so I couldn't have my loop differentiate and break when needed. Here is my final script:



On a side note, duplicity is pretty neat. I only wish it would support resuming of interrupted backup sessions so that I didn't have to do this in two steps. My current backup workflow is

PASSPHRASE="backup" duplicity --encrypt-key 77XABAX7 /home/youngian --exclude "**/.VirtualBox" --exclude "**/.kde" --exclude /home/youngian/tmp/ --exclude /home/youngian/backup/ file:///mnt/storage/duplicity_backups/ --volsize 100

...and then the above rsync script.

13 comments:

Unknown said...

Thank you! Your post did save me from spending another day reading man pages. This does just what I was trying to do.

Jason Martin said...

I've extended your script so that you don't need to edit each time + a few extras... and created a fork on gist.git hub.
See here...

http://gist.github.com/366269

Hope you find it usefull!!

Rob said...

thank you! you rule

Philippe said...

Very neat... I am at =the stage you were, meaning my rsync script looks very much like your first shot and is not really dealing well with error and retries.
You deserve a kudo

735984 said...

thanks ;-))

aguinar said...

Your post was really usefull for me!
I´ll show your script in my blog.
I hope you agree, if not please send me a message.

Thanks a lot!
(my blog: http://blog blog.aguinar.com)

Unknown said...

So the problem with the above approach is that if you are paranoid like myself and use "bash -e" basically everywhere, setting the initial return value to "false" will cause your script to exit.

Instead I used your script as inspiration and did it this way instead:

trap "exit;" INT TERM EXIT

retries=7

i=0

while ! rsync SOURCE USER@HOST:/PATH && \
[ $i -le $retries ]
do

echo "rsync failed! Retrying..."
let i=$i+1

done

trap - INT TERM EXIT


...mind you I haven't implemented it yet but it seemed to work when I replaced the rsync command with "false." ;-)

andymat said...

Is it possible to add a wait of 30 or 60 seconds between connection attempts?

Ian said...

@andymat Sure! You can change the rsync line from `rsync -avz ...` to `rsync -avz ... || sleep 30s`.

Roger Pack said...

somebody should tell rsync they need a "try forever" command line option...

David Mackenzie said...

Ian, thanks for this.
Question - after a successful transfer, the command doesn't work anymore until I restart my whole machine. I guess because the sucessful code is stored in memory.

I'm new to Unix, how do I reset it so I can issue the command again?

Thanks!

Matt said...

This script is pure gold, had to transfer one large file from Madagascar with a very unreliable network connection. This script worked like a charm, thankyou.

JamesH said...

Thanks! Still relevant today.