Rsync is a cool utility, especially when I'm trying to plonk my 10Gb backup onto Dreamhost's flaky backup server. But I wish I could make it retry when things go south. There are various threads on doing this, but it would seem it's not built into rsync itself.
The obvious solution is to check the return value, and if rsync returns anything but success, run it again. Here was my first try:
while [ $? -ne 0 ]; do rsync -avz --progress --partial -e "ssh -i /home/youngian/my_ssh_key" /mnt/storage/duplicity_backups backupuser@backup.dreamhost.com:.; doneThe problem with this is that if you want to halt the program, Ctrl-C only stops the current rsync process, and the loop helpfully starts another one immediately. Even worse, my connection kept breaking so hard that rsync would quit with the same "unkown" error code on connection problems as it did on a SIGINT, so I couldn't have my loop differentiate and break when needed. Here is my final script:
On a side note, duplicity is pretty neat. I only wish it would support resuming of interrupted backup sessions so that I didn't have to do this in two steps. My current backup workflow is
PASSPHRASE="backup" duplicity --encrypt-key 77XABAX7 /home/youngian --exclude "**/.VirtualBox" --exclude "**/.kde" --exclude /home/youngian/tmp/ --exclude /home/youngian/backup/ file:///mnt/storage/duplicity_backups/ --volsize 100...and then the above rsync script.
Posts
13 comments:
Thank you! Your post did save me from spending another day reading man pages. This does just what I was trying to do.
I've extended your script so that you don't need to edit each time + a few extras... and created a fork on gist.git hub.
See here...
http://gist.github.com/366269
Hope you find it usefull!!
thank you! you rule
Very neat... I am at =the stage you were, meaning my rsync script looks very much like your first shot and is not really dealing well with error and retries.
You deserve a kudo
thanks ;-))
Your post was really usefull for me!
I´ll show your script in my blog.
I hope you agree, if not please send me a message.
Thanks a lot!
(my blog: http://blog blog.aguinar.com)
So the problem with the above approach is that if you are paranoid like myself and use "bash -e" basically everywhere, setting the initial return value to "false" will cause your script to exit.
Instead I used your script as inspiration and did it this way instead:
trap "exit;" INT TERM EXIT
retries=7
i=0
while ! rsync SOURCE USER@HOST:/PATH && \
[ $i -le $retries ]
do
echo "rsync failed! Retrying..."
let i=$i+1
done
trap - INT TERM EXIT
...mind you I haven't implemented it yet but it seemed to work when I replaced the rsync command with "false." ;-)
Is it possible to add a wait of 30 or 60 seconds between connection attempts?
@andymat Sure! You can change the rsync line from `rsync -avz ...` to `rsync -avz ... || sleep 30s`.
somebody should tell rsync they need a "try forever" command line option...
Ian, thanks for this.
Question - after a successful transfer, the command doesn't work anymore until I restart my whole machine. I guess because the sucessful code is stored in memory.
I'm new to Unix, how do I reset it so I can issue the command again?
Thanks!
This script is pure gold, had to transfer one large file from Madagascar with a very unreliable network connection. This script worked like a charm, thankyou.
Thanks! Still relevant today.
Post a Comment