March 20, 2009

Rsync and retrying until we get it right

Ok, this isn't all that special, but I scoured the first two or three pages of Google results and didn't come up with anything that solved my problem. So here it is, Internet - may the next person be luckier than me and not have to read any man pages.

Rsync is a cool utility, especially when I'm trying to plonk my 10Gb backup onto Dreamhost's flaky backup server. But I wish I could make it retry when things go south. There are various threads on doing this, but it would seem it's not built into rsync itself.

The obvious solution is to check the return value, and if rsync returns anything but success, run it again. Here was my first try:

while [ $? -ne 0 ]; do rsync -avz --progress --partial -e "ssh -i /home/youngian/my_ssh_key" /mnt/storage/duplicity_backups backupuser@backup.dreamhost.com:.; done
The problem with this is that if you want to halt the program, Ctrl-C only stops the current rsync process, and the loop helpfully starts another one immediately. Even worse, my connection kept breaking so hard that rsync would quit with the same "unkown" error code on connection problems as it did on a SIGINT, so I couldn't have my loop differentiate and break when needed. Here is my final script:



On a side note, duplicity is pretty neat. I only wish it would support resuming of interrupted backup sessions so that I didn't have to do this in two steps. My current backup workflow is

PASSPHRASE="backup" duplicity --encrypt-key 77XABAX7 /home/youngian --exclude "**/.VirtualBox" --exclude "**/.kde" --exclude /home/youngian/tmp/ --exclude /home/youngian/backup/ file:///mnt/storage/duplicity_backups/ --volsize 100

...and then the above rsync script.
at

13 comments:

  1. UnknownJune 23, 2009 at 6:05 AM

    Thank you! Your post did save me from spending another day reading man pages. This does just what I was trying to do.

    ReplyDelete
  2. Jason MartinApril 14, 2010 at 4:58 PM

    I've extended your script so that you don't need to edit each time + a few extras... and created a fork on gist.git hub.
    See here...

    http://gist.github.com/366269

    Hope you find it usefull!!

    ReplyDelete
  3. RobJune 8, 2010 at 5:02 AM

    thank you! you rule

    ReplyDelete
  4. PhilippeMarch 19, 2011 at 7:42 AM

    Very neat... I am at =the stage you were, meaning my rsync script looks very much like your first shot and is not really dealing well with error and retries.
    You deserve a kudo

    ReplyDelete
  5. 735984February 15, 2012 at 6:50 AM

    thanks ;-))

    ReplyDelete
  6. aguinarFebruary 16, 2012 at 11:41 AM

    Your post was really usefull for me!
    I´ll show your script in my blog.
    I hope you agree, if not please send me a message.

    Thanks a lot!
    (my blog: http://blog blog.aguinar.com)

    ReplyDelete
  7. UnknownJune 14, 2012 at 11:37 AM

    So the problem with the above approach is that if you are paranoid like myself and use "bash -e" basically everywhere, setting the initial return value to "false" will cause your script to exit.

    Instead I used your script as inspiration and did it this way instead:

    trap "exit;" INT TERM EXIT

    retries=7

    i=0

    while ! rsync SOURCE USER@HOST:/PATH && \
    [ $i -le $retries ]
    do

    echo "rsync failed! Retrying..."
    let i=$i+1

    done

    trap - INT TERM EXIT


    ...mind you I haven't implemented it yet but it seemed to work when I replaced the rsync command with "false." ;-)

    ReplyDelete
  8. andymatOctober 17, 2012 at 3:49 PM

    Is it possible to add a wait of 30 or 60 seconds between connection attempts?

    ReplyDelete
  9. IanOctober 18, 2012 at 6:00 PM

    @andymat Sure! You can change the rsync line from `rsync -avz ...` to `rsync -avz ... || sleep 30s`.

    ReplyDelete
  10. Roger PackJune 9, 2014 at 11:34 AM

    somebody should tell rsync they need a "try forever" command line option...

    ReplyDelete
  11. David MackenzieOctober 18, 2014 at 12:18 AM

    Ian, thanks for this.
    Question - after a successful transfer, the command doesn't work anymore until I restart my whole machine. I guess because the sucessful code is stored in memory.

    I'm new to Unix, how do I reset it so I can issue the command again?

    Thanks!

    ReplyDelete
  12. MattJuly 13, 2017 at 9:51 AM

    This script is pure gold, had to transfer one large file from Madagascar with a very unreliable network connection. This script worked like a charm, thankyou.

    ReplyDelete
  13. JamesHSeptember 9, 2020 at 9:13 PM

    Thanks! Still relevant today.

    ReplyDelete

Note: Only a member of this blog may post a comment.