Back to Top

Wednesday, February 21, 2007

Using rsync on Windows

First of all, what is rsync?

It is a protocol and an implementation of it for bandwidth efficient file synchronization. In a nutshell it can synchronize two directories (one local and one remote) while making sure that only the minimal amount of data is transferred. It accomplishes this by breaking the files up in blocks and only transferring the blocks which are necessary (so if you appended - or preppended, or inserted in the midle - something to a file, only the new data will be transferred, not the whole file). You can find a more detailed description here, or you can read the rsync technical report and the original PhD thesis.

The problem (as with most great tools) is that there is no version for Windows. There are two Windows ports that I know of: DeltaCopy and cwRsync. They both are packages which bundle the Cygwin version of rsync with the needed DLLs (cygwin1.dll, and so on). Cygwin is an emulation layer for Windows which allows to create a Linux-e environment and run / compile many of the tools available for Linux without modification.

There are two things you should watch out for:

The first and most important: different versions of cygwin1.dll do not play well together. The symptoms include the rsync client entering in an infinite loop and consuming 100% of your processor and mysterious error messages which talk about shared section mismatches. If you encounter one of these, do a system wide search for cygwin1.dll and make sure that only one exists. Tricks like putting a cygwin1.dll with a different version in the directory of the executable and creating the .local file don't work either because different instances of the loaded DLL need to communicate between eachother and this is impossible if there is a version mismatch. As of my limited experience the cygwin1.dll included with the cwRsync distribution worked fine with other Cygwin programs, but there might be some hidden incompatibilities.

The second gotcha is that you have to refer to your directories in the form of /cygdrive//path/with/slashes. So for example E:\stuff\to\sync becomes /cygdrive/e/stuff/to/sync. Also I found that the most reliable way to run the rsync client is to use the current directory as source / target (eg. .). In this situation the pushd / popd batch commands come in handy (and remember, opposed to cd, they can also change the current drive).

Also, I use rsyncd / rsync in a client-server (also referred to as daemon) mode, not over SSH (because I use it inside a trusted network). In this case you must specify two colons when writing the network source, like this: 192.168.0.1::tosync.

For a more detailed tutorial check out this tutorial on rsync which also applies to cwRsync almost entirely with the caveats mentioned earlier.

And a final note about the modus operandi of rsync: while the synchronization is in progress a temporary file is created in the same directory. After the synchronization of that given file is done, the old file is overwritten with the new version. I mention this because it may be the case that you wish to synchronize things which are currently locked and I don't really know how well rsync handles this situation (given that under Linux, its original platform, it is possible to delete locked files as long as you have the rights to do so, without affecting programs which have a handle already open to it). So be cautious!

Update: if you are using Windows Server 2008 and you are getting error messages when trying to use rsync (something along the lines of failed to create shared section), make sure that you've updated to the latest version of it, which currently is 2.1.5. Also, it may refuse to install on Windows Server 2008. As a workaround you could install it on an older version of Windows (up to Server 2003) and copy over the files.

11 comments:

  1. cd /d d:\newdir
    allows you to change drive and directory.

    ReplyDelete
  2. Anonymous3:06 PM

    Thanks for the nice post!

    ReplyDelete
  3. Anonymous6:30 PM

    Hi there,
    I like your post, and I hope that you can help me.

    I would like to synchronize two server (trusted domain)
    10.62.50.138 has directory NASH and 10.62.50.135 has directory NASH (without data)

    What command should i use to synchronize these two servers?

    I tried :

    rsync -avr 10.62.50.138/nash /10.62.50.135/nash


    and got following:

    C:\Program Files\cwRsync\bin>rsync -avr 10.62.50.138/nash /10.62.50.135/nash
    building file list ... rsync: link_stat "/cygdrive/c/Program Files/cwRsync/bin/1
    0.62.50.138/nash" failed: No such file or directory (2)
    done
    rsync: push_dir#3 "/10.62.50.135" failed: No such file or directory (2)
    rsync error: errors selecting input/output files, dirs (code 3) at main.c(565) [
    receiver=2.6.9]
    rsync: connection unexpectedly closed (8 bytes received so far) [sender]
    rsync error: error in rsync protocol data stream (code 12) at io.c(453) [sender=
    2.6.9]

    Thanks in advance,

    Nash

    ReplyDelete
  4. Two things:

    - under cygwin you have to reference a directory as /cygdrive/drive/directory/with/slashes/not/back/slashes. For example c:\foo\bar would be written as /cygdrive/foo/bar

    - I don't know if you can actually synchronize two remove server (from what I understand this is what you're trying to do). Instead you should go to one of the servers and launch rsync locally on it like this:

    rsync //10.62.50.138/nash /cygdrive/c/foo/nash

    (supposing that you have the other directory accessible through Samba)

    ReplyDelete
  5. A comment on the last section, talking about temporary files. You can prevent rsync from using a temporary file by specifying the --inplace parameter, which forces rsync to update files without making a work-copy first; this avoids some of the issues.

    ReplyDelete
  6. Anonymous10:25 PM

    I do hav a doubt!!!
    when we installed cygwin, is there any conflict with other program which was already running in that server, b'coz cygwin installed lot of packages!!!!!
    please let me know ASAP

    ReplyDelete
  7. @cherry: please give more details about what you mean by "conflict with other programs". If you already had cygwin installed on the server, you might have problems with different versions of cygwin1.dll. If this is the case, you should try the advice given in the paragraph which begins with "The first and most important: different versions of cygwin1.dll do not play well together"...

    In the case that doesn't solve the problem, please get back with a more detailed description of the problem.

    ReplyDelete
  8. Does this work with Windows 7?

    ReplyDelete
  9. @Marcus: according to their website cwRsync supports Windows 7, both as client and as server.

    ReplyDelete
  10. Can we take backup using rsync for a remote server that has ONLY static IP.

    e.g: I have Server "A" - windows server 2008 R2. Server "B" is windows server 2003 R2 (with static IP)

    Can I use rsync to sych files, both the servers are not in same network.,

    ReplyDelete
    Replies
    1. As long as the servers can connect to each-other trough TCP, rsync will work (so you need check firewall, NAT and any other device which is in the middle).

      Delete