rsync under cygwin too slow

[Vinod Gupta]

Further to my earlier thread (Re: ls -lR too slow) which seems to have
died, I tried backing up my laptop to smbshared folder on desktop using
rsync. Both machines are running Vista+SP1 and on gigabit LAN sitting
next to each other. The following statistics displayed by rsync is
completely misleading:

rsync -rtuplg --stats --verbose --progress --modify-window=2
/cygdrive/c/localfolder /cygdrive/z/backups/
...
...
Number of files: 24518
Number of files transferred: 0
Total file size: 5805610746 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 497277
File list generation time: 188.978 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 644377
Total bytes received: 147104

sent 644377 bytes received 147104 bytes 454.22 bytes/sec
total size is 5805610746 speedup is 7335.12


The local and remote folders were already synchronized, that is why 0
files were transferred. File list generation did take only 3 minutes but
the whole process took more than 1.5 hours to finish and consumed a
whopping 1.9 GB of downloaded bytes and 0.4 GB of uploaded bytes as per
network counters. For what? Zero files, Zero bytes of file data.

As far as I can see, rsync under cygwin is also too slow to be useful.

Vinod

[Alfred von Campe]

rsync -rtuplg --stats --verbose --progress --modify-window=2 /
cygdrive/c/localfolder /cygdrive/z/backups/
...
...
The local and remote folders were already synchronized, that is why
0 files were transferred. File list generation did take only 3
minutes but the whole process took more than 1.5 hours to finish
and consumed a whopping 1.9 GB of downloaded bytes and 0.4 GB of
uploaded bytes as per network counters. For what? Zero files, Zero
bytes of file data.

I'm pretty sure this is because rsync has to copy all the files
locally to compare them to see if they need to be copied. Try
running the rsync as a service on the system that hosts the Z: drive
(or run rsync over ssh to the system that hosts the Z drive) and you
will see large performance improvement.

As far as I can see, rsync under cygwin is also too slow to be useful.

This has nothing to do with Cygwin or rsync. If you ran ROBOCOPY
under Windows using the same setup you would see the same performance
penalty.

Alfred

[KHMan]

Post by Vinod Gupta
rsync -rtuplg --stats --verbose --progress --modify-window=2
/cygdrive/c/localfolder /cygdrive/z/backups/
...
...
The local and remote folders were already synchronized, that is why 0
files were transferred. File list generation did take only 3 minutes
but the whole process took more than 1.5 hours to finish and consumed
a whopping 1.9 GB of downloaded bytes and 0.4 GB of uploaded bytes as
per network counters. For what? Zero files, Zero bytes of file data.

I'm pretty sure this is because rsync has to copy all the files locally
to compare them to see if they need to be copied. Try running the rsync
as a service on the system that hosts the Z: drive (or run rsync over
ssh to the system that hosts the Z drive) and you will see large
performance improvement.

Post by Vinod Gupta
As far as I can see, rsync under cygwin is also too slow to be useful.

This has nothing to do with Cygwin or rsync. If you ran ROBOCOPY under
Windows using the same setup you would see the same performance penalty.

Agree, I mirror stuff locally and to external drives using
Cygwin's rsync and sync'ing identical trees is never that slow.

Could it be a security program? For example, AVG 8 Free checks for
malware by default, IIRC, and the speed impact is really
noticeable, even on a basic dual core. I turned off that feature
after less than 5 minutes.

[KHMan]

Post by Vinod Gupta
rsync -rtuplg --stats --verbose --progress --modify-window=2
/cygdrive/c/localfolder /cygdrive/z/backups/
...
...
The local and remote folders were already synchronized, that is why 0
files were transferred. File list generation did take only 3 minutes
but the whole process took more than 1.5 hours to finish and consumed
a whopping 1.9 GB of downloaded bytes and 0.4 GB of uploaded bytes as
per network counters. For what? Zero files, Zero bytes of file data.

I'm pretty sure this is because rsync has to copy all the files locally
to compare them to see if they need to be copied. Try running the rsync
as a service on the system that hosts the Z: drive (or run rsync over
ssh to the system that hosts the Z drive) and you will see large
performance improvement.

Looks to me there is no --checksum option, so rsync skips based on
size and modtime. I do something similar to Vinod. Here is an
extra data point to compare against (y and u are on separate
drives on same machine):

$ time rsync -a --progress --stats --delete /cygdrive/y/Installs/
/cygdrive/u/Installs/
sending incremental file list

Number of files: 284
Number of files transferred: 0
Total file size: 2284961277 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 8583
File list generation time: 0.030 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 8606
Total bytes received: 22

sent 8606 bytes received 22 bytes 17256.00 bytes/sec
total size is 2284961277 speedup is 264830.93

real 0m0.678s
user 0m0.155s
sys 0m0.217s

With --checksum:

$ time rsync -ac --progress --stats --delete /cygdrive/y/Installs/
/cygdrive/u/Installs/
sending incremental file list

Number of files: 284
Number of files transferred: 0
Total file size: 2284961277 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 12951
File list generation time: 5.476 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 12974
Total bytes received: 22

sent 12974 bytes received 22 bytes 234.16 bytes/sec
total size is 2284961277 speedup is 175820.35

real 0m55.964s
user 0m23.388s
sys 0m7.202s

[snip snip]

[Sam Liddicott]

Its because that's not how rsync should be used over the network.

There should be a copy of rsync running on the remote machine that reads the remote files and sends only the checksums.

The setup shown here (as guessed) has the loca rsync read the remote files over the lan in order to get the checksums.

Disabling checksums and using timestamps can help but is much less certain.

Better to use rsyncover ssh.
Sam



-----Original Message-----
From: KHMan <***@gmail.com>
Sent: Tuesday, March 24, 2009 3:16 AM
To: ***@cygwin.com
Subject: Re: rsync under cygwin too slow

Post by Vinod Gupta
rsync -rtuplg --stats --verbose --progress --modify-window=2
/cygdrive/c/localfolder /cygdrive/z/backups/
...
...
The local and remote folders were already synchronized, that is why 0
files were transferred. File list generation did take only 3 minutes
but the whole process took more than 1.5 hours to finish and consumed
a whopping 1.9 GB of downloaded bytes and 0.4 GB of uploaded bytes as
per network counters. For what? Zero files, Zero bytes of file data.

I'm pretty sure this is because rsync has to copy all the files locally
to compare them to see if they need to be copied. Try running the rsync
as a service on the system that hosts the Z: drive (or run rsync over
ssh to the system that hosts the Z drive) and you will see large
performance improvement.

Looks to me there is no --checksum option, so rsync skips based on
size and modtime. I do something similar to Vinod. Here is an
extra data point to compare against (y and u are on separate
drives on same machine):

$ time rsync -a --progress --stats --delete /cygdrive/y/Installs/
/cygdrive/u/Installs/
sending incremental file list

Number of files: 284
Number of files transferred: 0
Total file size: 2284961277 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 8583
File list generation time: 0.030 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 8606
Total bytes received: 22

sent 8606 bytes received 22 bytes 17256.00 bytes/sec
total size is 2284961277 speedup is 264830.93

real 0m0.678s
user 0m0.155s
sys 0m0.217s

With --checksum:

$ time rsync -ac --progress --stats --delete /cygdrive/y/Installs/
/cygdrive/u/Installs/
sending incremental file list

Number of files: 284
Number of files transferred: 0
Total file size: 2284961277 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 12951
File list

[The entire original message is not included]