Discussion:
Mirror, Mirroring, Download, Downloading Cygwin Release Using rsync
L Anderson
2005-04-20 09:19:44 UTC
Permalink
From time to time, as even now, someone asks a question on how to
download Cygwin for installing, locally, on any number of machines,
burning a CD/DVD, etc. The canonical answer seems to be to use Cygwin's
Setup. If "Setup" doesn't run on your system of choice (e.g. FreeBSD,
GNU/Linux...), then how? Although "Setup" does run on my windows
machine, I don't normally use it to download release changes. Instead,
I use rsync, which should work as well on *nix type systems as on a M$
one. So here's how I do it and what works for me--see the bash script
below.

I use rsync to maintain a local Cygwin release mirror so I can use Setup
to locally install anything from Cygwin release on a number of PCs on my
network. I have been doing this for several years with virtually no
problems--except for the occasional rsync mirror that comes or goes.

The nice thing about rsync is that it downloads only what has changed
since the last rsync and with the "--delete" option, it deletes
everything on the local mirror that's not on the official Cygwin mirror
being used.(NB: Not so with wget. It just lets things pile up.) Also,
rsync is very robust in that, should the download be interrupted, it can
be restarted and it picks up, basically, from where it left off.
Moreover, you can still use Setup on an M$ machine to download and
install changes using the same local directory structure, e.g.
/cygdrive/x/cygwin/release. However, using the "--delete" option, the
directories left by Setup will be removed the next time rsync is run but
everything will be current.

Be advised that disk space may now be an issue. Several years ago, when
I started using rsync to mirror Cygwin release, it did so in about
800MB. Now it takes about 2.4GB--somebody has been busy--thanks!
However, frequent rsyncs will only need to download a fraction of that.

Below is the bash script I start manually on my Win98SE system. It's
brutish, lacks sophistication, but works for me. Because my Cygwin
install is Unix, I use Windows' Wordpad (write.exe) to read the log
files (hence, the ".wri" extension). Wordpad can handle both the file
size and line ends properly whereas notepad can't. The script first
runs rsync to download any changes to Setup.exe and Setup.ini, placing
them in /cygdrive/n/pub/Cygwin from where I can run Setup. The script
then runs rsync to update /cygdrive/n/pub/Cygwin/release with the latest
changes. If I first want to see what the new changes are (more or less)
without downloading, I do a "dry run" (rsync -n option). After
rsyncing, I can do a local install with Setup from any machine on my
network.

I just updated the script with the latest rsync mirrors found on the
Cygwin site. Normally, I only use mirror 1 or 2 in the list. If there
were more North American mirrors, I'd try my hand at selecting the one
to use randomly, rather than by menu--share the load.

I hope someone finds this information useful and that it's not redundant.


Regards,

Lowell Anderson



-----cut here-----
#!/bin/bash

# Select the rsync mirror to use.

while [ 1 ] ; do
echo ""
echo "Select the rsync mirror to use or exit:"
echo "1) rsync://mirrors.xmission.com/cygwin/ --Utah"
echo "2) rsync://mirrors.kernel.org/sources.redhat.com/cygwin/
--Palo Alto"
echo "3) rsync://ftp.esat.net/mirrors/sources.redhat.com/pub/cygwin/
--Ireland"
echo "4) rsync://ftp.gwdg.de/pub/linux/sources.redhat.com/cygwin/
--Germany"
echo "5) rsync://ftp.inf.tu-dresden.de/cygwin/ --Germany"
echo "6) rsync://ftp.kaist.ac.kr/cygwin/ --Korea"
echo "7) rsync://mirror.averse.net/cygwin/ --Singapore"
echo ""
echo "0) exit program"
echo ""
echo -n "Enter selection 0-7:"
read selection
echo ""

case $selection in
1 ) rsyncsite="rsync://mirrors.xmission.com/cygwin/"
break;;
2 ) rsyncsite="rsync://mirrors.kernel.org/sources.redhat.com/cygwin/"
break;;
3 )
rsyncsite="rsync://ftp.esat.net/mirrors/sources.redhat.com/pub/cygwin/"
break;;
4 )
rsyncsite="rsync://ftp.gwdg.de/pub/linux/sources.redhat.com/cygwin/"
break;;
5 ) rsyncsite="rsync://ftp.inf.tu-dresden.de/cygwin/"
break;;
6 ) rsyncsite="rsync://ftp.kaist.ac.kr/cygwin/"
break;;
7 ) rsyncsite="rsync://mirror.averse.net/cygwin/"
break;;
0 ) echo "OK! Exiting";
exit 1;;
* ) echo "Please enter one of 0 to 7"
esac

done

echo -n "Do you want a dry run? (yes/no) 'var1': "

read var1

if [ "$var1" = "yes" ]

then

rsync -vaunt --progress --stats --delete $rsyncsite/set*
/cygdrive/n/pub/cygwin > /cygdrive/n/pub/CygwinMgmt/drsetlog.wri

rsync -vaunrt --progress --stats --delete $rsyncsite/release
/cygdrive/n/pub/cygwin > /cygdrive/n/pub/CygwinMgmt/drreleaselog.wri

fi


if [ "$var1" = "no" ]

then

rsync -vaut --progress --stats --delete $rsyncsite/set*
/cygdrive/n/pub/cygwin > /cygdrive/n/pub/CygwinMgmt/setlog.wri

rsync -vaurt --progress --stats --delete $rsyncsite/release
/cygdrive/n/pub/cygwin > /cygdrive/n/pub/CygwinMgmt/releaselog.wri

else

echo "Just a dry run."

fi

echo "var1 = $var1"

echo

exit 0
Brian Dessent
2005-04-20 09:46:06 UTC
Permalink
Post by L Anderson
I use rsync to maintain a local Cygwin release mirror so I can use Setup
to locally install anything from Cygwin release on a number of PCs on my
network.
I do too but without the script part. I just have a line in my crontab:

00 10,22 * * * rsync -rlt --exclude=mail-archives
rsync://mirrors.kernel.org/sources.redhat.com/cygwin/ /d/cygwin-mirror/
Post by L Anderson
/dev/null
I don't really see why you need to do it in two steps, just use
--exclude to get rid of stuff you don't want and mirror the base
directory.

BTW, you could get rid of that .wri extension ugliness by mounting the
logfile directory textmode, and then name the output .txt. Or get a
better text editor - I recommend Metapad for free, UltraEdit for $.
(But I realize you're probably happy with what you have and don't need
someone telling you to change.)
Post by L Anderson
The nice thing about rsync is that it downloads only what has changed
since the last rsync and with the "--delete" option, it deletes
everything on the local mirror that's not on the official Cygwin mirror
being used.
That's more of a feature to me than a bug, as I like to have older
packages. Although with Peter's "Cygwin Time Machine" site that becomes
less of an issue.
Post by L Anderson
800MB. Now it takes about 2.4GB--somebody has been busy--thanks!
However, frequent rsyncs will only need to download a fraction of that.
You can cut that down significantly by using --exclude to ignore the
source packages. Of course, that means you can't install source
packages.
Post by L Anderson
were more North American mirrors, I'd try my hand at selecting the one
to use randomly, rather than by menu--share the load.
Heh. Well, that's a good sentiment. I use the kernel.org mirror, and
given the amount of traffic they serve I have good faith that several MB
of cygwin packages every week or two is not even a drop in their
bucket. It looks like they've taken away their neat little BW meter
from the home page but I seem to recall it was normally pushing out a
constant 100 to 200 MBit/s, and a whole lot more in times following
kernel releases. They've got big pipes. :)

Brian
L Anderson
2005-04-21 09:15:19 UTC
Permalink
Post by Brian Dessent
Post by L Anderson
I use rsync to maintain a local Cygwin release mirror so I can use Setup
to locally install anything from Cygwin release on a number of PCs on my
network.
00 10,22 * * * rsync -rlt --exclude=mail-archives
rsync://mirrors.kernel.org/sources.redhat.com/cygwin/ /d/cygwin-mirror/
Post by L Anderson
/dev/null
I don't really see why you need to do it in two steps, just use
--exclude to get rid of stuff you don't want and mirror the base
directory.
Initially I used --exclude but decided it was safer to ask for what I
want rather than for everything but what I don't want. If a subdirectory
that I don't want is added to the mirror, it gets downloaded until I
discover it and change the --exclude to exclude it. The mirror's
directory structure is more stable now than when I stared so it might
not be such a problem, however, .....
Post by Brian Dessent
You can cut that down significantly by using --exclude to ignore the
source packages. Of course, that means you can't install source
packages.
I know, but then we wouldn't have all that nifty source code easily at
hand to drive us crazy trying to figure out what it does :-) so that
someday we might be able to contribute.

You have given me a couple of things to think about, especially using
cron, thanks.

Regards,

Lowell Anderson
Brian Dessent
2005-04-21 10:05:35 UTC
Permalink
Post by L Anderson
Initially I used --exclude but decided it was safer to ask for what I
want rather than for everything but what I don't want. If a subdirectory
that I don't want is added to the mirror, it gets downloaded until I
discover it and change the --exclude to exclude it. The mirror's
directory structure is more stable now than when I stared so it might
not be such a problem, however, .....
You can do that too in a single pass:

mirror=mirrors.kernel.org
dir=sources.redhat.com/cygwin
dest=/d/cygwin-mirror
rsync -rlt $mirror::$dir/setup*\ $dir/release/ $dest

(The rsync man page covers specifying multiple sources in the section
'ADVANCED USAGE'.)
Post by L Anderson
I know, but then we wouldn't have all that nifty source code easily at
hand to drive us crazy trying to figure out what it does :-) so that
someday we might be able to contribute.
Of course. I find having the source packages very handy. But not
everyone has that need. For anyone curious, the 2.7GB current total
breaks down to 1.2GB of binary packages and 1.5GB of source packages.
You could fit a binary-only mirror on two CDRs, for example. Though
these days DVD-Rs are so common that it may not be a concern.

Brian
L Anderson
2005-04-21 16:53:17 UTC
Permalink
Post by Brian Dessent
Post by L Anderson
Initially I used --exclude but decided it was safer to ask for what I
want rather than for everything but what I don't want. If a subdirectory
that I don't want is added to the mirror, it gets downloaded until I
discover it and change the --exclude to exclude it. The mirror's
directory structure is more stable now than when I stared so it might
not be such a problem, however, .....
mirror=mirrors.kernel.org
dir=sources.redhat.com/cygwin
dest=/d/cygwin-mirror
rsync -rlt $mirror::$dir/setup*\ $dir/release/ $dest
In deference to the Bellman: "I like it, I like it, I like it!!!" It's
revisit my old script time!
Post by Brian Dessent
(The rsync man page covers specifying multiple sources in the section
'ADVANCED USAGE'.)
Thanks for this pointer and the help!

Regards,

Lowell

Loading...