Instant update mirroring

Instant Update Mirroring

The future tier'ed CPAN mirroring will use File::Rsync::Mirror::Recent (also known as rrr, version 0.1.1 or later required). It is also available from git.

This setup is just in testing for now; don't try it unless you want to help make sure it works!

You can use this for the full CPAN; no need to separately use rrr for the PAUSE directories. When we move this to "full production" the changes from PAUSE will come through with an extra delay of just 10-30 seconds.

How to install

curl --compressed -LO http://xrl.us/cpanm
chmod +x ./cpanm
./cpanm JSON File::Rsync::Mirror::Recent

Also check that you have rsync installed; version 3.x or newer is highly recommended.

Run the client

Use rrr-client:

   mkdir -p /home/mirrors/CPAN
   rrr-client --source cpan-rsync.perl.org::CPAN/RECENT.recent --target /home/mirrors/CPAN

Client resources

It will take a while - up to several hours - to get synced up. During this time the system may use a good deal of CPU and momentarily up to 1GB of memory. After that it'll get the latest changes every 20 seconds with virtually no CPU usage and just using about 10MB of memory -- basically less resources than running rsync just a few times a day. And you get instant updates!

What to test

The goal of this is to find and fix any bugs in the rrr tools and make sure that all updates to CPAN comes through to your mirror the same as the traditional "full" rsync.

If you want to test if all updates from the master have been fetched, you can run

   rsync -avn cpan-rsync.perl.org::CPAN /home/mirrors/CPAN

There should only be a few changes - files that have changed in the last minute. If there are more, run rsync again without the -n parameter to update your mirror. Please report any discrepancies!

In production it'll likely be recommended to run a full rsync every few days in case rrr missed something; but for the testing please don't do that. Maybe it won't be necessary at all.

rsync masters

• Public, no authentication required: cpan-rsync.perl.org
• For "Tier 1" mirrors, cpan-rsync-master.perl.org (the "master mirror")

Right now the two hostnames actually point to the same box, but when we're happy with the performance and reliability of the rrr tool, the plan is to make cpan-rsync.perl.org point to a group of instantly updated mirrors -- the "Tier 1" mirrors. With some clever DNS, we can even have clients of the tier 1 mirrors pull appropriately from a mirror in their country or on their continent.

If you wish to be a tier 1 mirror, please email [email protected] for a username/password for the master mirror for testing.