Setting up a mirroring cluster

General information

The Github API limit (currently, 5000 reqs/sec) makes it impossible to retrieve all data linked from events on a single node. For that, GHTorrent was designed to work on multiple phases in a distributed fashion. Depending on the data you want to collect, a cluster setup may be necessary.

A full GHTorrent cluster consists of the following types of nodes:

• Event retrieval nodes: Nodes that query the public Github event API for new events. More than one instances are required to both ensure that no events are lost due to spikes in event generation and that machine or network malfunctions the event collection machine do not affect the service.
• Linked data retrieval nodes: Retrieval of data linked by events is where the Github API is imposing the most significant restrictions.
• MongoDB shards: A MongoDB installation can be sharded (have the data spread on multiple nodes) on a per collection basis. Sharing MongoDB helps with both distributing the storage requirements and faster querying. Sharding is transparent to the application and therefore no modification is required to GHTorrent to work with a shared MongoDB. See more on MongoDB sharding here
• RabbitMQ active-active mirrors: RabbitMQ can work in cluster mode for high availability.

Recommended setup

To setup a GHTorrent cluster, at least 2 nodes are required. However, the more the available nodes, the more data can be collected and the more resilient to external errors or mirroring script bugs the cluster will be. The following sections assume a Debian-based operating system.

Setting up software dependencies

To prepare MongoDB:

$ mongo admin
> db.addUser('github', 'github')
> use github
> db.addUser('github', 'github')

To prepare RabbitMQ:

$ rabbitmqctl add_user github
$ rabbitmqctl set_permissions -p / github ".*" ".*" ".*"

# The following will enable the RabbitMQ web admin for the github user
# Not necessary to have, but good to debug and diagnose problems
$ rabbitmq-plugins enable rabbitmq_management
$ rabbitmqctl set_user_tags github administrator

Setting up a GHTorrent node