##Setup

$ sudo apt-get install python-pip python-virtualenv
$ git clone git://github.com/techism/crawler.git
$ cd crawler
$ virtualenv --no-site-packages venv 
$ source venv/bin/activate
(venv)$ pip install -r requirements.txt

Create aggregator/config.yaml