Create the virtualenv

$ python -m venv scrapy-mysql
$ cd scrapy-mysql
$ Scripts\activate.bat

Install dependencies

$ pip install -r requirements.txt
$ pip list

Pre-req & Warnings

• Run database.sql first to create database first.
• Edit CONNECTION_STRING in scrapy_spider/settings.py before running spider.

Run the app

$ scrapy crawl jobs_spider

pep8

$ pep8 --first --show-source ./scrapy_spider

autopep8

$ autopep8 --in-place --aggressive --aggressive scrapy_env.py