Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add archiver #1

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Add archiver #1

wants to merge 2 commits into from

Conversation

octopusinvitro
Copy link

@octopusinvitro octopusinvitro commented Nov 16, 2016

What does this do?

Uses scraped-page-archive to archive all pages scraped.

Why is this needed?

Elections happened recently (2016-11-20), and it's likely that the data on the official site will disappear, meaning any data we're not already picking up will be lost. Archiving it now gives us the chance to go back and re-scrape later even if it disappears.

Relevant Issue(s):

everypolitician/everypolitician-data#20544

Checklists:

Scraper page

Scraper is not under EP-scrapers account on morph.

Add archiving

The scraper was run locally to archive the current version of the page.

@davewhiteland
Copy link

davewhiteland commented Nov 18, 2016

After temperamental timeouts, the scraper finally ran to completion so archived all visited pages just now -- looks like it's captured all of them pre-election, phew!
https://github.com/everypolitician-scrapers/san_marino_council/tree/scraped-pages-archive

@tmtmtmtm
Copy link
Contributor

tmtmtmtm commented Jan 3, 2017

I've rebased this over master, but can't merge yet as the scraper is currently failing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants