Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR replaces the previous scraper to address the change in the london mayor/assembly website mysociety/theyworkforyou#1687
This is also adding some config files for docker and code linters. Linters are restricted to the
london-mayors-question
folder for the moment.The scraper talks to the london site in two places:
Because we have no way of knowing which questions have answers, all questions without answers need to be re-queried for an update.
The command to do this looks something like this:
And a version of this has replaced the commented out lines in
updatedaterange-parse
.It stores intermediate files in a
json_cache
directory. A initial populate will need to be done to catch up:There have been some updates to the overall requirements.txt - which hopefully shouldn't cause wider problems.
Import running for all info since 2020-12-20 seems to work fine in TWFY: