Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terminate the scraper if a there is a code fault in one of the modules #442

Open
nwinklareth opened this issue Jun 6, 2014 · 0 comments

Comments

@nwinklareth
Copy link
Contributor

This morning during a run, when the CSV writer was executing a program exception was thrown. This was not caught by scraper code, it was caught by the greenlet code, which terminated the task. The terminated task was the backend task, which meant that all of the database requests were queued up, which balloned the memory usage of the process. This behaviour is wrong.

Modules in the scraper should terminate when code exceptions occur or when other drastic events occur. They should report the event as part of the termination process.

@nwinklareth nwinklareth changed the title Terminate the scrape if a there is a code fault in one of the modules Terminate the scraper if a there is a code fault in one of the modules Jun 6, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant