You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This morning during a run, when the CSV writer was executing a program exception was thrown. This was not caught by scraper code, it was caught by the greenlet code, which terminated the task. The terminated task was the backend task, which meant that all of the database requests were queued up, which balloned the memory usage of the process. This behaviour is wrong.
Modules in the scraper should terminate when code exceptions occur or when other drastic events occur. They should report the event as part of the termination process.
The text was updated successfully, but these errors were encountered:
nwinklareth
changed the title
Terminate the scrape if a there is a code fault in one of the modules
Terminate the scraper if a there is a code fault in one of the modules
Jun 6, 2014
This morning during a run, when the CSV writer was executing a program exception was thrown. This was not caught by scraper code, it was caught by the greenlet code, which terminated the task. The terminated task was the backend task, which meant that all of the database requests were queued up, which balloned the memory usage of the process. This behaviour is wrong.
Modules in the scraper should terminate when code exceptions occur or when other drastic events occur. They should report the event as part of the termination process.
The text was updated successfully, but these errors were encountered: