You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, every thread has its own tor instance. Some instances are not working well, so they are changed until one is found. Usually, if request pool is not large (<10k urls), few instances do not find good tor route in time but will take tasks from the queue.
Currently, on the end of processing, we are killing processes with good tor routes (because there is nothing to do for them) and leave bad instances (because they are struggling to find route while they have taken the task from queue). We should probably redesign the code, to separate tor pools from task queue, so that we always pritoritize processing with good instances over processing with bad tor instances
The text was updated successfully, but these errors were encountered:
Currently, every thread has its own tor instance. Some instances are not working well, so they are changed until one is found. Usually, if request pool is not large (<10k urls), few instances do not find good tor route in time but will take tasks from the queue.
Currently, on the end of processing, we are killing processes with good tor routes (because there is nothing to do for them) and leave bad instances (because they are struggling to find route while they have taken the task from queue). We should probably redesign the code, to separate tor pools from task queue, so that we always pritoritize processing with good instances over processing with bad tor instances
The text was updated successfully, but these errors were encountered: