Handle RuntimeError #150
Handle RuntimeError #150
Conversation
Some times when warcprox runs for several days under load it freezes
and the last error in the log is:
```
WARNING:warcprox.warcproxy.WarcProxy:exception processing request
<socket.socket fd=53, family=AddressFamily.AF_INET,
type=SocketKind.SOCK_STREAM, proto=0, laddr=('207.241.225.241', 8003),
raddr=('207.241.225.241', 40738)> from ('207.241.225.241', 40738)
Traceback (most recent call last):
File "/usr/lib/python3.7/socketserver.py", line 316, in
_handle_request_noblock
self.process_request(request, client_address)
File "/opt/spn2/lib/python3.7/site-packages/warcprox/mitmproxy.py",
line 641, in process_request
self.process_request_thread, request, client_address)
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 172, in
submit
self._adjust_thread_count()
File "/usr/lib/python3.7/concurrent/futures/thread.py", line 193, in
_adjust_thread_count
t.start()
File "/usr/lib/python3.7/threading.py", line 852, in start
_start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
```
The process seems to run but it doesn't respond to any connection, not
even `status` requests.
We handle this exception and allow it to continue operation.
|
I tried to replicate this error in my dev environment but couldn't. In any case, I think that it would be good to handle this exception and continue operation. |
| # managed to finish before future.add_done_callback() ran | ||
| self.active_requests.pop(request, None) | ||
| except RuntimeError as exc: | ||
| self.logger.error("Error processing request %s", str(exc)) |
There was a problem hiding this comment.
you shouldn't call str(exc) that happens automatically because of %s if the error is logged and doesn't make that unneeded computation if it is filtered.
Additionally you could instead do, self.logger.error("Error processing request", exc_info=exc) than logging will add information about the exception by itself.
Before py3.5 sys.exc_info() will be used as bool(exc) is true
and after exc will be used directly.
https://docs.python.org/3/library/logging.html#logging.Logger.debug
Some times when warcprox runs for several days under load it freezes
and the last error in the log is:
The process seems to run but it doesn't respond to any connection, not even
statusrequests.We handle this exception and allow it to continue operation.