[5.0] don't run EOS VM OC's monitor compile task callback when socket being dtored

[spoonincode]

This may resolve #1823 & #1794 based on core dump analysis of an oc-monitor crash.

A single OC monitor process is created at launch of a parent process which plans to use OC (nodeos, a unit test application, etc). Critically, a single io_context is used in this single monitor process. As code_caches are created on the parent process, compile_monitor_sessions are created in the monitor process for each of those code_cache instances.

When a code_cache needs to compile a contract, it sends a message to its out of process compile_monitor_session. This will then call off to another process -- the OC trampoline -- but the most important actions here to note is that the compile_monitor_session notes in its current_compiles a socket to receive a reply on from the trampoline,

leap/libraries/chain/webassembly/runtimes/eos-vm-oc/compile_monitor.cpp

Line 110 in 81a9d5c

current_compiles.emplace_front(code_id, std::move(response_socket));

and then goes off to do an async_wait() on that socket,

leap/libraries/chain/webassembly/runtimes/eos-vm-oc/compile_monitor.cpp

Line 116 in 81a9d5c

socket.async_wait(local::datagram_protocol::socket::wait_read, [this, current_compile_it](auto ec) {

Consider that current_compiles holds some outstanding sockets because compiles are ongoing. Now consider that code_cache is destroyed. When that happens the read_message_from_nodeos() (which is a misnomer: it's really read_message_from_codecache) will indicate a signal,

leap/libraries/chain/webassembly/runtimes/eos-vm-oc/compile_monitor.cpp

Lines 57 to 61 in 81a9d5c

	void read_message_from_nodeos() {
	_nodeos_instance_socket.async_wait(local::datagram_protocol::socket::wait_read, [this](auto ec) {
	if(ec) {
	connection_dead_signal();
	return;

this signal is an indication that the compile_monitor_session can be destroyed (since its code_cache pairing has been destroyed). So.. it's destroyed,

leap/libraries/chain/webassembly/runtimes/eos-vm-oc/compile_monitor.cpp

Lines 211 to 215 in 81a9d5c

	_compile_sessions.front().connection_dead_signal.connect([&, it = _compile_sessions.begin()]() {
	ctx.post([&]() {
	_compile_sessions.erase(it);
	});
	});

As compile_monitor_session is destroyed its current_compiles is first to be dtored. As part of this destruction the local::datagram_protocol::socket will be destroyed ergo it will be cancel()ed egro it will make a callback to the async_wait() with boost::asio::error::operation_aborted. That's not strictly a problem, but things fall apart quickly here because the async_wait() handler then accesses an iterator to the currently-being-destroyed current_compiles, along with potentially using the currently-being-destroyed socket.

So, bail doing anything in async_wait()'s callback if there is an error.

Unfortunately while all the above makes sense, I can't really explain why the failure doesn't happen more often with just nodeos. My hunch is that a crashing oc-monitor on nodeos shutdown is simply never noticed nor has any negative side effects. It's only in these new tests that expect oc-monitor to (as designed) be long lived where something gets angry that oc-monitor has disappeared (because it crashed)

[linh2931]

Do you want to log this for potential debugging or just too much noises?

[heifner]

No higher than debug if you do log it; don't think we need it.

[spoonincode]

Logging config won't be active in the monitor process since it's forked off before main() -- so debug won't be visible, and any error/warn/info messages won't honor the user's logging config either. So probably best to not log anything here.