OutOfMemoryException causes messages to be treated as poison

[mikeminutillo]

Describe the bug

Description

When the endpoint process does not have sufficient memory available, messages may be treated as poison and moved to the error queue. Unfortunately, poison messages are not decorated with the headers required for ServiceControl which means that the message will not be visible in ServicePulse or ServiceInsight.

Expected behavior

When an OutOfMemoryException occurs, the message should follow the standard recoverability rules, being automatically retried and eventually moved to the error queue where it can be retried (after the memory pressure problem has been resolved).

Actual behavior

When an OutOfMemoryException occurs, the message is immediately moved to the error queue and it lacks a FailedQ header that would enable it to be retried.

Versions

Please list the version of the relevant packages or applications in which the bug exists.

• 7.x
• 6.x

Steps to reproduce

1. Create a new endpoint with SQL Transport
2. Set up a ServiceControl instance with SQL Transport
3. Run the endpoint in a memory constrained environment
4. Send messages to the endpoint
5. Wait for the Custom Check in ServiceControl to trigger indicating that the message could not be imported.
6. Run ServiceControl in maintenance mode to inspect the message. Note that the NServiceBus.FailedQ header is missing
7. Investigate the endpoint logs looking for OutOfMemoryException

Relevant log output

No response

Additional Information

Workarounds

Possible solutions

1. Increase the memory available to the endpoint process.
2. Run ServiceControl in maintenance mode and add a header with a key of NServiceBus.FailedQ and a value of the endpoint address that the message should be returned to
3. Run the re-import function of ServiceControl
4. Retry the failed message

Additional information

The root cause seems to be this code which treats any exception thrown while parsing a message as the message being poison.