Fix race condition in SimpleNioTransportTests.testTracerLog (opensear…

…ch-project#12139) (opensearch-project#12145)

The test currently blocks on receiving a response to the request for the
"internal:testNotSeen" action. However, that response is sent from
TransportService before the trace logger [writes its log message][1].
Since the test was not polling for this "sent response" log message to
appear that meant it was possible for the test to remove/stop the mock
log appender concurrently with the logging of that final message. The
fix is to include this final log message as an expectation, so the test
will poll until this message appears and the logger should be quiescent
when the appender is removed and stopped.

[1]: https://github.com/opensearch-project/OpenSearch/blob/71f1fabe149fd0777edf44502ace4a8f0911feeb/server/src/main/java/org/opensearch/transport/TransportService.java#L1273


(cherry picked from commit f55b9e0)

Signed-off-by: Andrew Ross <[email protected]>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

test/framework/src/main/java/org/opensearch/transport/AbstractSimpleTransportTestCase.java

@@ -1284,9 +1284,17 @@ public String executor() {
                 Level.TRACE,
                 notSeenReceived
             );
+            final String notSeenResponseSent = ".*\\[internal:testNotSeen].*sent response.*";
+            final MockLogAppender.LoggingExpectation notSeenResponseSentExpectation = new MockLogAppender.PatternSeenEventExpectation(
+                "sent response",
+                "org.opensearch.transport.TransportService.tracer",
+                Level.TRACE,
+                notSeenResponseSent
+            );
 
             appender.addExpectation(notSeenSentExpectation);
             appender.addExpectation(notSeenReceivedExpectation);
+            appender.addExpectation(notSeenResponseSentExpectation);
 
             PlainTransportFuture<StringMessageResponse> future = new PlainTransportFuture<>(noopResponseHandler);
             serviceA.sendRequest(nodeB, "internal:testNotSeen", new StringMessageRequest(""), future);