[BUG] org.opensearch.snapshots.ConcurrentSnapshotsIT.testQueuedOperationsOnClusterManagerRestart fails intermittently

[reta]

Describe the bug

The org.opensearch.snapshots.ConcurrentSnapshotsIT.testQueuedOperationsOnMasterRestart fails intermittently

1> [2022-03-05T01:29:45,305][WARN ][o.o.s.SnapshotShardsService] [node_t1] [[index-one][0]][test-repo:snapshot-one/Fh6ASurfS9eE0PWPrZeJ7w] failed to snapshot shard
  1> org.opensearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
  1> 	at org.opensearch.common.util.concurrent.FutureUtils.rethrowExecutionException(FutureUtils.java:104) ~[main/:?]
  1> 	at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:96) ~[main/:?]
  1> 	at org.opensearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:123) ~[main/:?]
  1> 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.OpenSearchExecutors$DirectExecutorService.execute(OpenSearchExecutors.java:296) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:118) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:110) [main/:?]
  1> 	at java.util.ArrayList.forEach(ArrayList.java:1541) [?:?]
  1> 	at org.opensearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:110) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:173) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:147) [main/:?]
  1> 	at org.opensearch.action.StepListener.innerOnFailure(StepListener.java:80) [main/:?]
  1> 	at org.opensearch.action.NotifyOnceListener.onFailure(NotifyOnceListener.java:60) [main/:?]
  1> 	at org.opensearch.action.support.GroupedActionListener.onFailure(GroupedActionListener.java:96) [main/:?]
  1> 	at org.opensearch.repositories.blobstore.BlobStoreRepository.lambda$fileQueueListener$69(BlobStoreRepository.java:2713) [main/:?]
  1> 	at org.opensearch.action.ActionListener$2.onFailure(ActionListener.java:109) [main/:?]
  1> 	at org.opensearch.action.ActionRunnable.onFailure(ActionRunnable.java:101) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:780) [main/:?]
  1> 	at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [main/:?]
  1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  1> Caused by: java.util.concurrent.ExecutionException: java.io.IOException: already closed
  1> 	at org.opensearch.common.util.concurrent.BaseFuture$Sync.getValue(BaseFuture.java:281) ~[main/:?]
  1> 	at org.opensearch.common.util.concurrent.BaseFuture$Sync.get(BaseFuture.java:255) ~[main/:?]
  1> 	at org.opensearch.common.util.concurrent.BaseFuture.get(BaseFuture.java:77) ~[main/:?]
  1> 	at org.opensearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:89) ~[main/:?]
  1> 	... 20 more

To Reproduce
REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.snapshots.ConcurrentSnapshotsIT.testQueuedOperationsOnMasterRestart" -Dtests.seed=897564BD2331D8E0 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=ar-DZ -Dtests.timezone=Etc/GMT-8 -Druntime.java=11

Expected behavior
Test should pass

Plugins
N/A

Screenshots
N/A

Host/Environment (please complete the following information):

• OS: CI
• Version: 1.3.x

Additional context
See please Add any other context about the problem here.

[dreamer-89]

Closing this ticket as not failure instance reported in last 2 months. Used script to determine failure.
#1715 (comment)