[BUG] org.opensearch.cluster.service.TaskBatcherTests.testNoTasksAreDroppedInParallelSubmission is flaky

[kkewwei]

Describe the bug

[BUG] org.opensearch.cluster.service.TaskBatcherTests.testNoTasksAreDroppedInParallelSubmission is flaky

Related component

Other

To Reproduce

./gradlew ':server:test' --tests "org.opensearch.cluster.service.TaskBatcherTests.testNoTasksAreDroppedInParallelSubmission" -Dtests.seed=F1295294CAA9FEE5 -Dtests.security.manager=true -Dtests.jvms=1 -Dtests.heap.size=1g -Dtests.jvm.argline="-XX:ReservedCodeCacheSize=96m" -Dtests.locale=id-ID -Dtests.timezone=Antarctica/Mawson -Druntime.java=17

If numberOfThreads is 8 and tasksSubmittedPerThread =1024, it will create at most 8192 threads, which will cause native memory oom.

OpenSearch/server/src/test/java/org/opensearch/cluster/service/TaskBatcherTests.java

Line 282 in f3d2bee

public void testNoTasksAreDroppedInParallelSubmission() throws BrokenBarrierException, InterruptedException {

public void testNoTasksAreDroppedInParallelSubmission() throws BrokenBarrierException, InterruptedException {
        int numberOfThreads = randomIntBetween(2, 8);
        TaskExecutor[] executors = new TaskExecutor[numberOfThreads];
        for (int i = 0; i < numberOfThreads; i++) {
            executors[i] = new TaskExecutor();
        }

        int tasksSubmittedPerThread = randomIntBetween(2, 1024);
       ......
        for (int i = 0; i < numberOfThreads; i++) {
            final int index = i;
            Thread thread = new Thread(() -> {
                try {
                    for (int j = 0; j < tasksSubmittedPerThread; j++) {
                        int taskNumber = j;
                        Thread taskThread = new Thread(() -> {
                                ........
                         }
                        taskThread.start();
                     }
                  }
              }
              thread.start();
          }
          ......

In my case, when it create 2603 threads, and it throw such exception:

  2>   2603) Thread[id=12952, name=Thread-8997, state=WAITING, group=TGRP-TaskBatcherTests]
  1> [2024-03-21T08:22:49,024][INFO ][o.o.c.s.TaskBatcherTests ] [testNoTasksAreDroppedInParallelSubmission] after test
  2> NOTE: leaving temporary files on disk at: /home/code/server/build/testrun/test/temp/org.opensearch.cluster.service.TaskBatcherTests_F1295294CAA9FEE5-001
  2> Mar 21, 2024 11:22:49 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
  2> java.lang.Exception: Suite timeout exceeded (>= 1200000 msec).
        at __randomizedtesting.SeedInfo.seed([F1295294CAA9FEE5]:0)
  2> REPRODUCE WITH: ./gradlew ':server:test' --tests "org.opensearch.cluster.service.TaskBatcherTests" -Dtests.seed=F1295294CAA9FEE5 -Dtests.security.manager=true -Dtests.jvms=1 -Dtests.heap.size=1g -Dtests.jvm.argline="-XX:ReservedCodeCacheSize=96m" -Dtests.locale=en-US -Dtests.timezone=Asia/Shanghai -Druntime.java=17
  2> com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=11551, name=Thread-7596, state=RUNNABLE, group=TGRP-TaskBatcherTests]

        Caused by:
        java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
            at __randomizedtesting.SeedInfo.seed([F1295294CAA9FEE5]:0)
            at java.base/java.lang.Thread.start0(Native Method)
            at java.base/java.lang.Thread.start(Thread.java:802)
            at org.opensearch.cluster.service.TaskBatcherTests.lambda$testNoTasksAreDroppedInParallelSubmission$4(TaskBatcherTests.java:334)

Expected behavior

Test should pass

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[shwetathareja]

This doesnt look like a test issue.
The test failed doe to memory constraint

 java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

[shwetathareja]

Thanks @kkewwei for fixing it.