[BUG] org.opensearch.recovery.RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}} is flaky

[reta]

Describe the bug

The test case org.opensearch.recovery.RecoveryWhileUnderLoadIT.testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}} is flaky:

java.lang.AssertionError: All incoming requests on node [node_s4] should have finished. Expected 0 but got 368; pending tasks [[{
  "node" : "node_s4",
  "id" : 76,
  "type" : "direct",
  "action" : "indices:admin/seq_no/retention_lease_sync[p]",
  "status" : {
    "phase" : "primary"
  },
  "description" : "retention_lease_sync shardId=[test][3]",
  "start_time" : "2024-05-24T17:25:25.617Z",
  "start_time_in_millis" : 1716571525617,
  "running_time" : "1m",
  "running_time_in_nanos" : 65752135745,
  "cancellable" : false,
  "cancelled" : false,
  "parent_task_id" : "MWBqDHQmSkefpKFbNwRdHg:75",
  "headers" : { },
  "resource_stats" : {
    "average" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "total" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "min" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "max" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "thread_info" : {
      "thread_executions" : 0,
      "active_threads" : 0
    }
  }
},{
  "node" : "node_s4",
  "id" : 57,
  "type" : "transport",
  "action" : "internal:index/shard/recovery/start_recovery",
  "description" : "",
  "start_time" : "2024-05-24T17:25:25.549Z",
  "start_time_in_millis" : 1716571525549,
  "running_time" : "1m",
  "running_time_in_nanos" : 65820120649,
  "cancellable" : false,
  "cancelled" : false,
  "headers" : { },
  "resource_stats" : {
    "average" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "total" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "min" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "max" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "thread_info" : {
      "thread_executions" : 0,
      "active_threads" : 0
    }
  }
},{
  "node" : "node_s4",
  "id" : 75,
  "type" : "transport",
  "action" : "retention_lease_sync",
  "status" : {
    "phase" : "starting"
  },
  "description" : "retention_lease_sync shardId=[test][3]",
  "start_time" : "2024-05-24T17:25:25.617Z",
  "start_time_in_millis" : 1716571525617,
  "running_time" : "1m",
  "running_time_in_nanos" : 65752504066,
  "cancellable" : false,
  "cancelled" : false,
  "headers" : { },
  "resource_stats" : {
    "average" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "total" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "min" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "max" : {
      "cpu_time_in_nanos" : 0,
      "memory_in_bytes" : 0
    },
    "thread_info" : {
      "thread_executions" : 0,
      "active_threads" : 0
    }
  }
}]]

Related component

Storage

To Reproduce

 ./gradlew ':server:internalClusterTest' --tests "org.opensearch.recovery.RecoveryWhileUnderLoadIT" -Dtests.method="testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest {p0={"cluster.indices.replication.strategy":"DOCUMENT"}}" -Dtests.seed=FE4B39551EA78350

Expected behavior

The test must always pass

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
Standard

Host/Environment (please complete the following information):

• CI

Additional context

• https://build.ci.opensearch.org/job/gradle-check/39211/testReport/junit/org.opensearch.recovery/RecoveryWhileUnderLoadIT/testRecoverWhileUnderLoadAllocateReplicasRelocatePrimariesTest__p0___cluster_indices_replication_strategy___DOCUMENT___/

[sachinpkale]

[Storage Triage - attendees 1 2 3 4 5 6 7 8 9 10 ]

Moving to This Quarter

[sachinpkale]

Not able to repro on local even after 1000+ iterations. Closing.