[BUG] ClusterManagerTaskThrottlingIT.testTimeoutWhileThrottling is flaky

[andrross]

Observed a couple times such as: https://build.ci.opensearch.org/job/gradle-check/6359/

Can usually be reproduced with the following:

./gradlew ':server:internalClusterTest' -Dtests.iters=30 --tests "org.opensearch.clustermanager.ClusterManagerTaskThrottlingIT"

Example failure:

REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.opensearch.clustermanager.ClusterManagerTaskThrottlingIT.testTimeoutWhileThrottling {seed=[F40BF37A733FFE89:6AF0FE6F91F21DCC]}" -Dtests.seed=F40BF37A733FFE89 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=fr-BE -Dtests.timezone=Asia/Tel_Aviv -Druntime.java=17

org.opensearch.clustermanager.ClusterManagerTaskThrottlingIT > testTimeoutWhileThrottling {seed=[F40BF37A733FFE89:6AF0FE6F91F21DCC]} FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([F40BF37A733FFE89:6AF0FE6F91F21DCC]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.failNotEquals(Assert.java:835)
        at org.junit.Assert.assertEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:633)
        at org.opensearch.clustermanager.ClusterManagerTaskThrottlingIT.testTimeoutWhileThrottling(ClusterManagerTaskThrottlingIT.java:175)

[andrross]

@dhwanilpatel Care to take a look since this is a newly introduced test?

[dblock]

#5142

[dblock]

#5144

[dblock]

@Bukhtawar @shwetathareja @dhwanilpatel can someone please take this? This one failed 3 times for me in 4 PRs this AM.