[BUG] Fatal Error on getting AD results, testing doesn't discover this.

[amitgalitz]

Describe the bug
Run ./gradlew :run for AD
Create detector
wait for detector to get results and create result index

Fatal error occurs:

fatal error in thread [opensearch[integTest-0][ad-threadpool][T#1]], exiting
java.lang.NoSuchMethodError: 'org.opensearch.action.admin.indices.create.CreateIndexRequest org.opensearch.action.admin.indices.create.CreateIndexRequest.mapping(java.lang.String, java.lang.String, org.opensearch.common.xcontent.XContentType)'
	at org.opensearch.jobscheduler.spi.utils.LockService.createLockIndex(LockService.java:86)
	at org.opensearch.jobscheduler.spi.utils.LockService.acquireLock(LockService.java:119)
	at org.opensearch.ad.AnomalyDetectorJobRunner.lambda$runJob$2(AnomalyDetectorJobRunner.java:164)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)

This is due to job-scheduler-spi dependency using a method that no longer exists in core. This was fixed with this PR: opensearch-project/job-scheduler#155. However all tests are passing. I will further update this with potential solutions that could be as easy as the dependency not being properly updated and used but this issue also signifies that this isn't caught in our CI currently or there is a mismatch between local and CI dependencies.

Followed these steps to more confidently say this isn't a cache problem:

(1) ./gradlew --stop to ensure all daemons are stopped

(2) delete all folders/files in .gradle folder in the project

(3) delete .gradle folder in /home/username/

(4) reboot

as well as deleting local m2 repos

[aksingh-es]

+1.Thanks for pointing this out.
Tried creating an index , ingesting data and then creating an AD detector.
AD job failed with below trace :
`» [2022-04-01T17:30:03,847][INFO ][o.o.a.AnomalyDetectorJobRunner] [integTest-0] Start to run AD job 3Icr5n8BOkw5uIPzcS6l
» [2022-04-01T17:30:04.300891Z] [BUILD] Stopping node

=== Standard error of node node{::integTest-0} ===
» ↓ last 40 non error or warning messages from /local/home/snghes/anomaly-detection/build/testclusters/integTest-0/logs/opensearch.stderr.log ↓
» fatal error in thread [opensearch[integTest-0][ad-threadpool][T#1]], exiting
» java.lang.NoSuchMethodError: 'org.opensearch.action.admin.indices.create.CreateIndexRequest org.opensearch.action.admin.indices.create.CreateIndexRequest.mapping(java.lang.String, java.lang.String, org.opensearch.common.xcontent.XContentType)'
» at org.opensearch.jobscheduler.spi.utils.LockService.createLockIndex(LockService.java:86)
» at org.opensearch.jobscheduler.spi.utils.LockService.acquireLock(LockService.java:119)
» at org.opensearch.ad.AnomalyDetectorJobRunner.lambda$runJob$2(AnomalyDetectorJobRunner.java:164)
» at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
» at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
» at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733)
» at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
» at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
» at java.base/java.lang.Thread.run(Thread.java:832)

Task :run FAILED

`

[amitgalitz]

Found error, the job-scheduler zip currently in the repo was compiled right before the fix was made on job-scheduler. The spi that is pulled from maven is only at compile time and the actual job-scheduler zip that is installed into the integCluster is the zip in src/test/resources/job-scheduler. Updating this to use latest from ci.opensearch.org.