Add retryable AOSS HTTP response

[seankao-az]

Description

Add retryable AOSS http response.
Please check the linked issue.
For AOSS, retry if status code is 400 and error message contains "resource_already_exists_exception"

Related Issues

• [BUG] resource_already_exists_exception when using async OpenSearch service #950

Tests

24/12/08 13:50:00.835 INFO {QID=****} (main) RetryableHttpAsyncClient: Building retryable http async client with options: FlintRetryOptions{maxRetries=3, retryableStatusCodes=429,502, retryableExceptionClassNames=Optional.empty}

...

24/12/08 13:50:01.047 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Checking if status code is retryable: 400
24/12/08 13:50:01.047 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Status code 400 check result: false
24/12/08 13:50:01.047 INFO {QID=****} (main) HttpAOSSResultPredicate: Checking if response is retryable
24/12/08 13:50:01.049 INFO {QID=****} (main) HttpAOSSResultPredicate: Check retryable response result: true
24/12/08 13:50:01.049 ERROR {QID=****} (main) FlintRetryOptions: Attempt to execute request failed: ExecutionAttemptedEvent[result=HTTP/1.1 400 Bad Request [content-type: application/json; charset=UTF-8, content-length: 423, x-envoy-upstream-service-time: 134, date: Sun, 08 Dec 2024 13:50:00 GMT, server: aoss-amazon-m, x-request-id: ****] shaded.flint.org.apache.http.entity.BufferedHttpEntity@4d2acd3c, exception=null]
24/12/08 13:50:01.994 WARN {QID=****} (main) FlintRetryOptions: Retrying failed request at #1

...

24/12/08 13:50:02.118 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Checking if status code is retryable: 400
24/12/08 13:50:02.118 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Status code 400 check result: false
24/12/08 13:50:02.118 INFO {QID=****} (main) HttpAOSSResultPredicate: Checking if response is retryable
24/12/08 13:50:02.118 INFO {QID=****} (main) HttpAOSSResultPredicate: Check retryable response result: true
24/12/08 13:50:02.118 ERROR {QID=****} (main) FlintRetryOptions: Attempt to execute request failed: ExecutionAttemptedEvent[result=HTTP/1.1 400 Bad Request [content-type: application/json; charset=UTF-8, content-length: 423, x-envoy-upstream-service-time: 134, date: Sun, 08 Dec 2024 13:50:00 GMT, server: aoss-amazon-m, x-request-id: f249d2d6-e137-949e-acb6-97291707981b] shaded.flint.org.apache.http.entity.BufferedHttpEntity@5f3e07b5, exception=null]
24/12/08 13:50:04.125 WARN {QID=****} (main) FlintRetryOptions: Retrying failed request at #2

...

24/12/08 13:50:04.270 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Checking if status code is retryable: 400
24/12/08 13:50:04.271 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Status code 400 check result: false
24/12/08 13:50:04.271 INFO {QID=****} (main) HttpAOSSResultPredicate: Checking if response is retryable
24/12/08 13:50:04.271 INFO {QID=****} (main) HttpAOSSResultPredicate: Check retryable response result: true
24/12/08 13:50:04.271 ERROR {QID=****} (main) FlintRetryOptions: Attempt to execute request failed: ExecutionAttemptedEvent[result=HTTP/1.1 400 Bad Request [content-type: application/json; charset=UTF-8, content-length: 423, x-envoy-upstream-service-time: 134, date: Sun, 08 Dec 2024 13:50:00 GMT, server: aoss-amazon-m, x-request-id: f249d2d6-e137-949e-acb6-97291707981b] shaded.flint.org.apache.http.entity.BufferedHttpEntity@1229de5e, exception=null]
24/12/08 13:50:08.258 WARN {QID=****} (main) FlintRetryOptions: Retrying failed request at #3
24/12/08 13:50:08.362 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Checking if status code is retryable: 400
24/12/08 13:50:08.363 INFO {QID=****} (main) HttpStatusCodeResultPredicate: Status code 400 check result: false
24/12/08 13:50:08.363 INFO {QID=****} (main) HttpAOSSResultPredicate: Checking if response is retryable
24/12/08 13:50:08.363 INFO {QID=****} (main) HttpAOSSResultPredicate: Check retryable response result: true
24/12/08 13:50:08.363 ERROR {QID=****} (main) FlintRetryOptions: Attempt to execute request failed: ExecutionAttemptedEvent[result=HTTP/1.1 400 Bad Request [content-type: application/json; charset=UTF-8, content-length: 423, x-envoy-upstream-service-time: 134, date: Sun, 08 Dec 2024 13:50:00 GMT, server: aoss-amazon-m, x-request-id: f249d2d6-e137-949e-acb6-97291707981b] shaded.flint.org.apache.http.entity.BufferedHttpEntity@a56d443, exception=null]
24/12/08 13:50:08.434 ERROR {QID=****} (main) CustomLogging: {"timestamp":1733665808434,"severityText":"ERROR","severityNumber":17,"body":{"message":"OpenSearch Operation failed.","statusCode":400},"attributes":{"domainName":"opensearch","clientId":"UNKNOWN","exception.type":"org.opensearch.OpenSearchStatusException","exception.message":"OpenSearch exception [type=resource_already_exists_exception, reason=Update to the indices [****] failed due to either concurrent update or deletion of the indices]"}}
OpenSearchStatusException[OpenSearch exception [type=resource_already_exists_exception, reason=Update to the indices [****] failed due to either concurrent update or deletion of the indices]]
	at org.opensearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:207)
	at org.opensearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2228)
	at org.opensearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2205)
	at org.opensearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1924)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[dai-chen]

Shall we backport to 0.6? Is 0.6.1 the next release?

[seankao-az]

yes

[seankao-az]

rebased for the sbt error in CI
there's one new commit before that: 1942b80
also updated description with test log