Wait after create index complete

[seankao-az]

Description

Add a new spark config spark.datasource.flint.request.completionDelayMillis to set a wait time after OpenSearch index creation is complete.
It's set to 2000 if using aoss service by default, otherwise it's 0.

Related Issues

• [BUG] resource_already_exists_exception when using async OpenSearch service #950

Testing

Create index with default config

24/11/26 23:49:04 INFO FlintOpenSearchClient: Creating Flint index flint_myglue_test_default_no_wait_time with metadata FlintMetadata(...)
24/11/26 23:49:04 INFO FlintOpenSearchClient: Creating Flint index flint_myglue_test_default_no_wait_time
24/11/26 23:49:04 INFO RetryableHttpAsyncClient: Building retryable http async client with options: FlintRetryOptions{maxRetries=3, retryableStatusCodes=429,502, retryableExceptionClassNames=Optional.empty}
24/11/26 23:49:04 INFO HttpStatusCodeResultPredicate: Checking if status code is retryable: 200
24/11/26 23:49:04 INFO HttpStatusCodeResultPredicate: Status code 200 check result: false
24/11/26 23:49:04 INFO FlintOpenSearchIndexMetadataService: Updating Flint index flint_myglue_test_default_no_wait_time with metadata FlintMetadata(...)
24/11/26 23:49:04 INFO RetryableHttpAsyncClient: Building retryable http async client with options: FlintRetryOptions{maxRetries=3, retryableStatusCodes=429,502, retryableExceptionClassNames=Optional.empty}
24/11/26 23:49:04 INFO HttpStatusCodeResultPredicate: Checking if status code is retryable: 200
24/11/26 23:49:04 INFO HttpStatusCodeResultPredicate: Status code 200 check result: false

Create index with config --conf spark.datasource.flint.request.completionDelayMillis=2000

24/11/26 23:50:20 INFO FlintOpenSearchClient: Creating Flint index flint_myglue_test_default_with_wait_time with metadata FlintMetadata(...)
24/11/26 23:50:20 INFO FlintOpenSearchClient: Creating Flint index flint_myglue_test_default_with_wait_time
24/11/26 23:50:20 INFO RetryableHttpAsyncClient: Building retryable http async client with options: FlintRetryOptions{maxRetries=3, retryableStatusCodes=429,502, retryableExceptionClassNames=Optional.empty}
24/11/26 23:50:20 INFO HttpStatusCodeResultPredicate: Checking if status code is retryable: 200
24/11/26 23:50:20 INFO HttpStatusCodeResultPredicate: Status code 200 check result: false
24/11/26 23:50:22 INFO FlintOpenSearchIndexMetadataService: Updating Flint index flint_myglue_test_default_with_wait_time with metadata FlintMetadata(...)
24/11/26 23:50:22 INFO RetryableHttpAsyncClient: Building retryable http async client with options: FlintRetryOptions{maxRetries=3, retryableStatusCodes=429,502, retryableExceptionClassNames=Optional.empty}
24/11/26 23:50:22 INFO HttpStatusCodeResultPredicate: Checking if status code is retryable: 200
24/11/26 23:50:22 INFO HttpStatusCodeResultPredicate: Status code 200 check result: false

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[noCharger]

what's the consideration of constant wait_after VS retry?

[seankao-az]

what's the consideration of constant wait_after VS retry?

Actually we need both. AOSS internal workflow will also conflict with our update mapping call, and in that case our mapping call will just fail. We can use backoff retry as main method to avoid conflict, and wait_after as a configurable adjustment in production if we consistently see conflict happening within x seconds of index creation.

[seankao-az]

confirmed working in gamma