Scoped the default S3 endpoint to region of the client #9459

raghuvanshraj · 2023-08-21T15:12:56Z

Description

The default S3 endpoint in the repository-s3 plugin was the S3 global endpoint, which caused buckets created in regions other than in us-east-1 to fail PUT repository calls with a 307 due to delay in the DNS propagation for the new bucket. This change modifies the default endpoint to the region specific one.

Related Issues

[BUG][Opensearch] Opensearch snapshot management: can't select AWS region for configuring s3 bucket. #9265

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-21T15:24:18Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23126/
CommitID: 8029958
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-21T15:50:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23127/
CommitID: bb7481b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-21T16:23:37Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23129/
CommitID: f9a938a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-22T04:36:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23153/
CommitID: 0ac9745
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

raghuvanshraj · 2023-08-22T08:51:18Z

macOS precommit failing due to the following issue: actions/runner-images#8104

time="2023-08-22T08:46:42Z" level=warning msg="[hostagent] failed to open the QMP socket \"/Users/runner/.lima/colima/qmp.sock\", forcibly killing QEMU"

github-actions · 2023-08-22T08:53:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23173/
CommitID: 2ad6cc7
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Raghuvansh Raj <[email protected]>

github-actions · 2023-08-22T09:52:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.classMethod
      1 org.opensearch.common.util.concurrent.QueueResizableOpenSearchThreadPoolExecutorTests.testResizeQueueDown

URL: https://build.ci.opensearch.org/job/gradle-check/23180/
CommitID: 6a96cbc
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-08-22T09:53:43Z

Codecov Report

Attention: Patch coverage is 62.50000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 70.42%. Comparing base (48d4087) to head (6a96cbc).
Report is 1011 commits behind head on main.

Files	Patch %	Lines
...org/opensearch/repositories/s3/S3AsyncService.java	50.00%	0 Missing and 2 partials ⚠️
...java/org/opensearch/repositories/s3/S3Service.java	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #9459      +/-   ##
============================================
- Coverage     71.09%   70.42%   -0.68%     
+ Complexity    57433    56780     -653     
============================================
  Files          4776     4776              
  Lines        270811   270817       +6     
  Branches      39582    39584       +2     
============================================
- Hits         192545   190728    -1817     
- Misses        62068    63792    +1724     
- Partials      16198    16297      +99

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

raghuvanshraj · 2023-08-22T10:08:45Z

Tagging @dblock for review.

gbbafna · 2023-08-22T15:29:49Z

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3Service.java

+        String s3Endpoint = Strings.hasText(clientSettings.region)
+            ? "s3." + Region.of(clientSettings.region).toString() + ".amazonaws.com"
+            : GLOBAL_S3_ENDPOINT;


can we retrieve the endpoint from sdk itself , else this will break for CN and other non-classic partitions .

dblock

Good call. This needs some tests, please.

opensearch-trigger-bot · 2023-09-27T15:20:37Z