Stream read pool and default s3 timeouts tuning

[vikasvb90]

Description

On upload of large shards (large segments or more number of smaller segments), we noticed s3 errors. For large segments, api timeouts were observed and for large shards of small segments connection acquisition timeout was observed as below. Also, for small instances having low cpu cores, stream read pool capacity becomes low which processes upload requests slowly.
This change is to increase the capacity of stream read pool and increase connection acquisition and api timeouts.

[2023-10-25T07:47:37,139][ERROR][o.o.i.s.RemoteDirectory  ] [51844e1377c6a637328871dc96b00d5e] Failed to upload blob _3h_Lucene90_0.tim
software.amazon.awssdk.core.exception.SdkClientException: Failed to send multipart upload requests.
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
        at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
        at org.opensearch.repositories.s3.async.AsyncTransferManager.handleException(AsyncTransferManager.java:278)
        at org.opensearch.repositories.s3.async.AsyncTransferManager.lambda$handleExceptionOrResponse$14(AsyncTransferManager.java:227)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:79)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28969/
• CommitID: 8ce5fde
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28970/
• CommitID: f0a373e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Compatibility status:

Checks if related components are compatible with change 6ed696f

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/28974/
• CommitID: 17013ad
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[kkmr]

Also, do we have any perf results showing how things have improved?

[vikasvb90]

Also, do we have any perf results showing how things have improved?

This is not about perf. This is to handle upload bursts occurring due to large file uploads. Increasing timeouts is a temporary change till we use a separate dedicated slow client which can handle special cases by putting them behind a producer-consumer queue and processing them slowly without impacting critical uploads.

For large uploads happening on a smaller instance, this timeout may still not be sufficient.

[github-actions]

❌ Gradle check result for 3ba52e3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 6ed696f: SUCCESS

[codecov]

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (5bb6cae) 71.21% compared to head (6ed696f) 71.36%.

Files	Patch %	Lines
...opensearch/repositories/s3/S3RepositoryPlugin.java	0.00%	11 Missing ⚠️
...ensearch/repositories/s3/StatsMetricPublisher.java	66.66%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10912      +/-   ##
============================================
+ Coverage     71.21%   71.36%   +0.14%     
- Complexity    58926    59002      +76     
============================================
  Files          4890     4890              
  Lines        277434   277447      +13     
  Branches      40313    40313              
============================================
+ Hits         197567   197989     +422     
+ Misses        63464    62992     -472     
- Partials      16403    16466      +63     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Bukhtawar]

I feel these should made a function of cores. For eg: instances with fewer cores will run out of higher read timeouts with too many connections since we have limited threads to process

cc: @vikasvb90

[vikasvb90]

Yes, it should be but we will have to first fix the blocking reads from disk happening in stream reader pool and then tune all timeouts as well as this connection count based on benchmarks.