Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Remote Store] Add support for Remote Translog Store stats in _remotestore/stats/ API #9263

Conversation

BhumikaSaini-Amazon
Copy link
Contributor

@BhumikaSaini-Amazon BhumikaSaini-Amazon commented Aug 11, 2023

Description

This is an incremental PR for PR #8908. This PR adds the logic to surface stats related to Remote Translog Store upload and download flows, when queried via the _remotestore/stats/ API.

A sample response would look like:

"4": [ // shard number
  {
    "routing" : {
      "state" : "STARTED",
      "primary" : true, // note that this is the primary copy
      "node" : "9GGM_5tWSIGjx9gd4-P4cg"
    },
    "segment" : { // segment stats
      "download" : { }, // segment stats for download flow
      "upload" : { } // segment stats for upload flow
    },
    "translog" : { // translog stats
      "upload" : { // translog stats for upload flow
        "last_successful_upload_timestamp" : 1692192354058, // System time in millis when the last successful upload completed
        "total_uploads" : { // Number of syncs to Remote Translog Store
          "started" : 24,
          "failed" : 0,
          "succeeded" : 24
        },
        "total_uploads_in_bytes" : { // Number of bytes based on files (.tlog, .ckp, metadata) uploaded
          "started" : 2556,
          "failed" : 0,
          "succeeded" : 2556
        },
        "total_upload_time_in_millis" : 125, // Total time duration in millis spent on remote store uploads
        "upload_size_in_bytes" : { // Moving avg of bytes uploaded in a successful sync to remote (i.e. all files successfully uploaded for that sync to remote)
          "moving_avg" : 255.6
        },
        "upload_speed_in_bytes_per_sec" : { // Moving avg of bytes per sec speed for a successful sync to remote (i.e. all files successfully uploaded for that sync to remote)
          "moving_avg" : 21575.3
        },
        "upload_time_in_millis" : { // Moving avg of time duration in millis spent on a successful sync to remote (i.e. all files successfully uploaded for that sync to remote)
          "moving_avg" : 12.5
        }
      },
      "download" : { // translog stats for download flow
        "last_successful_download_timestamp" : 1692192296107, // System time in millis when the last successful download completed
        "total_downloads" : { // Number of syncs to Remote Translog Store
          "succeeded" : 7
        },
        "total_downloads_in_bytes" : { // Number of bytes based on files (.tlog, .ckp, metadata) successfully downloaded
          "succeeded" : 664
        },
        "total_download_time_in_millis" : 53, // Total time duration in millis spent on successful remote store downloads
        "download_size_in_bytes" : { // Moving avg of bytes downloaded in a successful sync from remote (i.e. all files successfully downloaded for that sync from remote)
          "moving_avg" : 664.0
        },
        "download_speed_in_bytes_per_sec" : { // Moving avg of bytes per sec speed for a successful sync from remote (i.e. all files successfully downloaded for that sync from remote)
          "moving_avg" : 12528.0
        },
        "download_time_in_millis" : { // Moving avg of time duration in millis spent on a successful sync from remote (i.e. all files successfully downloaded for that sync from remote)
          "moving_avg" : 53.0
        }
      }
    }
  },
  {
    "routing" : {
      "state" : "STARTED",
      "primary" : false, // note that this is the replica copy
      "node" : "zXMfpae7RBW9pIloLbxQiw"
    },
    "segment" : { // segment stats
      "download" : { }, // segment stats for download flow
      "upload" : { } // segment stats for upload flow
    },
    "translog" : { // translog stats
      "upload" : { }, // translog stats for upload flow
      "download" : { } // translog stats for download flow
    }
  }
]

Testing: gist

Related Issues

#8311

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 28m 52s

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 23m 58s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@BhumikaSaini-Amazon BhumikaSaini-Amazon force-pushed the rts-stats-incremental-pr-2 branch from 90fe741 to 9adab8b Compare August 14, 2023 05:04
@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 25m 13s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@opensearch-trigger-bot
Copy link
Contributor

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

BUILD SUCCESSFUL in 25m 20s

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Compatibility status:

Checks if related components are compatible with change cc052d3

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Compatibility status:

Checks if related components are compatible with change 875a5e5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Compatibility status:

Checks if related components are compatible with change 84c6374

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Compatibility status:

Checks if related components are compatible with change 5aedc39

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

Signed-off-by: Bhumika Saini <[email protected]>
@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Compatibility status:

Checks if related components are compatible with change ca4e588

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Compatibility status:

Checks if related components are compatible with change 7476bdf

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2023

Gradle Check (Jenkins) Run Completed with:

@codecov
Copy link

codecov bot commented Sep 4, 2023

Codecov Report

Merging #9263 (7476bdf) into main (731e043) will decrease coverage by 0.03%.
Report is 3 commits behind head on main.
The diff coverage is 84.54%.

@@             Coverage Diff              @@
##               main    #9263      +/-   ##
============================================
- Coverage     71.25%   71.23%   -0.03%     
- Complexity    57811    57863      +52     
============================================
  Files          4806     4809       +3     
  Lines        272240   272615     +375     
  Branches      39730    39755      +25     
============================================
+ Hits         193996   194185     +189     
- Misses        61941    62104     +163     
- Partials      16303    16326      +23     
Files Changed Coverage Δ
...testore/stats/TransportRemoteStoreStatsAction.java 41.02% <0.00%> (-3.42%) ⬇️
...java/org/opensearch/discovery/DiscoveryModule.java 90.27% <ø> (ø)
...arch/index/remote/RemoteStorePressureSettings.java 87.09% <ø> (ø)
...ain/java/org/opensearch/indices/IndicesModule.java 98.23% <ø> (-0.02%) ⬇️
server/src/main/java/org/opensearch/node/Node.java 85.56% <60.00%> (-0.55%) ⬇️
...rg/opensearch/index/translog/RemoteFsTranslog.java 70.05% <75.86%> (-0.06%) ⬇️
...ch/index/remote/RemoteTranslogTransferTracker.java 79.51% <79.51%> (ø)
...search/index/shard/RemoteStoreRefreshListener.java 86.18% <80.00%> (-4.98%) ⬇️
...dex/translog/transfer/TranslogTransferManager.java 72.22% <81.13%> (-4.74%) ⬇️
...arch/gateway/remote/RemoteClusterStateService.java 81.03% <83.33%> (+0.10%) ⬆️
... and 12 more

... and 480 files with indirect coverage changes

@sachinpkale sachinpkale merged commit cf50add into opensearch-project:main Sep 4, 2023
@sachinpkale sachinpkale added the backport 2.x Backport to 2.x branch label Sep 4, 2023
@opensearch-trigger-bot
Copy link
Contributor

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9263-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 cf50add98f2935b469b715aa28dd7b248f36b4ba
# Push it to GitHub
git push --set-upstream origin backport/backport-9263-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9263-to-2.x.

BhumikaSaini-Amazon pushed a commit to BhumikaSaini-Amazon/OpenSearch that referenced this pull request Sep 4, 2023
sachinpkale pushed a commit that referenced this pull request Sep 5, 2023
…store/stats/ API (#9738)

* [Remote Store] Add support for Remote Translog Store stats in _remotestore/stats/ API (#9263)

---------

Signed-off-by: Bhumika Saini <[email protected]>

* Empty commit to retrigger build

Signed-off-by: Bhumika Saini <[email protected]>

---------

Signed-off-by: Bhumika Saini <[email protected]>
kaushalmahi12 pushed a commit to kaushalmahi12/OpenSearch that referenced this pull request Sep 12, 2023
…estore/stats/` API (opensearch-project#9263)

---------

Signed-off-by: Bhumika Saini <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Kaushal Kumar <[email protected]>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…estore/stats/` API (opensearch-project#9263)

---------

Signed-off-by: Bhumika Saini <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Ivan Brusic <[email protected]>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…estore/stats/` API (opensearch-project#9263)

---------

Signed-off-by: Bhumika Saini <[email protected]>
Signed-off-by: Ashish Singh <[email protected]>
Signed-off-by: Shivansh Arora <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch backport-failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants