Update the developer guide to add a section to run remote backed storage from dev environment #16223

srikanthpadakanti · 2024-10-07T20:04:24Z

Description

Add a section to run remote backed storage from dev environment

Related Issues

Check List

Functionality includes testing.
[*] Commits are signed per the DCO using --signoff
[*] API changes companion pull request created, if applicable.
[*] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Srikanth Padakanti <[email protected]>

sandeshkr419 · 2024-10-07T21:58:15Z

DEVELOPER_GUIDE.md

+
+
+    // Add AWS credentials to the keystore
+    keystore 's3.client.default.access_key', '<access_key>'


I don't think storing credentials in this file should be a recommended practice.

How about reading from env variables instead as an alternative unless there are better ways:

keystore 's3.client.default.access_key', System.getenv('AWS_ACCESS_KEY_ID') keystore 's3.client.default.secret_key', System.getenv('AWS_SECRET_ACCESS_KEY') keystore 's3.client.default.session_token', System.getenv('AWS_SESSION_TOKEN')

export AWS_ACCESS_KEY_ID=<access_key> export AWS_SECRET_ACCESS_KEY=<secret_key> export AWS_SESSION_TOKEN=<session_token>

In that way onlly authorized users are able to run this script.

I understand that storing credentials directly isn't the recommended approach, and I agree. You can definitely export these values and access them through system variables. My goal here is to demonstrate how to run OpenSearch locally while connecting to remote storage.

The reason I’m using them directly in the file is that my tokens [access_key, secret, token] are temporarily generated and have a validity of only 45 minutes.

Made the recommended changes.

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti

Made the recommended changes.

srikanthpadakanti · 2024-10-08T21:08:41Z

DEVELOPER_GUIDE.md

+
+
+    // Add AWS credentials to the keystore
+    keystore 's3.client.default.access_key', '<access_key>'


I understand that storing credentials directly isn't the recommended approach, and I agree. You can definitely export these values and access them through system variables. My goal here is to demonstrate how to run OpenSearch locally while connecting to remote storage.

The reason I’m using them directly in the file is that my tokens [access_key, secret, token] are temporarily generated and have a validity of only 45 minutes.

Made the recommended changes.

srikanthpadakanti

Made the recommended changes.

dblock

Thanks! I made a bunch of nitpicky asks, choose the ones that you think are good ones.

CHANGELOG.md

DEVELOPER_GUIDE.md

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]>

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti

Considered the recommended suggestions and made sure to incorporate everyone of them.

srikanthpadakanti

Made the recommended changes

Signed-off-by: Srikanth Padakanti <[email protected]>

dblock

More nits, sorry to be annoying. Thanks for hanging in here with me!

The TOC is the one really needed.

dblock · 2024-10-16T10:40:35Z

DEVELOPER_GUIDE.md

@@ -556,6 +556,40 @@ Then, you need to apply patterns for git-secrets, you can install the AWS standa
 git secrets --register-aws
 ```

+#### Remote Backed Storage


The developer guide has a TOC on the top that needs to be re-generated/updated.

dblock · 2024-10-16T10:40:54Z

DEVELOPER_GUIDE.md

+```
+testClusters {
+   runTask {
+    // Add following lines to enable remote store


To be consistent with the text below.

Suggested change

// Add following lines to enable remote store

// Enable remote store

dblock · 2024-10-16T10:41:06Z

DEVELOPER_GUIDE.md

+    setting 'node.attr.remote_store.repository.my-repository.settings.bucket', '<bucket_name>'
+    setting 'node.attr.remote_store.repository.my-repository.settings.base_path', '<base_path>'
+    setting 'node.attr.remote_store.repository.my-repository.settings.region', '<region>' # e.g. us-west-2
+


Nit: remove one of the empty lines.

dblock · 2024-10-16T10:41:43Z

DEVELOPER_GUIDE.md

+    keystore 's3.client.default.secret_key', System.getenv('AWS_SECRET_ACCESS_KEY')
+    keystore 's3.client.default.session_token', System.getenv('AWS_SESSION_TOKEN')
+```
+Then run by giving the required plugin as parameter


Suggested change

Then run by giving the required plugin as parameter

Then run by specifying the required plugin as parameter.

dblock · 2024-10-16T10:42:08Z

DEVELOPER_GUIDE.md

+```
+Then run by giving the required plugin as parameter
+```
+Exporting values to use as system variable. 


Remove this line, it's text and can't be copy-pasted.

dblock · 2024-10-16T10:42:37Z

DEVELOPER_GUIDE.md

+export AWS_SECRET_ACCESS_KEY=<secret_key>
+export AWS_SESSION_TOKEN=<session_token>
+
+Example: plugin can be 'repository-s3'


In this section it has to be repository-s3, just remove this line.

opensearch-trigger-bot · 2024-11-15T15:22:25Z

This PR is stalled because it has been open for 30 days with no activity.

Update the developer guide opensearch-project#16096

034004a

Signed-off-by: Srikanth Padakanti <[email protected]>

Update CHANGELOG.md

95652b7

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti changed the title ~~Update the developer guide #16096~~ Update the developer guide to add a section to run remote backed storage from dev environment Oct 7, 2024

sandeshkr419 reviewed Oct 7, 2024

View reviewed changes

srikanthpadakanti added 2 commits October 8, 2024 16:12

Update DEVELOPER_GUIDE.md

c5bf212

Signed-off-by: Srikanth Padakanti <[email protected]>

Merge branch 'opensearch-project:main' into main

0cedb5d

srikanthpadakanti commented Oct 8, 2024

View reviewed changes

dblock requested changes Oct 10, 2024

View reviewed changes

github-actions bot added the documentation Improvements or additions to documentation label Oct 10, 2024

dbwiddis reviewed Oct 11, 2024

View reviewed changes

DEVELOPER_GUIDE.md Show resolved Hide resolved

srikanthpadakanti and others added 6 commits October 14, 2024 22:40

Update CHANGELOG.md

5f1d643

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]>

Update DEVELOPER_GUIDE.md

4bf02f4

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]>

Update DEVELOPER_GUIDE.md

94316e9

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]>

Update DEVELOPER_GUIDE.md

07dec52

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]>

Update DEVELOPER_GUIDE.md

acb0d62

Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]> Signed-off-by: Srikanth Padakanti <[email protected]>

Update DEVELOPER_GUIDE.md

3fb28c7

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti commented Oct 15, 2024

View reviewed changes

srikanthpadakanti requested a review from dblock October 15, 2024 03:51

srikanthpadakanti commented Oct 15, 2024

View reviewed changes

Merge branch 'main' into main

8cd5151

Signed-off-by: Srikanth Padakanti <[email protected]>

dblock requested changes Oct 16, 2024

View reviewed changes

opensearch-trigger-bot bot added the stalled Issues that have stalled label Nov 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the developer guide to add a section to run remote backed storage from dev environment #16223

Update the developer guide to add a section to run remote backed storage from dev environment #16223

srikanthpadakanti commented Oct 7, 2024 •

edited by dblock

Loading

sandeshkr419 Oct 7, 2024

srikanthpadakanti Oct 8, 2024

srikanthpadakanti left a comment

srikanthpadakanti Oct 8, 2024

srikanthpadakanti left a comment

dblock left a comment

srikanthpadakanti left a comment

srikanthpadakanti left a comment

dblock left a comment •

edited

Loading

dblock Oct 16, 2024

dblock Oct 16, 2024

dblock Oct 16, 2024

dblock Oct 16, 2024

dblock Oct 16, 2024

dblock Oct 16, 2024

opensearch-trigger-bot bot commented Nov 15, 2024



		// Add AWS credentials to the keystore
		keystore 's3.client.default.access_key', '<access_key>'

	// Add following lines to enable remote store
	// Enable remote store

	Then run by giving the required plugin as parameter
	Then run by specifying the required plugin as parameter.

Update the developer guide to add a section to run remote backed storage from dev environment #16223

Are you sure you want to change the base?

Update the developer guide to add a section to run remote backed storage from dev environment #16223

Conversation

srikanthpadakanti commented Oct 7, 2024 • edited by dblock Loading

Description

Related Issues

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikanthpadakanti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikanthpadakanti left a comment

Choose a reason for hiding this comment

dblock left a comment

Choose a reason for hiding this comment

srikanthpadakanti left a comment

Choose a reason for hiding this comment

srikanthpadakanti left a comment

Choose a reason for hiding this comment

dblock left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opensearch-trigger-bot bot commented Nov 15, 2024

srikanthpadakanti commented Oct 7, 2024 •

edited by dblock

Loading

dblock left a comment •

edited

Loading