Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move opensearch-oci to the internal plugins directory in this repo #6148

Open
wbeckler opened this issue Feb 1, 2023 · 6 comments
Open

Move opensearch-oci to the internal plugins directory in this repo #6148

wbeckler opened this issue Feb 1, 2023 · 6 comments
Labels
enhancement Enhancement or improvement to existing feature or request

Comments

@wbeckler
Copy link

wbeckler commented Feb 1, 2023

Is your feature request related to a problem? Please describe.
Today there are options to store data in hdfs, azure, and s3, implemented in internal plugins here: https://github.com/opensearch-project/OpenSearch/tree/main/plugins. For some reason, the Oracle version of this is contained in a separate repo, https://github.com/opensearch-project/opensearch-oci-object-storage

The existence of the separate repo means extra maintenance and complexity. If there is a benefit to it being external, then there shouldn't be implementations in this repo for S3 and Azure.

Describe the solution you'd like
Move the contents of https://github.com/opensearch-project/opensearch-oci-object-storage to https://github.com/opensearch-project/OpenSearch/tree/main/plugins

@reta
Copy link
Collaborator

reta commented Feb 2, 2023

@wbeckler at some point we discussed the opposite [1], to move off repository plugins into external repository (play of words :D, but I mean external Github repository). AFAIK we have not made any conclusions out of it, there are pros and cons for both approaches. Probably the most annoying one is the need for monolith release even if single plugin changes (we have a few bugs sneaked in in the past). @dblock @nknize do you think it is worth resuming the discussion?

[1] #1754

@andrross
Copy link
Member

andrross commented Feb 2, 2023

Moving the repository plugins to external GitHub repositories seems like the right direction long term, so I'm hesitant to move the OCI plugin into the monolith. This might actually be a good place to trial the external GitHub repository approach. The first step is to get the OCI plugin into proper shape to include in the OpenSearch distribution. If we get that process working well then we can separate out the other repositories (or if it doesn't work well give up on the approach and move OCI into the monolith :) ).

@reta
Copy link
Collaborator

reta commented Feb 2, 2023

This might actually be a good place to trial the external GitHub repository approach.

@andrross we did that ~ a year ago: https://github.com/reta/opensearch-repository-plugins as part of POC #1754

@andrross
Copy link
Member

andrross commented Feb 2, 2023

@reta But we never actually used it when creating distributions, right? I think that will be the real test of the overhead involved (e.g. updating version numbers for new releases even when there are no other changes, etc).

@reta
Copy link
Collaborator

reta commented Feb 2, 2023

@reta But we never actually used it when creating distributions, right? I think that will be the real test of the overhead involved (e.g. updating version numbers for new releases even when there are no other changes, etc).

Oh no, right, we basically had a repo which was 100% buildable (at the time), but we didn't push it further. I think it was a first step to estimate how hard that part would be.

@minalsha minalsha removed the untriaged label Feb 3, 2023
@saratvemulapalli
Copy link
Member

+1 There is benefit to keep these plugins in their dedicated repos, it keeps OpenSearch leaner which enables us to move faster. None of these plugins shouldn't block building/releasing OpenSearch.
Moving OCI to the core will bring added baggage and generally we want to pull out these plugins from the core.

If there is a benefit to it being external, then there shouldn't be implementations in this repo for S3 and Azure.

These plugins are inherited from the fork, if we have maintainers and resources we should move them away from the core.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request
Projects
None yet
Development

No branches or pull requests

5 participants