Move Lucene snapshot from custom repository to artifacts.opensearch.org

[mch2]

Is your feature request related to a problem? Please describe

Today OpenSearch hosts snapshot versions of Lucene at
repo.setUrl("https://d1nvenhzbhpy0q.cloudfront.net/snapshots/lucene/");. These snapshot versions are used to build/test against unreleased versions of Lucene on main & 2.x example. This extra url is a bit sloppy, confusing and requires extra management when we already have a url for artifacts. It would be great to host these instead under the https://artifacts.opensearch.org/snapshots or ci.opensearch.org umbrella.

Describe the solution you'd like

Create a repository under https://artifacts.opensearch.org/snapshots/lucene to host the snapshots. I think all that would be required here is a new path?
For context the existing repo was set up using the https://github.com/opensearch-project/data-prepper/tree/main/release/staging-resources-cdk.

The workflow pushing built snapshots is here. All that would need to change there is the bucket/role used.

Describe alternatives you've considered

Leaving as is.

Additional context

After this is created we can update the WF to push to the new bucket (or both temporarily), update RepositoriesSetupPlugin and any downstream using the old url directly.

Acceptance Criteria

Lucene Snapshots should be downloaded from ci.opensearch.org

[nknize]

+1

[gaiksaya]

Hey!

I believe ci.opensearch.org is a better choice. We eventually want to move https://artifacts.opensearch.org/snapshots/ to ci.opensearch.org/snapshots too. Will take a look into integrating it.

Thanks!

[peterzhuamazon]

I want to mention that apart from moving the snapshot, we also need to start a campaign to make sure repos that use that snapshot will have the new url.

Ex: https://github.com/opensearch-project/ml-commons/blob/main/build.gradle#L33

[peterzhuamazon]

Just talked with @mch2 on this and get the latest status.
Will start working on this task by 1st switching to the staging account (ci.opensearch.org).

Thanks.

[peternied]

We've seen build & test failures [1] from the current hosting setup - 🤞 this change would improve the support and stability.

• [1] [Flaky] Compile failure due to 403 from lucene snapshots security#3597

[peterzhuamazon]

Start looking at the setups to add a new role for the syncing.

[peterzhuamazon]

Able to connect to my own s3 bucket and also improve the github workflow more.
Next is to setup the account connection between aws and github actions, and raise PRs.

Thanks.

[peterzhuamazon]

@mch2 will take care of the dependencies update once the link is available.
We are able to create roles to allow the process to happen.

[peterzhuamazon]

All backend setups are completed now.
PR in OpenSearch:

• Adding lucene snapshot to the ci staging repository OpenSearch#11241

[peterzhuamazon]

Old snapshots are synced to new bucket now.

[peterzhuamazon]

With PR merged now I think @mch2 will start testing both after publishing a new snapshot.

[peterzhuamazon]

New location: https://ci.opensearch.org/ci/dbc/snapshots/lucene/

[peterzhuamazon]

Hi @mch2 we are closing this issue as Infra task is completed, please open another issue if you want to track changing urls.

Thanks.

[peterzhuamazon]

Hi @peternied @owaiskazi19 @mch2 :

Recently I have noticed you updated a lot of the lucene snapshots url.
However it is not artifacts.opensearch.org but ci.opensearch.org.
As I stated in this comment here:
#3874 (comment)

Would you mind correct this url in all repos affected so repos can correctly consume the changes.

Thanks!

[peternied]

My Bad