[Bug]: Checksum check failing for OpenSearch

[joshuarrrr]

Describe the bug

OpenSearch Dashboards out-of-the-box experience downloads a snapshot of OpenSearch to start up and run some integration tests and functional tests. However, there is logic before starting OpenSearch artifact automatically downloaded to verify the checksum. Today, I started noticing that this logic fails when attempting to run integration tests locally:

artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz does not match expected checksum
      expected: a5083ba0bcb0fa5aec8011171af440ed12c8587823f178190ad59032d94988a9656676140ff7b7e736cc8b9b197ee00e4f518871336e92a0917778c553fc4a71
      received: 7708084722a9456373a7d5af958f19d7658bf5edd61c4537399651bb4e43dd690688e2f95cd3d330f5dbbe829ef0403a1d46762969affb2b75a95ad90e847493

It also causes the jenkins test run to fail.

Related to #1497

To reproduce

To reproduce:

Ensure no running version of OpenSearch
unset TEST_OPENSEARCH_FROM
git checkout main
yarn osd bootstrap
yarn test:jest_integration

Notice failure in integration tests.

Expected behavior

Integration tests work out of the box and thus the CI will succeed. Developers should be able to run yarn test:jest_integration without setting TEST_OPENSEARCH_FROM

Screenshots

No response

Host / Environment

1. Local ubuntu
2. Jenkins

Additional context

No response

Relevant log output

artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz does not match expected checksum
      expected: a5083ba0bcb0fa5aec8011171af440ed12c8587823f178190ad59032d94988a9656676140ff7b7e736cc8b9b197ee00e4f518871336e92a0917778c553fc4a71
      received: 7708084722a9456373a7d5af958f19d7658bf5edd61c4537399651bb4e43dd690688e2f95cd3d330f5dbbe829ef0403a1d46762969affb2b75a95ad90e847493

[peterzhuamazon]

We havent even start 3.0 building process, the auto pipeline run is either not producing a working tarball or it is failing.
We are focusing on 2.0.0 GA release for now.

[dblock]

We may have a problem here (or two).

1. Whatever automation is downloading the .tar.gz should not be trying to download 3.0, even though it's already being built. The next major is 2.0.
2. The checksum not matching would be a surprise, but I don't know whether we're signing -SNAPSHOT builds at all?

[peterzhuamazon]

1. You should download from ci.opensearch.org not artifacts.opensearch.org, we havent use the old pipeline for years, even though we are still updating /snapshots it depends on we have a good build on dev jenkins, and you didnt show us which checksum you are comparing against the tarball.
2. @dblock there is no signing at all for any staging, we are here talking about the checksum.

Thanks.

[dblock]

@joshuarrrr Please let us know if ^ doesn't help?

[bbarani]

@joshuarrrr Can you please check the above comments and confirm?

[joshuarrrr]

In dashboards, the artifact URL was set to artifacts.opensearch.org in this PR: opensearch-project/OpenSearch-Dashboards#466

I'll check with the team, but we may need further guidance on how to update the various scripts that reference that base path.

[kavilla]

We can update the reference we might run into the caching issue again with the checksum. With the new reference do we know if the checksum cache buster is more aggressive than the "deprecated" reference?

[peterzhuamazon]

@kavilla
ci.opensearch.org bust every 10min or something or even shorter.
artifacts.opensearch.org bust every 24hrs because that is production.

[bbarani]

@joshuarrrr Do you need any additional details?

[kavilla]

@bbarani,

Yes will be prioritizing this in our repo. However, looking at the bucket and the code I'm not able to figure out where an example OpenSearch Snapshot. Do you or @peterzhuamazon have an example?

Thanks!

[ARud12]

Good day @kavilla

Is there an estimated time when this issue will be fixed?

[kavilla]

Good day @kavilla

Is there an estimated time when this issue will be fixed?

Should be relatively quick fix once provided the updated snapshot URL for OpenSearch but I'm blocked until provided that.

[bbarani]

@kavilla You should be able to get the updated snapshot -min artifact for 3.0 version from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz

[prudhvigodithi]

Hey @joshuarrrr can you please add some updates, if you are still having any issue using
https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-linux-x64-latest.tar.gz
Thank you

[peterzhuamazon]

We suggest using the ci.opensearch.org repo instead of the artifacts.opensearch.org repo.

Sample: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-3.0.0-linux-x64.tar.gz

The latest url is also existing there.

[kavilla]

Sounds good! Thank you!

[peterzhuamazon]

We will keep this open for now as we have these notes:

1. For Dashboards, depends on how the implementation is, could directly pull the artifacts from ci.opensearch.org instead of artifacts.opensearch.org.
2. For OpenSearch, the distribution downloader (?) function that every other plugin uses during their integration, is hardcoded to artifacts.opensearch.org, we need to first switch in the core repo to point to ci.opensearch.org then the downstream can consume that change.
3. This means we might need to add sha512 to every artifacts in ci.opensearch.org, as both opensearch and dashboards are using the checksum to verify the downloads (confirmed on dashboards part at least).

[bbarani]

@saratvemulapalli How hard is to make the base URL dynamic in distribution downloader?

[peterzhuamazon]

Hi @bbarani @prudhvigodithi I just find the old PRs we worked on for this particular case with @mch2.
opensearch-project/OpenSearch#904

As you can see there are two places you need to change.

1. Change the snapshot ivy repo to ci.opensearch.org
2. Change the path of snapshot to whatever path leads to the artifacts on ci.opensearch.org.

Additional concerns:

• Do we need to build new artifacts particularly with --snapshot param or does the distribution downloader plugin can natively taking artifacts built without --snapshot while still considering it as a snapshot artifact.

Thanks.

[saratvemulapalli]

@saratvemulapalli How hard is to make the base URL dynamic in distribution downloader?

@bbarani It is some effort. But its already done :)
See documentation[1], users can use PcustomDistributionUrl to override the url dynamically.

[1] https://github.com/opensearch-project/opensearch-plugins/blob/main/TESTING.md#distribution-download-plugin

[bbarani]

@saratvemulapalli It looks like the plugin teams prefer to modify it without using PcustomDistributionUrl parameter

#2216 (comment)

[saratvemulapalli]

@bbarani

Well, thats the only configuration we have. Is infra team going to deprecate use of artifacts.opensearch.org?
Why do we have two different endpoints?

[peterzhuamazon]

@bbarani

Well, thats the only configuration we have. Is infra team going to deprecate use of artifacts.opensearch.org? Why do we have two different endpoints?

Hi @saratvemulapalli

1. We are not deprecating artifacts.opensearch.org endpoint.
2. When we 1st start opensearch there is no proper staging bucket, therefore we put both snapshots and releases on the production artifacts.opensearch.org bucket.
3. Now we have proper staging bucket we want to switch the snapshots artifacts back to staging on ci.opensearch.org.

Thanks.

[bbarani]

@saratvemulapalli Can you provide your inputs?

[prudhvigodithi]

I believe we can close this issue, as no changes are being made in terms of snapshots retrieval artifacts.opensearch.org should continue to work, it worth to reopen a new issue, if snapshots retrieval is being changed from artifacts.opensearch.org to ci.opensearch.org, which is actually the right thing to do ;).
Closing this issue, please re-open if required.
Thank you