OpenSearch Snapshot's checksum is invalid

[ciesiolka]

This issue was mentioned once in @3372, but since it is closed, I decided to open a new one.

Describe the bug

yarn run opensearch snapshot command fails due to non-matching checksums.

To reproduce

1. Download OpenSearch-Dashboards repository
2. run yarn run osd bootstrap
3. run yarn run opensearch snapshot

Expected behavior

yarn run opensearch snapshot command is executed successfully.

Screenshots

see log output

Host / Environment

OpenSearch-Dashboard@3268e3722
Windows 10 x64

Additional context

None

Relevant log output

C:\opensearch\OpenSearch-Dashboards>yarn run opensearch snapshot --network-timeout 1000000
yarn run v1.22.19
$ scripts/use_node scripts/opensearch snapshot --network-timeout 1000000
 info Installing from snapshot
   │ info version: 3.0.0
   │ info install path: C:\opensearch\OpenSearch-Dashboards\.opensearch\3.0.0
   │ info license: oss
   │ info Verifying snapshot URL at https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip
   │ info downloading artifact from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip
   │ info downloading artifact checksum from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip.sha512
   │ERROR artifact downloaded from https://artifacts.opensearch.org/snapshots/core/opensearch/3.0.0-SNAPSHOT/opensearch-min-3.0.0-SNAPSHOT-windows-x64-latest.zip does not match expected checksum
   │        expected: 565d4792a00913f094e77ea8d0e0047dc628fcb27d7e2b870126213aa90dd66c12ab76f8ff61d57cede6294ee7018afb7ea2e86b6e1ff91abd526350dc3fc890
   │        received: 1f7dabac5c98117078b8289eb410dacec71513a765c94017b00329cab81388e1781b0afce4b759ae5e07b03c9ab6be1010080b9ebf0b9640c4316aa18542fd9f
   │        headers:
   │          accept-ranges: bytes
   │          connection: close
   │          content-length: 246071451
   │          content-type: application/zip
   │          date: Thu, 14 Dec 2023 09:49:04 GMT
   │          etag: "4134a72eb78c6832804653e566e2393d-47"
   │          last-modified: Thu, 14 Dec 2023 07:16:08 GMT
   │          server: AmazonS3
   │          via: 1.1 b42eb3d87622a28f769121f374a92fda.cloudfront.net (CloudFront)
   │          x-amz-cf-id: ZGjEmtcIpLselNIp-9g10JO2el1qrH_9hPrXf73cpLIfaFatfyKCfg==
   │          x-amz-cf-pop: WAW51-P4
   │          x-amz-replication-status: COMPLETED
   │          x-amz-server-side-encryption: AES256
   │          x-amz-version-id: CIuO5c5BuhitcP0vpMsLvBEuQppPP68j
   │          x-cache: Hit from cloudfront
   │        content[500 / 246071451 base64]: UEsDBBQAAAgIAAAAQQAAAAAAAgAAAAAAAAAaAAAAb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC8DAFBLAwQUAAAICAAAAEEAAAAAAAIAAAAAAAAAHgAAAG9wZW5zZWFyY2gtMy4wLjAtU05BUFNIT1QvbGliLwMAUEsDBBQAAAgIAAAAQQAa+HlGSsvKAJcY3AA7AAAAb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC9saWIvb3BlbnNlYXJjaC0zLjAuMC1TTkFQU0hPVC5qYXKsugOUJVGyLlzmKdu2bdu2bZtdZpdtV5dtddl2ddm22X/PP/euN3gz7973bq6TefbOjIiz4uzQF7kVZUDBUIGAoKCAgICEgP5ygAD99YD+c8qJqQrRScmLM4ACKf4d4Zd2mUohMBDQ5h9q1L8llBOSlxIXU1GllxN3S9RxWGREDN3W+PUbMRIDscHkuEXK85c9TT4GeoIjkoM8r2kMMk9DDvHLo6wKXWTF3QnXdjaXybu85SNoNoVBL7eZK5bjeEtWZ/lzlG8Qfkgw8407CC0216NgrLobTF/ScxHNQJ3KFytbE5uIpfgsfIidgmnTEUUTP69xeaYMat/lkSNyh8hE6Zh712hUaXw8O/gMYZznwLqkuXIdqYNp5pDXd5OfCcFY4o8=
Done in 8.95s.

Originally posted by @ciesiolka in #3372 (comment)

[prudhvigodithi]

[Untriage]
hey @ciesiolka Looks to me like this an issue artifacts.opensearch.org cloud front cache, using ci.opensearch.org should resolve the issue
Example:
https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/windows/x64/zip/builds/opensearch/dist/opensearch-min-3.0.0-windows-x64.zip

For snapshots we can get the latest min artifact from the ci.opensearch.org, do we need to verify the sha512 for min snapshot non prod artifacts that are meant for testing?

Adding @bbarani @peterzhuamazon

[gaiksaya]

Posted one of the solution to invalidate cloudfront on each upload here: #4262 (comment)

[dblock]

That link to latest should be a redirect, not a direct download. In that case it would not fail checksum.

[gaiksaya]

That link to latest should be a redirect, not a direct download. In that case it would not fail checksum.

@dblock The opensearch snapshots are stored in artifacts.opensearch.org which do not have latest redirect.

[dblock]

That link to latest should be a redirect, not a direct download. In that case it would not fail checksum.

@dblock The opensearch snapshots are stored in artifacts.opensearch.org which do not have latest redirect.

Is there an option to fix that instead?

[prudhvigodithi]

We can migrate the entire snapshots logic to ci.opensearch.org, which is now available with latest URL Example: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/3.0.0/latest/linux/x64/tar/builds/opensearch/dist/opensearch-min-3.0.0-linux-x64.tar.gz, but I have noticed there are some references artifacts.opensearch.org hard coded in OpenSearch repo for integration tests which needs to be updated and also today we dont sign with sha512 for an artifact in ci.opensearch.org so this validation logic should be removed or sigh with sha512 for min artifacts pushed to ci.opensearch.org.

But I see even better way is to publish the opensearch-min-3.0.0-linux-x64.tar.gz to maven, so that we dont have worry about these cache and sha512 issues.
@dblock @bbarani @peterzhuamazon @gaiksaya what do you think?

[gaiksaya]

I think I got the reason behind why this happening.

See the workflow here: https://build.ci.opensearch.org/blue/organizations/jenkins/publish-opensearch-min-snapshots/detail/publish-opensearch-min-snapshots/621/pipeline/63
It uploaded the actual artifact for windows but failed to upload the sha512 and other checksum due to extension mismatch. @peterzhuamazon has tried to fix this in opensearch-project/opensearch-build-libraries#318 but looks like we were facing issues with tests in this repo. Will try to figure out and get this in.

Thanks!

Regarding your query @prudhvigodithi we do have a request open to publish entire distribution here #2069

[gaiksaya]

Hi @ciesiolka ,

The fix is in. Please observe the pattern for next few days and let us know if the issue continues to persist. If not, please feel free to close this issue.
Thank you!

[rishabh6788]

Closing this issue, feel free to reopen if you face the issue again.