Add darwin-arm64-tar and no-jdk-darwin-arm64-tar archive distributions.

[sharp-pixel]

Signed-off-by: Cédric Pelvet [email protected]

Description

Add darwin-arm64-tar and no-jdk-darwin-arm64-tar flavors for the archives distribution.

Issues Resolved

#1667

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[opensearch-ci-bot]

Can one of the admins verify this patch?

[dblock]

Nice, thanks!

[opensearch-ci-bot]

✅   Gradle Wrapper Validation success 42864d2cf6e6d59ae4525005403647a709e417e0

[dblock]

start gradle check

[opensearch-ci-bot]

✅   Gradle Precommit success 42864d2cf6e6d59ae4525005403647a709e417e0

[opensearch-ci-bot]

❌   Gradle Check failure 42864d2cf6e6d59ae4525005403647a709e417e0
Log 1369

Reports 1369

[opensearch-ci-bot]

❌   Gradle Check failure 42864d2cf6e6d59ae4525005403647a709e417e0
Log 1364

Reports 1364

[dblock]

start gradle check

[opensearch-ci-bot]

❌   Gradle Check failure 42864d2cf6e6d59ae4525005403647a709e417e0
Log 1374

Reports 1374

[sharp-pixel]

Looking at the build error, looks like an issue with networking.

> Task :test:fixtures:s3-fixture:composeUp

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-with-session-token_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-repositories-metering_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-other_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-with-ec2_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
Creating 56aa93ed60e3e11b06e2067e06b0dd8c_s3-fixture__s3-fixture-with-ecs_1              ... done

Not sure how my changes could impact this. Or maybe an issue on GitHub's side

[sharp-pixel]

I had a deeper look at what's happening by running locally and it looks like there is something wrong with tests around BWC:

Project :qa:full-cluster-restart declares a dependency from configuration 'opensearch_distro_extracted_testclusters-qa-full-cluster-restart-v1.3.0-1-1.3.0-' to configuration 'expanded-darwin-arm64-tar' which is not declared in the descriptor for project :distribution:bwc:minor.

I spent some time in these tests but I cannot understand where this comes from. Can anyone help here? Thanks

[dblock]

I had a deeper look at what's happening by running locally and it looks like there is something wrong with tests around BWC:

Project :qa:full-cluster-restart declares a dependency from configuration 'opensearch_distro_extracted_testclusters-qa-full-cluster-restart-v1.3.0-1-1.3.0-' to configuration 'expanded-darwin-arm64-tar' which is not declared in the descriptor for project :distribution:bwc:minor.

I spent some time in these tests but I cannot understand where this comes from. Can anyone help here? Thanks

@saratvemulapalli is familiar with this, help?

[opensearch-ci-bot]

❌   Gradle Check failure dbd35c49f41b0141aa756b11a81474cce18da221
Log 1644

Reports 1644

[dblock]

This needs a rebase please.

[dblock]

@sharp-pixel Do you want help with this? Want to finish it up?

[saratvemulapalli]

I had a deeper look at what's happening by running locally and it looks like there is something wrong with tests around BWC:

Project :qa:full-cluster-restart declares a dependency from configuration 'opensearch_distro_extracted_testclusters-qa-full-cluster-restart-v1.3.0-1-1.3.0-' to configuration 'expanded-darwin-arm64-tar' which is not declared in the descriptor for project :distribution:bwc:minor.

I spent some time in these tests but I cannot understand where this comes from. Can anyone help here? Thanks

@saratvemulapalli is familiar with this, help?

I somehow missed this. Will take a look.

[saratvemulapalli]

@sharp-pixel could you rebase this and merge from main?
We could run another gradle check and start debugging.

[sharp-pixel]

@sharp-pixel could you rebase this and merge from main? We could run another gradle check and start debugging.

I rebased my changes. I did more debugging on my side from on an Apple Silicon Mac, and it looks like the BWC process is checking compatibility with previous versions (starting from 7.x) with the darwin-arm4 arch. The problem is that these distributions don't exist. I would need to explicitely say that the first darwin-arm64 version is this one for the backward compatibility checks, but couldn't find out how.

[sharp-pixel]

@sharp-pixel Do you want help with this? Want to finish it up?

Yes, sorry for the delay, was checking more things on my side + holidays

[opensearch-ci-bot]

✅   Gradle Check success d6ce60b
Log 1783

Reports 1783

[sharp-pixel]

Beware that even though all tests pass here, ./gradlew check is failing on an M1 mac due to BWC (see my previous comment about missing previous versions in this new architecture)

[dblock]

Beware that even though all tests pass here, ./gradlew check is failing on an M1 mac due to BWC (see my previous comment about missing previous versions in this new architecture)

Yes, this is a step forward, but we have more work to do to CI/CD for Darwin.

[dblock]

@sharp-pixel We have a parent item, opensearch-project/opensearch-build#38, so if there's something we can do here next to advance the big picture?

[saratvemulapalli]

@sharp-pixel could you rebase this and merge from main? We could run another gradle check and start debugging.

I rebased my changes. I did more debugging on my side from on an Apple Silicon Mac, and it looks like the BWC process is checking compatibility with previous versions (starting from 7.x) with the darwin-arm4 arch. The problem is that these distributions don't exist. I would need to explicitely say that the first darwin-arm64 version is this one for the backward compatibility checks, but couldn't find out how.

Could you open an issue with the failures?
I believe we have 2 steps here:

1. Run the CI in Darwin Arm64
2. Fix BWC for Darwin Arm64

[dblock]

1. Run the CI in Darwin Arm64

#1449

2. Fix BWC for Darwin Arm64

I'll let @sharp-pixel do that and describe it nicely ;)

[radar]

Hello, I was wondering if this could be applied to the 1.3.x branch as well? I would like to be able to install that version specifically on my Mac.

[dblock]

@radar AFAIK there are no more 1.3.x releases except for high sev security patches, https://github.com/orgs/opensearch-project/projects/1. That said you can PR it manually into 1.x and 1.3 and we can probably merge it since it's just build scripts.

[opensearch-trigger-bot]

The backport to 1.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.x 1.x
# Navigate to the new working tree
cd .worktrees/backport-1.x
# Create a new branch
git switch --create backport/backport-1668-to-1.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 1bb88641788abdbfd72b76c9741880cb3b1ae37a
# Push it to GitHub
git push --set-upstream origin backport/backport-1668-to-1.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.x

Then, create a pull request where the base branch is 1.x and the compare/head branch is backport/backport-1668-to-1.x.

[radar]

No problems @dblock, thank you for the headsup.