Skip to content

prudhvigodithi/opensearch-build

Folders and files

NameName
Last commit message
Last commit date
Sep 3, 2024
Aug 7, 2023
Jun 4, 2024
Aug 20, 2024
Aug 19, 2024
Jul 7, 2022
Sep 3, 2024
Aug 20, 2024
Sep 28, 2022
Sep 3, 2024
May 12, 2023
Aug 27, 2024
Aug 9, 2024
Aug 28, 2024
Aug 28, 2024
Aug 29, 2024
May 12, 2023
Jan 24, 2022
Jan 12, 2024
Nov 30, 2021
Oct 24, 2023
Mar 27, 2024
May 17, 2022
Jun 6, 2023
Dec 9, 2021
Mar 10, 2022
Jul 29, 2021
Jul 29, 2021
Jul 29, 2021
Mar 27, 2024
Jul 29, 2021
May 29, 2024
Aug 23, 2021
Jul 9, 2024
Jul 9, 2024
Jul 18, 2024
May 16, 2024
Aug 15, 2024
Sep 28, 2022
Jun 11, 2024
Sep 28, 2022
Sep 28, 2022
Sep 28, 2022
Nov 11, 2021
Jul 7, 2022
Jul 7, 2022
Sep 28, 2022
May 16, 2023
Dec 2, 2021
Oct 6, 2021
Sep 28, 2022
Aug 1, 2023
Apr 22, 2024
Nov 11, 2021
Sep 28, 2022
Aug 6, 2024
Nov 17, 2022
Dec 9, 2021

Repository files navigation

OpenSearch Build

python groovy manifests codecov

Releasing OpenSearch

Please refer to the release process document for detailed information on how to release the OpenSearch and OpenSearch Dashboards software.

Releases and Versions

The OpenSearch project releases as versioned distributions of OpenSearch, OpenSearch Dashboards, and the OpenSearch plugins. It follows semantic versioning. Software, such as Data Prepper, clients, and the Logstash output plugin, are versioned independently of the OpenSearch Project. They also may have independent releases from the main project distributions. The OpenSearch Project may also release software under alpha, beta, release candidate, and generally available labels. The definition of when to use these labels is derived from the Wikipedia page on Software release lifecycle. Below is the definition of when to use each label.

Release labels:

  • Alpha - The code is released with instructions to build. Built distributions of the software may not be available. Some features many not be complete. Additional testing and development work is planned. Distributions will be postfixed with -alphaX where "X" is the number of the alpha version (e.g., "2.0-alpha1").
  • Beta - Built distributions of the software are available. All features are completed. Additional testing and development work is planned. Distributions will be postfixed with -betaX where "X" is the number of the beta version (e.g., "2.0.0-beta1").
  • Release Candidate - Built distributions of the software are available. All features are completed. Code is tested and minimal validation remains. At this stage the software is potentially stable and will release unless significant bugs emerge. Distributions will be postfixed with -rcX where "X" is the number of the release candidate version (e.g., "2.0.0-rc1").
  • Generally Available - Built distributions of the software are available. All features are completed and documented. All testing is completed. Distributions for generally available versions are not postfixed with an additional label (e.g., "2.0.0").

Onboarding a New Plugin

Plugin owners can follow the Onboarding Process to onboard their plugins to the release process.

Building and Testing an OpenSearch Distribution

The distribution workflow builds a complete OpenSearch and OpenSearch Dashboards distribution from source. You can currently build 1.0, 1.1, 1.1-SNAPSHOT and 1.2 versions. This system performs a top-down build of all components required for a specific OpenSearch and OpenSearch Dashboards release, then assembles a distribution. The input to the system is a manifest that defines the order in which components should be built. All manifests for our current releases are here.

Building from Source

./build.sh manifests/1.3.0/opensearch-1.3.0.yml 

This builds OpenSearch 1.3.0 from source, placing the output into ./builds/opensearch.

See build workflow for more information.

Assembling a Distribution

./assemble.sh builds/opensearch/manifest.yml

The assembling step takes output from the build step, installs plugins, and assembles a full distribution into the dist folder.

See assemble workflow for more information.

Building Patches

A patch release contains output from previous versions mixed with new source code. Manifests can mix such references. See opensearch-1.3.1.yml for an example.

OpenSearch is often released with changes in opensearch-min, and no changes to plugins other than a version bump. This can be performed by a solo engineer following a cookbook. See also opensearch-build#1375 which aims to automate incrementing versions for the next development iteration.

Min Snapshots

Snapshots for OpenSearch core/min can be downloaded and used in CI's, local development, etc using below links:

Linux:

https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-linux-x64-latest.tar.gz

macOS:

https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-darwin-x64-latest.tar.gz

Windows:

https://artifacts.opensearch.org/snapshots/core/opensearch/<version>-SNAPSHOT/opensearch-min-<version>-SNAPSHOT-windows-x64-latest.zip

CI/CD Environment

We build, assemble, and test our artifacts on docker containers. We provide docker files in docker/ci folder, and images on staging docker hub repositories. All Jenkins pipelines can be found in jenkins. Jenkins itself is in the process of being made public and its CDK open-sourced.

See jenkins and docker for more information.

Build Numbers

The distribution URL and the build output manifest include a Jenkins auto-incremented build number. For example, the manifest from OpenSearch build 5905 contains the following.

build:
  name: OpenSearch
  version: 2.2.0
  platform: linux
  architecture: x64
  distribution: rpm
  id: '5905'

Latest Distribution URL

Use the latest keyword in the URL to obtain the latest build for a given version. For example https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/rpm/dist/opensearch/manifest.yml redirects to build 5905 at the time of writing this.

The latest keyword is resolved to a specific build number by checking the index.json file. This file has contents such as this.

{"latest":"5905"}

The file is updated when a distribution build job is completed for the given product and version (or is created when such distribution job succeeds for the first time). Since one distribution build job consists of multiple stages for different combinations of distribution type, platform and architecture, the index.json is only modified once all stages succeed. With this said, the latest URL only works when the distribution build job succeeds at least once for the given product and version.

The resolution logic is implemented in the CloudFront URL rewriter. The TTL (time to live) is set to 5 mins which means that the latest URL may need up to 5 mins to get new contents after index.json is updated.

All the artifacts accessible through the regular distribution URL can be accessed by the latest URL. This includes both OpenSearch Core, OpenSearch Dashboards Core and their plugins.

For example, you can download the latest .tar.gz distribution build of OpenSearch 2.2.0 directly at https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.2.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.2.0-linux-x64.tar.gz, without having to first download and parse the complete build manifest.

For plugin artifacts, you can also use the latest keyword to get the latest plugin artifacts with a known version. E.g. in order to get performance-analyzer x64 tarball artifacts for 2.1.0, you can obtain it with link https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/latest/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip, which will redirect you to https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.1.0/5757/linux/x64/tar/builds/opensearch/plugins/opensearch-performance-analyzer-2.1.0.0.zip.

For bundled artifacts, here are some examples for LINUX and Windows:

  • Linux Tar: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/linux/x64/tar/dist/opensearch/opensearch-2.4.0-linux-x64.tar.gz
  • Windows Zip: https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.4.0/latest/windows/x64/zip/dist/opensearch/opensearch-2.4.0-windows-x64.zip

Testing the Distribution

Tests the OpenSearch distribution, including integration, backwards-compatibility and performance tests.

./test.sh <test-type> <test-manifest-path> <path>

See src/test_workflow for more information.

Checking Release Notes

Workflow to check if the release notes exist or not and shows the latest commit for OpenSearch and Dashboard distributions.

To run:

./release_notes.sh check manifests/2.2.0/opensearch-2.2.0.yml --date 2022-07-26

See src/release_notes_workflow for more information.

Signing Artifacts

For all types of signing within OpenSearch project we use opensearch-signer-client (in progress of being open-sourced) which is a wrapper around internal signing system and is only available for authenticated users. The input requires a path to the build manifest or directory containing all the artifacts or a single artifact.

Usage:

./sign.sh builds/opensearch/manifest.yml

The tool currently supports following platforms for signing.

PGP

Anything can be signed using PGP signing eg: tarball, any type of file, etc. A .sig file will be returned containing the signature. OpenSearch and OpenSearch dashboards distributions, components such as data prepper, etc. as well as maven artifacts are signed using PGP signing. See this page for how to verify signatures.

Windows

Windows signing can be used to sign windows executables such as .msi, .msp, .msm, .cab, .dll, .exe, .appx, .appxbundle, .msix, .msixbundle, .sys, .vxd, .ps1, .psm1, and any PE file that is supported by Signtool.exe. Various windows artifacts such as SQL OBDC, opensearch-cli, etc are signed using this method. Windows code signing uses EV (Extended Validated) code signing certificates.

Types of signing/Details Digest Cipher Key Size
PGP SHA1 AES-128 2048
Windows SHA256 RSA
RPM SHA512 RSA 4096

Signing RPM artifacts

RPM artifacts are signed via a legacy shell script which uses a macros template. See this commit for more information and this issue to add RPM artifact signing functionality to the above signing system. Currently we are only signing OpenSearch and OpenSearch dashboards RPM distributions using this method.

See src/sign_workflow for more information.

Making a Release

Releasing for Linux and Windows

The Linux / Windows release is managed by a team at Amazon following this release template (e.g. opensearch-build#2649).

Releasing for FreeBSD

The FreeBSD ports and packages for OpenSearch are managed by a community OpenSearch Team at FreeBSD. When a new release is rolled out, this team will update the port and commit it to the FreeBSD ports tree. Anybody is welcome to help the team by providing patches for upgrading the ports following the FreeBSD Porter's Handbook instructions.

Releasing for macOS

At this moment there's no official macOS distribution. However, this project does support building and assembling OpenSearch for macOS. See opensearch-build#37 and #38 for more details.

Utilities

Checking Out Source

The checkout workflow checks out source code for a given manifest for further examination.

./checkout.sh manifests/1.3.0/opensearch-1.3.0.yml

See src/checkout_workflow for more information.

Cross-Platform Builds

You can perform cross-platform builds. For example, build and assemble a Windows distribution on macOS.

export JAVA_HOME=$(/usr/libexec/java_home) # required by OpenSearch install-plugin during assemble
./build.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot --platform windows
./assemble.sh builds/opensearch/manifest.yml

This will produce dist/opensearch-1.3.0-SNAPSHOT-windows-x64.zip on Linux and macOS.

Sanity Checking the Bundle

This workflow runs sanity checks on every component present in the bundle, executed as part of the manifests workflow in this repository. It ensures that the component GitHub repositories are correct and versions in those components match the OpenSearch version.

The following example sanity-checks components in the OpenSearch 1.3.0 manifest.

./ci.sh manifests/1.3.0/opensearch-1.3.0.yml --snapshot

See src/ci_workflow for more information.

Auto-Generating Manifests

The manifests workflow reacts to version increments in OpenSearch and its components by extracting Gradle properties from project branches. When a new version is found, a new input manifest is added to manifests, and a pull request is opened (e.g. opensearch-build#491).

Show information about existing manifests.

./manifests.sh list

Check for updates and create any new manifests.

./manifests.sh update

See src/manifests_workflow for more information.

Deploying Infrastructure

This project uses jenkins as the build infrastructure for building, testing and releasing the artifacts. The infrastructure is deployed using CDK and code can be found in opensearch-ci repository.

Contributing

See developer guide and how to contribute to this project.

Getting Help

If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository.

For more information, see project website and documentation. If you need help and are unsure where to open an issue, try forums.

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.

Security

If you discover a potential security issue in this project we ask that you notify OpenSearch Security directly via email to [email protected]. Please do not create a public GitHub issue.

License

This project is licensed under the Apache v2.0 License.

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.

About

🧰 OpenSearch / OpenSearch-Dashboards Build Systems

Resources

License

Code of conduct

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 68.5%
  • Shell 10.1%
  • Groovy 9.6%
  • Dockerfile 6.5%
  • HTML 3.9%
  • PowerShell 1.1%
  • Other 0.3%