-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Build updates and fix for docker mount issue in v1.9 #643
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,128 +1,36 @@ | ||
#!/usr/bin/env bash | ||
|
||
# Licensed to Elasticsearch B.V. under one or more contributor | ||
# license agreements. See the NOTICE file distributed with | ||
# this work for additional information regarding copyright | ||
# ownership. Elasticsearch B.V. licenses this file to you under | ||
# the Apache License, Version 2.0 (the "License"); you may | ||
# not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, | ||
# software distributed under the License is distributed on an | ||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
# KIND, either express or implied. See the License for the | ||
# specific language governing permissions and limitations | ||
# under the License. | ||
function setup { | ||
export BENCHMARK_HOME=$GITHUB_WORKSPACE | ||
|
||
# fail this script immediately if any command fails with a non-zero exit code | ||
set -e | ||
# fail on pipeline errors, e.g. when grepping | ||
set -o pipefail | ||
# fail on any unset environment variables | ||
set -u | ||
export THESPLOG_FILE=$BENCHMARK_HOME/.benchmark/logs/actor-system-internal.log | ||
export THESPLOG_FILE_MAXSIZE=204800 # default is 50 KiB | ||
export THESPLOG_THRESHOLD=INFO # default log level is WARNING | ||
|
||
function update_pyenv { | ||
# need to have the latest pyenv version to ensure latest patch releases are installable | ||
cd $HOME/.pyenv/plugins/python-build/../.. && git pull origin master --rebase && cd - | ||
} | ||
|
||
function build { | ||
export THESPLOG_FILE="${THESPLOG_FILE:-${BENCHMARK_HOME}/.benchmark/logs/actor-system-internal.log}" | ||
# this value is in bytes, the default is 50kB. We increase it to 200kiB. | ||
export THESPLOG_FILE_MAXSIZE=${THESPLOG_FILE_MAXSIZE:-204800} | ||
# adjust the default log level from WARNING | ||
export THESPLOG_THRESHOLD="INFO" | ||
|
||
# turn nounset off because some of the following commands fail if nounset is turned on | ||
set +u | ||
|
||
export PATH="$HOME/.pyenv/bin:$PATH" | ||
export TERM=dumb | ||
export LC_ALL=en_US.UTF-8 | ||
update_pyenv | ||
eval "$(pyenv init -)" | ||
# ensure pyenv shims are added to PATH, see https://github.com/pyenv/pyenv/issues/1906 | ||
eval "$(pyenv init --path)" | ||
eval "$(pyenv virtualenv-init -)" | ||
|
||
make prereq | ||
make install | ||
make precommit | ||
make test | ||
# Init pyenv. | ||
PATH=$HOME/.pyenv/shims:$PATH:$HOME/.pyenv/bin | ||
} | ||
|
||
function build_it { | ||
export THESPLOG_FILE="${THESPLOG_FILE:-${BENCHMARK_HOME}/.benchmark/logs/actor-system-internal.log}" | ||
# this value is in bytes, the default is 50kB. We increase it to 200kiB. | ||
export THESPLOG_FILE_MAXSIZE=${THESPLOG_FILE_MAXSIZE:-204800} | ||
# adjust the default log level from WARNING | ||
export THESPLOG_THRESHOLD="INFO" | ||
|
||
# turn nounset off because some of the following commands fail if nounset is turned on | ||
set +u | ||
|
||
export PATH="$HOME/.pyenv/bin:$PATH" | ||
export TERM=dumb | ||
export LC_ALL=en_US.UTF-8 | ||
export BENCHMARK_HOME="$GITHUB_WORKSPACE" | ||
function build_and_unit_test { | ||
setup | ||
|
||
update_pyenv | ||
eval "$(pyenv init -)" | ||
# ensure pyenv shims are added to PATH, see https://github.com/pyenv/pyenv/issues/1906 | ||
eval "$(pyenv init --path)" | ||
eval "$(pyenv virtualenv-init -)" | ||
set -e | ||
make develop | ||
make lint | ||
make test | ||
} | ||
|
||
python3_version=`python3 --version` | ||
echo "Python3 version is ... $python3_version" | ||
function run_it { | ||
setup | ||
|
||
python3 -m pip install opensearch-benchmark | ||
docker pull ubuntu/squid:latest | ||
|
||
make prereq | ||
make install | ||
make precommit | ||
|
||
# make it38, it39, etc. | ||
# make it38, it39, etc. so they run as concurrent GHA jobs | ||
make "it${1//./}" | ||
} | ||
|
||
function license-scan { | ||
# turn nounset off because some of the following commands fail if nounset is turned on | ||
set +u | ||
|
||
export PATH="$HOME/.pyenv/bin:$PATH" | ||
eval "$(pyenv init -)" | ||
# ensure pyenv shims are added to PATH, see https://github.com/pyenv/pyenv/issues/1906 | ||
eval "$(pyenv init --path)" | ||
eval "$(pyenv virtualenv-init -)" | ||
|
||
make prereq | ||
# only install depdencies that are needed by end users | ||
make install-user | ||
fossa analyze | ||
} | ||
|
||
function archive { | ||
# Treat unset env variables as an error, but only in this function as there are other functions that allow unset variables | ||
set -u | ||
|
||
# this will only be done if the build number variable is present | ||
BENCHMARK_DIR=${BENCHMARK_HOME}/.benchmark | ||
if [[ -d ${BENCHMARK_DIR} ]]; then | ||
find ${BENCHMARK_DIR} -name "*.log" -printf "%P\\0" | tar -cvjf ${BENCHMARK_DIR}/${BUILD_NUMBER}.tar.bz2 -C ${BENCHMARK_DIR} --transform "s,^,ci-${BUILD_NUMBER}/," --null -T - | ||
else | ||
echo "Benchmark directory [${BENCHMARK_DIR}] not present. Ensure the BENCHMARK_DIR environment variable is correct" | ||
exit 1 | ||
fi | ||
} | ||
$@ | ||
|
||
if declare -F "$1" > /dev/null; then | ||
"$@" | ||
exit | ||
else | ||
echo "Please specify a function to run" | ||
exit 1 | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -23,16 +23,29 @@ This document will walk you through on what's needed to start contributing code | |
|
||
### Prerequisites | ||
|
||
- **Pyenv** : Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding. | ||
For more details please refer to the [PyEnv installation instructions](https://github.com/pyenv/pyenv#installation). | ||
- **pyenv**: Install `pyenv` and follow the instructions in the output of `pyenv init` to set up your shell and restart it before proceeding. | ||
For more details please refer to the [pyenv installation instructions](https://github.com/pyenv/pyenv#installation). | ||
|
||
**Optional Step:** For Debian-based systems, install the following modules to continue with the next steps: | ||
`pyenv` requires that the C compiler and development libraries be installed, so that the specified Python versions can be build from source. The installation instructions vary from platform to platform. | ||
|
||
For Debian-based systems, install the following modules to continue with the next steps: | ||
``` | ||
sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev \ | ||
libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ | ||
xz-utils tk-dev libffi-dev liblzma-dev git | ||
``` | ||
|
||
For Amazon Linux 2023, run the following command: | ||
``` | ||
sudo yum -y install gcc openssl-devel bzip2-devel libffi-devel ncurses-devel sqlite-devel readline-devel zlib-devel xz-devel | ||
``` | ||
|
||
On the Mac platform, XCode needs to be installed as well as some additional required libraries: | ||
``` | ||
xcode-select --install | ||
brew install pyenv jq zlib xz | ||
``` | ||
Comment on lines
+38
to
+47
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This was needed. Thanks for adding this! |
||
|
||
- **JDK**: Although OSB is a Python application, it optionally builds and provisions OpenSearch clusters. JDK version 17 is used to build the current version of OpenSearch. Please refer to the [build setup requirements](https://github.com/opensearch-project/OpenSearch/blob/ca564fd04f5059cf9e3ce8aba442575afb3d99f1/DEVELOPER_GUIDE.md#install-prerequisites). | ||
Note that the `javadoc` executable should be available in the JDK installation. An earlier version of the JDK can be used, but not all the integration tests will pass. | ||
|
||
|
@@ -54,15 +67,8 @@ For those working on WSL2, it is recommended to clone the repository and set up | |
After you git cloned the forked copy of OpenSearch Benchmark, use the following command-line instructions to set up OpenSearch Benchmark for development: | ||
``` | ||
cd opensearch-benchmark | ||
make prereq | ||
make install | ||
``` | ||
|
||
NOTE: `make prereq` produces the following message. | ||
``` | ||
IMPORTANT: please add `eval "$(pyenv init -)"` to your bash profile and restart your terminal before proceeding any further. | ||
make develop | ||
``` | ||
This line is commonly thought of as an error message but rather it's just a warning. Unless you haven't already added `eval "$(pyenv init -)"` to your bash profile and restarted your terminal, then feel free to proceed forward. This eval statement is necessary in the startup configuration as it allows Pyenv to manage python versions by adding python shims to your path. If you experience any issues, please see https://github.com/pyenv/pyenv. | ||
|
||
Depending on the platform and shell you have, use the following command to activate the virtual environment: | ||
|
||
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we insert the OpenSearch license?