Skip to content

Commit

Permalink
replacing 'tarball' with 'artefact' where it makes most sense
Browse files Browse the repository at this point in the history
  • Loading branch information
@truib
truib committed Mar 26, 2024
1 parent 2e8e0fd commit e705494
Show file tree
Hide file tree
Showing 5 changed files with 87 additions and 82 deletions.
39 changes: 22 additions & 17 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,9 @@ You can exit the virtual environment simply by running `deactivate`.

### <a name="step4.1"></a>Step 4.1: Installing tools to access S3 bucket

The [`scripts/eessi-upload-to-staging`](https://github.com/EESSI/eessi-bot-software-layer/blob/main/scripts/eessi-upload-to-staging) script uploads a tarball and an associated metadata file to an S3 bucket.
The
[`scripts/eessi-upload-to-staging`](https://github.com/EESSI/eessi-bot-software-layer/blob/main/scripts/eessi-upload-to-staging)
script uploads an artefact and an associated metadata file to an S3 bucket.

It needs two tools for this:
* the `aws` command to actually upload the files;
Expand Down Expand Up @@ -444,14 +446,17 @@ information about the result of the command that was run (can be empty).

The `[deploycfg]` section defines settings for uploading built artefacts (tarballs).
```
tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
artefact_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
```
`tarball_upload_script` provides the location for the script used for uploading built software packages to an S3 bucket.
`artefact_upload_script` provides the location for the script used for uploading built software packages to an S3 bucket.

```
endpoint_url = URL_TO_S3_SERVER
```
`endpoint_url` provides an endpoint (URL) to a server hosting an S3 bucket. The server could be hosted by a commercial cloud provider like AWS or Azure, or running in a private environment, for example, using Minio. The bot uploads tarballs to the bucket which will be periodically scanned by the ingestion procedure at the Stratum 0 server.
`endpoint_url` provides an endpoint (URL) to a server hosting an S3 bucket. The
server could be hosted by a commercial cloud provider like AWS or Azure, or
running in a private environment, for example, using Minio. The bot uploads
artefacts to the bucket which will be periodically scanned by the ingestion procedure at the Stratum 0 server.


```ini
Expand All @@ -466,7 +471,7 @@ bucket_name = {
}
```

`bucket_name` is the name of the bucket used for uploading of tarballs.
`bucket_name` is the name of the bucket used for uploading of artefacts.
The bucket must be available on the default server (`https://${bucket_name}.s3.amazonaws.com`), or the one provided via `endpoint_url`.

`bucket_name` can be specified as a string value to use the same bucket for all target repos, or it can be mapping from target repo id to bucket name.
Expand All @@ -481,7 +486,7 @@ The `upload_policy` defines what policy is used for uploading built artefacts to
|`upload_policy` value|Policy|
|:--------|:--------------------------------|
|`all`|Upload all artefacts (mulitple uploads of the same artefact possible).|
|`latest`|For each build target (prefix in tarball name `eessi-VERSION-{software,init,compat}-OS-ARCH)` only upload the latest built artefact.|
|`latest`|For each build target (prefix in artefact name `eessi-VERSION-{software,init,compat}-OS-ARCH)` only upload the latest built artefact.|
|`once`|Only once upload any built artefact for the build target.|
|`none`|Do not upload any built artefacts.|

Expand All @@ -496,30 +501,30 @@ deployment), or a space delimited list of GitHub accounts.
no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments
```
This defines a message that is added to the status table in a PR comment
corresponding to a job whose tarball should have been uploaded (e.g., after
corresponding to a job whose artefact should have been uploaded (e.g., after
setting the `bot:deploy` label).


```
metadata_prefix = LOCATION_WHERE_METADATA_FILE_GETS_DEPOSITED
tarball_prefix = LOCATION_WHERE_TARBALL_GETS_DEPOSITED
artefact_prefix = LOCATION_WHERE_TARBALL_GETS_DEPOSITED
```

These two settings are used to define where (which directory) in the S3 bucket
(see `bucket_name` above) the metadata file and the tarball will be stored. The
(see `bucket_name` above) the metadata file and the artefact will be stored. The
value `LOCATION...` can be a string value to always use the same 'prefix'
regardless of the target CVMFS repository, or can be a mapping of a target
repository id (see also `repo_target_map` below) to a prefix.

The prefix itself can use some (environment) variables that are set within
the upload script (see `tarball_upload_script` above). Currently those are:
the upload script (see `artefact_upload_script` above). Currently those are:
* `'${github_repository}'` (which would be expanded to the full name of the GitHub
repository, e.g., `EESSI/software-layer`),
* `'${legacy_aws_path}'` (which expands to the legacy/old prefix being used for
storing tarballs/metadata files, the old prefix is
storing artefacts/metadata files, the old prefix is
`EESSI_VERSION/TARBALL_TYPE/OS_TYPE/CPU_ARCHITECTURE/TIMESTAMP/`), _and_
* `'${pull_request_number}'` (which would be expanded to the number of the pull
request from which the tarball originates).
request from which the artefact originates).
Note, it's important to single-quote (`'`) the variables as shown above, because
they may likely not be defined when the bot calls the upload script.

Expand All @@ -529,7 +534,7 @@ The list of supported variables can be shown by running
**Examples:**
```
metadata_prefix = {"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
tarball_prefix = {
artefact_prefix = {
"eessi-pilot-2023.06": "",
"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"
}
Expand Down Expand Up @@ -657,9 +662,9 @@ running_job = job `{job_id}` is running

The `[finished_job_comments]` section sets templates for messages about finished jobs.
```
success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir
success = :grin: SUCCESS tarball `{artefact_name}` ({artefact_size} GiB) in job dir
```
`success` specifies the message for a successful job that produced a tarball.
`success` specifies the message for a successful job that produced an artefact.

```
failure = :cry: FAILURE
Expand Down Expand Up @@ -687,12 +692,12 @@ no_tarball_message = Slurm output lacks message about created tarball.
`no_tarball_message` is used to signal the lack of a message about a created tarball.

```
no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir.
no_matching_tarball = No tarball matching `{artefact_pattern}` found in job dir.
```
`no_matching_tarball` is used to signal a missing tarball.

```
multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected.
multiple_tarballs = Found {num_artefacts} tarballs in job dir - only 1 matching `{artefact_pattern}` expected.
```
`multiple_tarballs` is used to report that multiple tarballs have been found.

Expand Down
20 changes: 10 additions & 10 deletions app.cfg.example
View file
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ no_build_permission_comment = Label `bot:build` has been set by user `{build_lab

[deploycfg]
# script for uploading built software packages
tarball_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging
artefact_upload_script = PATH_TO_EESSI_BOT/scripts/eessi-upload-to-staging

# URL to S3/minio bucket
# if attribute is set, bucket_base will be constructed as follows
Expand Down Expand Up @@ -160,11 +160,11 @@ upload_policy = once
# value can be a space delimited list of GH accounts
deploy_permission =

# template for comment when user who set a label has no permission to trigger deploying tarballs
# template for comment when user who set a label has no permission to trigger deploying artefacts
no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_labeler}`, but this person does not have permission to trigger deployments

# settings for where (directory) in the S3 bucket to store the metadata file and
# the tarball
# the artefact
# - Can be a string value to always use the same 'prefix' regardless of the target
# CVMFS repository, or can be a mapping of a target repository id (see also
# repo_target_map) to a prefix.
Expand All @@ -173,17 +173,17 @@ no_deploy_permission_comment = Label `bot:deploy` has been set by user `{deploy_
# * 'github_repository' (which would be expanded to the full name of the GitHub
# repository, e.g., 'EESSI/software-layer'),
# * 'legacy_aws_path' (which expands to the legacy/old prefix being used for
# storing tarballs/metadata files) and
# storing artefacts/metadata files) and
# * 'pull_request_number' (which would be expanded to the number of the pull
# request from which the tarball originates).
# request from which the artefact originates).
# - The list of supported variables can be shown by running
# `scripts/eessi-upload-to-staging --list-variables`.
# - Examples:
# metadata_prefix = {"eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
# tarball_prefix = {"eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
# artefact_prefix = {"eessi-pilot-2023.06": "", "eessi.io-2023.06": "new/${github_repository}/${pull_request_number}"}
# If left empty, the old/legacy prefix is being used.
metadata_prefix =
tarball_prefix =
artefact_prefix =


[architecturetargets]
Expand Down Expand Up @@ -247,14 +247,14 @@ running_job = job `{job_id}` is running


[finished_job_comments]
success = :grin: SUCCESS tarball `{tarball_name}` ({tarball_size} GiB) in job dir
success = :grin: SUCCESS tarball `{artefact_name}` ({artefact_size} GiB) in job dir
failure = :cry: FAILURE
no_slurm_out = No slurm output `{slurm_out}` in job dir
slurm_out = Found slurm output `{slurm_out}` in job dir
missing_modules = Slurm output lacks message "No missing modules!".
no_tarball_message = Slurm output lacks message about created tarball.
no_matching_tarball = No tarball matching `{tarball_pattern}` found in job dir.
multiple_tarballs = Found {num_tarballs} tarballs in job dir - only 1 matching `{tarball_pattern}` expected.
no_matching_tarball = No tarball matching `{artefact_pattern}` found in job dir.
multiple_tarballs = Found {num_artefacts} tarballs in job dir - only 1 matching `{artefact_pattern}` expected.
job_result_unknown_fmt = <details><summary>:shrug: UNKNOWN _(click triangle for detailed information)_</summary><ul><li>Job results file `{filename}` does not exist in job directory, or parsing it failed.</li><li>No artefacts were found/reported.</li></ul></details>
job_test_unknown_fmt = <details><summary>:shrug: UNKNOWN _(click triangle for detailed information)_</summary><ul><li>Job test file `{filename}` does not exist in job directory, or parsing it failed.</li></ul></details>

Expand Down
40 changes: 20 additions & 20 deletions scripts/eessi-upload-to-staging
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ function check_file_name

function create_metadata_file
{
_tarball=$1
_artefact=$1
_url=$2
_repository=$3
_pull_request_number=$4
Expand All @@ -50,10 +50,10 @@ function create_metadata_file
--arg un $(whoami) \
--arg ip $(curl -s https://checkip.amazonaws.com) \
--arg hn "$(hostname -f)" \
--arg fn "$(basename ${_tarball})" \
--arg sz "$(du -b "${_tarball}" | awk '{print $1}')" \
--arg ct "$(date -r "${_tarball}")" \
--arg sha256 "$(sha256sum "${_tarball}" | awk '{print $1}')" \
--arg fn "$(basename ${_artefact})" \
--arg sz "$(du -b "${_artefact}" | awk '{print $1}')" \
--arg ct "$(date -r "${_artefact}")" \
--arg sha256 "$(sha256sum "${_artefact}" | awk '{print $1}')" \
--arg url "${_url}" \
--arg repo "${_repository}" \
--arg pr "${_pull_request_number}" \
Expand All @@ -70,6 +70,11 @@ function create_metadata_file
function display_help
{
echo "Usage: $0 [OPTIONS] <filenames>" >&2
echo " -a | --artefact-prefix PREFIX - a directory to which the artefact" >&2
echo " shall be uploaded; BASH variable" >&2
echo " expansion will be applied; arg '-l'" >&2
echo " lists variables that are defined at" >&2
echo " the time of expansion" >&2
echo " -e | --endpoint-url URL - endpoint url (needed for non AWS S3)" >&2
echo " -h | --help - display this usage information" >&2
echo " -i | --pr-comment-id - identifier of a PR comment; may be" >&2
Expand All @@ -88,11 +93,6 @@ function display_help
echo " link the upload to a PR" >&2
echo " -r | --repository FULL_NAME - a repository name ACCOUNT/REPONAME;" >&2
echo " used to link the upload to a PR" >&2
echo " -t | --tarball-prefix PREFIX - a directory to which the tarball" >&2
echo " shall be uploaded; BASH variable" >&2
echo " expansion will be applied; arg '-l'" >&2
echo " lists variables that are defined at" >&2
echo " the time of expansion" >&2
}

if [[ $# -lt 1 ]]; then
Expand Down Expand Up @@ -123,14 +123,18 @@ github_repository="EESSI/software-layer"

# provided via options in the bot's config file app.cfg and/or command line argument
metadata_prefix=
tarball_prefix=
artefact_prefix=

# other variables
legacy_aws_path=
variables="github_repository legacy_aws_path pull_request_number"

while [[ $# -gt 0 ]]; do
case $1 in
-a|--artefact-prefix)
artefact_prefix="$2"
shift 2
;;
-e|--endpoint-url)
endpoint_url="$2"
shift 2
Expand Down Expand Up @@ -167,10 +171,6 @@ while [[ $# -gt 0 ]]; do
github_repository="$2"
shift 2
;;
-t|--tarball-prefix)
tarball_prefix="$2"
shift 2
;;
-*|--*)
echo "Error: Unknown option: $1" >&2
exit 1
Expand Down Expand Up @@ -204,17 +204,17 @@ for file in "$*"; do
basefile=$( basename ${file} )
if check_file_name ${basefile}; then
if tar tf "${file}" | head -n1 > /dev/null; then
# 'legacy_aws_path' might be used in tarball_prefix or metadata_prefix
# 'legacy_aws_path' might be used in artefact_prefix or metadata_prefix
# its purpose is to support the old/legacy method to derive the location
# where to store the tarball and metadata file
# where to store the artefact and metadata file
export legacy_aws_path=$(basename ${file} | tr -s '-' '/' \
| perl -pe 's/^eessi.//;' | perl -pe 's/\.tar\.gz$//;' )
if [ -z ${tarball_prefix} ]; then
if [ -z ${artefact_prefix} ]; then
aws_path=${legacy_aws_path}
else
export pull_request_number
export github_repository
aws_path=$(envsubst <<< "${tarball_prefix}")
aws_path=$(envsubst <<< "${artefact_prefix}")
fi
aws_file=$(basename ${file})
echo "Creating metadata file"
Expand All @@ -233,7 +233,7 @@ for file in "$*"; do
cat ${metadata_file}

echo Uploading to "${url}"
echo " store tarball at ${aws_path}/${aws_file}"
echo " store artefact at ${aws_path}/${aws_file}"
upload_to_staging_bucket \
"${file}" \
"${bucket_name}" \
Expand Down
Loading

0 comments on commit e705494

Please sign in to comment.