Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

config file provided before command #458

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

config file provided before command #458

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
503a1ec
config can be provided before command
pahatz Oct 16, 2024
b601045
fix tests
pahatz Oct 21, 2024
072234b
config comes only before command
pahatz Oct 22, 2024
a471ce6
Merge branch 'main' into feature/config-before-command
pahatz Oct 22, 2024
4b57b91
Apply suggestions from code review
pahatz Oct 23, 2024
13a028f
concat strings differently
pahatz Oct 25, 2024
7cb9cf0
Merge branch 'main' into feature/config-before-command
pahatz Oct 28, 2024
3c7e86a
Apply suggestions from code review
pahatz Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions .github/integration/tests/tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,11 +49,11 @@ files="data_file.c4gh"
check_encypted_file $files

# Upload a specific file and check it
./sda-cli upload -config testing/s3cmd.conf data_file.c4gh
./sda-cli -config testing/s3cmd.conf upload data_file.c4gh
check_uploaded_file "test/$user/data_file.c4gh" data_file.c4gh


if ./sda-cli list -config testing/s3cmd.conf | grep -q 'data_file.c4gh'
if ./sda-cli -config testing/s3cmd.conf list | grep -q 'data_file.c4gh'
then
echo "Listed file from s3 backend"
else
Expand All @@ -62,7 +62,7 @@ else
fi

# Try to upload a file twice with the --force-overwrite flag
output=$(./sda-cli upload -config testing/s3cmd.conf --force-overwrite data_file.c4gh)
output=$(./sda-cli -config testing/s3cmd.conf upload --force-overwrite data_file.c4gh)

# Create and encrypt multiple files in a folder

Expand All @@ -76,7 +76,7 @@ check_encypted_file "data_files_enc/data_file.c4gh data_files_enc/data_file1.c4g
for k in data_file.c4gh data_file1.c4gh
do
# Upload and check file
./sda-cli upload -config testing/s3cmd.conf --force-overwrite "data_files_enc/$k"
./sda-cli -config testing/s3cmd.conf upload --force-overwrite "data_files_enc/$k"
check_uploaded_file "test/$user/$k" "$k"
done

Expand All @@ -90,7 +90,7 @@ cp data_files_enc/data_file.c4gh data_files_enc/dir1/dir2/data_file.c4gh
cp data_files_enc/data_file.c4gh data_files_enc/dir1/dir2/data_file2.c4gh

# Upload a folder recursively and a single file
./sda-cli upload -config testing/s3cmd.conf -r data_files_enc/dir1 data_files_enc/data_file3.c4gh
./sda-cli -config testing/s3cmd.conf upload -r data_files_enc/dir1 data_files_enc/data_file3.c4gh

# Check that files were uploaded with the local path prefix `data_files_enc` stripped from the target path
for k in dir1/data_file.c4gh dir1/dir2/data_file.c4gh dir1/dir2/data_file2.c4gh data_file3.c4gh
Expand All @@ -102,10 +102,10 @@ done

# Upload a folder recursively and a single file in a specified upload folder
uploadDir="testfolder"
./sda-cli upload -config testing/s3cmd.conf -targetDir "$uploadDir" -r data_files_enc/dir1 data_files_enc/data_file3.c4gh
./sda-cli -config testing/s3cmd.conf upload -targetDir "$uploadDir" -r data_files_enc/dir1 data_files_enc/data_file3.c4gh

# Do it again to test that we can pass -targetDir at the end
./sda-cli upload --force-overwrite -config testing/s3cmd.conf -r data_files_enc/dir1 data_files_enc/data_file3.c4gh -targetDir "$uploadDir"
./sda-cli -config testing/s3cmd.conf upload --force-overwrite -r data_files_enc/dir1 data_files_enc/data_file3.c4gh -targetDir "$uploadDir"

# Check that files were uploaded with the local path prefix `data_files_enc` stripped from the
# target path and into the specified upload folder
Expand All @@ -117,7 +117,7 @@ done
# Upload all contents of a folder recursively to a specified upload folder

uploadDir="testfolder2"
./sda-cli upload -config testing/s3cmd.conf -targetDir "$uploadDir" -r data_files_enc/dir1/.
./sda-cli -config testing/s3cmd.conf upload -targetDir "$uploadDir" -r data_files_enc/dir1/.

# Check that files were uploaded with the local path prefix `data_files_enc/dir1` stripped from the
# target path and into the specified upload folder
Expand All @@ -132,7 +132,7 @@ mkdir data_files_unenc && mkdir data_files_unenc/dir1
cp data_file data_files_unenc/. && cp data_file data_files_unenc/dir1/data_file1

uploadDir="testEncryptUpload"
./sda-cli upload -config testing/s3cmd.conf -encrypt-with-key sda_key.pub.pem -r data_files_unenc -targetDir "$uploadDir"
./sda-cli -config testing/s3cmd.conf upload -encrypt-with-key sda_key.pub.pem -r data_files_unenc -targetDir "$uploadDir"

check_encypted_file "data_files_unenc/data_file.c4gh" "data_files_unenc/dir1/data_file1.c4gh"

Expand All @@ -150,7 +150,7 @@ else
fi

# Download file by using the sda download service
./sda-cli download -config testing/s3cmd-download.conf -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir test-download main/subfolder/dummy_data.c4gh
./sda-cli -config testing/s3cmd-download.conf download -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir test-download main/subfolder/dummy_data.c4gh

# Check if file exists in the path
if [ ! -f "test-download/main/subfolder/dummy_data" ]; then
Expand All @@ -168,7 +168,7 @@ fi
rm -r test-download

# Check listing files in a dataset
output=$(./sda-cli list -config testing/s3cmd-download.conf -dataset https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080)
output=$(./sda-cli -config testing/s3cmd-download.conf list -dataset https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080)
expected="FileIDSizePathurn:neic:001-0011.0MB5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8_elixir-europe.org/main/subfolder/dummy_data.c4ghurn:neic:001-0021.0MB5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8_elixir-europe.org/main/subfolder2/dummy_data2.c4ghurn:neic:001-0031.0MB5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8_elixir-europe.org/main/subfolder2/random/dummy_data3.c4ghDatasetsize:3.1MB"
if [[ "${output//[$' \t\n\r']/}" == "${expected//[$' \t\n\r']/}" ]]; then
echo "Successfully listed files in dataset"
Expand All @@ -178,7 +178,7 @@ else
fi

# Check listing datasets
output=$(./sda-cli list -config testing/s3cmd-download.conf --datasets -url http://localhost:8080)
output=$(./sda-cli -config testing/s3cmd-download.conf list --datasets -url http://localhost:8080)
expected="https://doi.example/ty009.sfrrss/600.45asasga"
if [[ $output == *"$expected"* ]]; then
echo "Successfully listed datasets"
Expand All @@ -188,7 +188,7 @@ else
fi

# Download whole dataset by using the sda-download feature
./sda-cli download -config testing/s3cmd-download.conf -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-dataset --dataset
./sda-cli -config testing/s3cmd-download.conf download -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-dataset --dataset

filepaths="download-dataset/main/subfolder/dummy_data download-dataset/main/subfolder2/dummy_data2 download-dataset/main/subfolder2/random/dummy_data3"

Expand All @@ -210,7 +210,7 @@ else
echo "Failed to create a user key pair for downloading encrypted files"
exit 1
fi
./sda-cli download -pubkey user_key.pub.pem -config testing/s3cmd-download.conf -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir test-download main/subfolder/dummy_data.c4gh
./sda-cli -config testing/s3cmd-download.conf download -pubkey user_key.pub.pem -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir test-download main/subfolder/dummy_data.c4gh

# check if file exists in the path
if [ ! -f "test-download/main/subfolder/dummy_data.c4gh" ]; then
Expand Down Expand Up @@ -315,7 +315,7 @@ rm -r test-download

# Download recursively a folder
echo "Downloading content of folder"
./sda-cli download -config testing/s3cmd-download.conf -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-folder --recursive main/subfolder2
./sda-cli -config testing/s3cmd-download.conf download -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-folder --recursive main/subfolder2

folderpaths="download-folder/main/subfolder2/dummy_data2 download-folder/main/subfolder2/random/dummy_data3"

Expand All @@ -330,7 +330,7 @@ done
rm -r download-folder

# Download file by providing the file id
./sda-cli download -config testing/s3cmd-download.conf -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-fileid urn:neic:001-001
./sda-cli -config testing/s3cmd-download.conf download -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-fileid urn:neic:001-001

# Check if file exists in the path
if [ ! -f "download-fileid/main/subfolder/dummy_data" ]; then
Expand All @@ -349,7 +349,7 @@ rm -r download-fileid

# Download the file paths content of a text file
echo "Downloading content of a text file"
./sda-cli download -config testing/s3cmd-download.conf -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-from-file --from-file testing/file-list.txt
./sda-cli -config testing/s3cmd-download.conf download -dataset-id https://doi.example/ty009.sfrrss/600.45asasga -url http://localhost:8080 -outdir download-from-file --from-file testing/file-list.txt

# Check if the content of the text file has been downloaded
content_paths="download-from-file/main/subfolder/dummy_data download-from-file/main/subfolder2/dummy_data2"
Expand Down
38 changes: 19 additions & 19 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,13 +107,13 @@ The Priority order for the access token is:
Now that the configuration file is downloaded, the file(s) can be uploaded to the archive using the binary file created in the first step of this guide. To upload a specific file, use the following command:

```bash
./sda-cli upload -config <configuration_file> <encrypted_file_to_upload>
./sda-cli -config <configuration_file> upload <encrypted_file_to_upload>
```

where `<configuration_file>` the file downloaded in the previous step and `<encrypted_file_to_upload>` a file encrypted in the earlier steps. The tool also allows for uploading multiple files at once, by listing them separated with space like:

```bash
./sda-cli upload -config <configuration_file> <encrypted_file_1_to_upload> <encrypted_file_2_to_upload>
./sda-cli -config <configuration_file> upload <encrypted_file_1_to_upload> <encrypted_file_2_to_upload>
```

Note that the files will be uploaded in the base folder of the user.
Expand All @@ -123,15 +123,15 @@ Note that the files will be uploaded in the base folder of the user.
One can also upload entire directories recursively, i.e. including all contained files and folders while keeping the local folder structure. This can be achieved with the `-r` flag, e.g. running:

```bash
./sda-cli upload -config <configuration_file> -r <folder_to_upload>
./sda-cli -config <configuration_file> upload -r <folder_to_upload>
```

will upload `<folder_to_upload>` as is, i.e. with the same inner folder and file structure as the local one, to the archive.

It is also possible to specify multiple directories and files for upload with the same command. For example,

```bash
./sda-cli upload -config <configuration_file> -r <encrypted_file_1_to_upload> <encrypted_file_2_to_upload> <folder_1_to_upload> <folder_2_to_upload>
./sda-cli -config <configuration_file> upload -r <encrypted_file_1_to_upload> <encrypted_file_2_to_upload> <folder_1_to_upload> <folder_2_to_upload>
```

However, if `-r` is omitted in the above, any folders will be skipped during upload.
Expand All @@ -141,15 +141,15 @@ However, if `-r` is omitted in the above, any folders will be skipped during upl
The user can specify a different path for uploading files/folders with the `-targetDir` flag followed by the name of the folder. For example, the command:

```bash
./sda-cli upload -config <configuration_file> -r <encrypted_file_1_to_upload> <folder_1_to_upload> -targetDir <upload_folder>
./sda-cli -config <configuration_file> upload -r <encrypted_file_1_to_upload> <folder_1_to_upload> -targetDir <upload_folder>
```

will create `<upload_folder>` under the user's base folder with contents `<upload_folder>/<encrypted_file_1_to_upload>` and `<upload_folder>/<folder_1_to_upload>`. Note that the given `<upload_folder>` may well be a folder path, e.g. `<folder1/folder2>`, and in this case `<encrypted_file_1_to_upload>` will be uploaded to `folder1/folder2/<encrypted_file_1_to_upload>`.

As a side note it is possible to include all the contents of a directory with `/.`, for example,

```bash
./sda-cli upload -config <configuration_file> -r <folder_to_upload>/. -targetDir <new_folder_name>
./sda-cli -config <configuration_file> upload -r <folder_to_upload>/. -targetDir <new_folder_name>
```

will upload all contents of `<folder_to_upload>` to `<new_folder_name>` recursively, effectively renaming `<folder_to_upload>` upon upload to the archive.
Expand All @@ -159,15 +159,15 @@ will upload all contents of `<folder_to_upload>` to `<new_folder_name>` recursiv
It is possible to combine the encryption and upload steps with the use of the flag `--encrypt-with-key` followed by the path of the crypt4gh public key to be used for encryption. In this case, the input list of file arguments can only contain _unencrypted_ source files. For example the following,

```bash
./sda-cli upload -config <configuration_file> --encrypt-with-key <public_key> <unencrypted_file_to_upload>
./sda-cli -config <configuration_file> upload --encrypt-with-key <public_key> <unencrypted_file_to_upload>
```

will encrypt `<unencrypted_file_to_upload>` using `<public_key>` as public key and upload the created `<file_to_upload.c4gh>` in the base folder of the user.

Encrypt on upload can be combined with any of the flags above. For example,

```bash
./sda-cli upload -config <configuration_file> --encrypt-with-key <public_key> -r <folder_to_upload_with_unencrypted_data> -targetDir <new_folder_name>
./sda-cli -config <configuration_file> upload --encrypt-with-key <public_key> -r <folder_to_upload_with_unencrypted_data> -targetDir <new_folder_name>
```

will first encrypt all files in `<folder_to_upload_with_unencrypted_data>` and then upload the folder recursively (selecting only the created `c4gh` files) under `<new_folder_name>`.
Expand All @@ -190,13 +190,13 @@ All the following cases require a [configuration file to be downloaded](https://
This feature returns all the files in the user's inbox recursively and can be executed using:

```bash
./sda-cli list [-config <configuration_file>]
./sda-cli [-config <configuration_file>] list
```

It also allows for requesting files/filepaths with a specified prefix using:

```bash
./sda-cli list [-config <configuration_file>] <prefix>
./sda-cli [-config <configuration_file>] list <prefix>
```

This command will return any file/path starting with the defined `<prefix>`.
Expand All @@ -207,7 +207,7 @@ If no config is given by the user, the tool will look for a previous login from
To list datasets or the files within a dataset that the user has access to, the `--datasets` flag should be used and the login URL must be provided:

```bash
./sda-cli list -config <configuration_file> --datasets --url <login_url> (--bytes)
./sda-cli -config <configuration_file> list --datasets --url <login_url> (--bytes)
```

where `<login_url>` is the URL where the user goes for authentication. This command returns a list of accessible datasets, including the number
Expand All @@ -216,7 +216,7 @@ of files in each dataset and the total size of it. The `--bytes` flag is optiona
To list the files within a particular dataset, the user must provide the dataset ID (which can be obtained by executing the previous command):

```bash
./sda-cli list -config <configuration_file> -dataset <datasetID> -url <login_url> (--bytes)
./sda-cli -config <configuration_file> list -dataset <datasetID> -url <login_url> (--bytes)
```

This command returns a list of files within the dataset, including their file IDs, file sizes and file paths. The `--bytes` flag is optional and it will display the size of the files in bytes.
Expand All @@ -241,22 +241,22 @@ For downloading files the user also needs to know the download service URL and t
For downloading one specific file the user needs to provide the path or the id (the id should **NOT** have "/") of this file by running the command below:

```bash
./sda-cli download -config <configuration_file> -dataset-id <datasetID> -url <download-service-URL> [<filepath> or <fileid>]
./sda-cli -config <configuration_file> download -dataset-id <datasetID> -url <download-service-URL> [<filepath> or <fileid>]
```

where `<configuration_file>` the file downloaded in the [previous step](#download-the-configuration-file), `<dataset_id>` the ID of the dataset and `<filepath>` the path of the file (or `<fileid>` the id of the file) in the dataset.
The tool also allows for downloading multiple files at once, by listing their filepaths (or file ids) separated with space and it also allows for selecting a folder where the files will be downloaded, using the `outdir` argument:

```bash
./sda-cli download -config <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> <path/to/file1> <other/path/to/file2> ... (or <fileID_1> <fileID_2> ...)
./sda-cli -config download <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> <path/to/file1> <other/path/to/file2> ... (or <fileID_1> <fileID_2> ...)
```

#### Download files recursively

For downloading the content of a folder (including subfolders) the user need to add the `--recursive` flag followed by the path(s) of the folder(s):

```bash
./sda-cli download -config <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> --recursive path/to/folder1 path/to/folder2 ...
./sda-cli -config download <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> --recursive path/to/folder1 path/to/folder2 ...
```

#### Download from file
Expand All @@ -265,15 +265,15 @@ For downloading multiple files the user can provide a text file with the paths o
In this case user needs to use the `--from-file` flag and at the end user needs to provide the path of the text file with the paths of the files to download:

```bash
./sda-cli download -config <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> --from-file <path/to/text_file>
./sda-cli -config download <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> --from-file <path/to/text_file>
```

#### Download all the files of the dataset

For downloading the whole dataset the user needs add the `--dataset` flag and NOT providing any filepaths:

```bash
./sda-cli download -config <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> --dataset
./sda-cli -config download <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> --dataset
```

where the dataset will be downloaded in the `<outdir>` directory be keeping the original folder structure of the dataset.
Expand All @@ -283,7 +283,7 @@ where the dataset will be downloaded in the `<outdir>` directory be keeping the
When a [public key](#create-keys) is provided, you can download files that are encrypted on the server-side with that public key. The command is similar to downloading the unencrypted files except that a public key is provided through the `-pubkey` flag. For example:

```bash
./sda-cli download -pubkey <public-key-file> -config <configuration_file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> <filepath_1_to_download> <filepath_2_to_download> ...
./sda-cli -config <configuration_file> download -pubkey <public-key-file> -dataset-id <datasetID> -url <download-service-url> -outdir <outdir> <filepath_1_to_download> <filepath_2_to_download> ...
```

After a successful download, the encrypted files can be [decrypted](#decrypt-file) using the private key corresponding to the provided public key.
Expand Down Expand Up @@ -330,7 +330,7 @@ htsget -dataset <datasetID> -filename <filename> -reference <reference-number> -
for example:

```
./sda-cli htsget -config testing/s3cmd.conf -dataset DATASET0001 -filename htsnexus_test_NA12878 -reference 11 -host http://localhost:8088 -pubkey testing/c4gh.pub.pem -output ~/tmp/result.c4gh --force-overwrite
./sda-cli -config testing/s3cmd.conf htsget -dataset DATASET0001 -filename htsnexus_test_NA12878 -reference 11 -host http://localhost:8088 -pubkey testing/c4gh.pub.pem -output ~/tmp/result.c4gh --force-overwrite
```

# Developers' section
Expand Down
Loading
Loading