It is a backend service providing a REST API to manage datasets.
The API is described in detail using OpenAPI v3 specification standard format in the file api-docs/API-reference-v1.yaml
(rendered by Redoc here: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc).
Some operations require authentication using the OpenId-Connect standard protocol. Basically, when the user wants to login, the client application must redirect to the authentication service, to obtain a bearer token. Then, that token must be included in the "Authorization" header of any request sent to the dataset-service. More details below.
Basic API operations:
- POST /api/datasets
- GET /api/datasets/{id}
- PATCH /api/datasets/{id}
- GET /api/datasets
Below there is a walkthrough by examples with CURL.
Other managed routes outside /api/:
- GET /web/{staticFilePath}
static files needed to load the client app (frontend) in the browser - GET /{anyOtherPath}
internally redirected to /web/index.html, the client app with JS code that manages locally other routes like /datasets
DSS_ENDPOINT=https://chaimeleon-eu.i3m.upv.es/dataset-service/api
Some operations require previous authentication using the OpenId-Connect standard protocol. It is recommended to use an OpenID-Connect library for the programming language of your client application, it will ease your work providing things like the automatic refresh of the token. You can use a generic library or a particular library for the implementation used in your project. For example, if Keycloak is used, you can use the "client adapter": https://www.keycloak.org/docs/latest/securing_apps/#supported-platforms
If you use React: https://www.npmjs.com/package/@react-keycloak/web
Basically the user will be redirected to the authentication service with a URL like this: https://chaimeleon-eu.i3m.upv.es/auth/realms/CHAIMELEON/protocol/openid-connect/auth/?scope=openid+email+profile&response_type=id_token&client_id=dataset-service-ui&redirect_uri=https://chaimeleon-eu.i3m.upv.es/dataset-service/
The last parameter ("redirect_uri") contains the URI of our client application. When the user comes back to our client application, the bearer token will be included in the URL as a parameter named 'id_token'.
For development/testing purposes you can use curl to obtain a token:
echo 'Write <yourUserName> <yourPassword>' && read DSS_USER DSS_PASS
DSS_TOKEN=$(curl -d "client_id=dataset-explorer" -d "username=$DSS_USER" -d "password=$DSS_PASS" -d "grant_type=password" \
"https://chaimeleon-eu.i3m.upv.es/auth/realms/CHAIMELEON/protocol/openid-connect/token" | jq -r ".access_token")
Warning: please ensure you are using "https" in the URL to avoid send password in clear text.
Once the token has been obtained it must be included in a header like this:
Authorization: bearer <token>
The is the JWT (JSON Web Token) provided by the authorization service that will be verified (sign, expiration time, etc.) by the dataset-service. Also the user data will be extracted from the token in order to set the author of the dataset (in case of creation) and the author of the operation to send to the tracer service.
POST /datasets
With the authorization header and the dataset properties within the body in JSON format. If success, the code 201 will be returned. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message.
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/datasets/operation/createDataset
Example:
$ curl -i -X POST \
-H "Authorization: bearer $DSS_TOKEN" \
-H "Content-Type: application/json" \
-d '{"name": "TestDataset3", "description": "This is a dataset for testing.", "studies": [{ "studyId": "5e57a4356af19d299c17026d", "studyName": "GMIBG2DECUERPOENTERO", "subjectName": "17B76FEW", "pathInDatalake": "blancagomez/17B76FEW_Neuroblastoma/GMIBG2DECUERPOENTERO20160225", "series": [{"folderName": "serie1", "tags":[]}, {"folderName": "serie2", "tags":[]}, {"folderName": "serie3", "tags":[]}], "url": "" }, { "studyId": "5e5629835938d32160636353", "studyName": "RM431RMRENAL", "subjectName": "17B76FEW", "pathInDatalake": "blancagomez/17B76FEW_Neuroblastoma/RM431RMRENAL20130820", "series": [{"folderName": "serie1", "tags":[]}], "url": "" }, { "studyId": "5e6a422939b892367c8a5c23", "studyName": "TCPEDITRICOABDOMINOPLVICOCONCONTRASTE", "subjectName": "17B76FEW", "pathInDatalake": "blancagomez/17B76FEW_Neuroblastoma/TCPEDITRICOABDOMINOPLVICOCONCONTRASTE20150129", "series": [{"folderName": "serie1", "tags":[]}], "url": "" }, { "studyId": "5e6b449a3144dc2bc0841efc", "studyName": "RM411RMABDOMEN", "subjectName": "21N56F7T", "pathInDatalake": "blancagomez/21N56F7T_Neuroblastoma/RM411RMABDOMEN20100804", "series": [{"folderName": "serie1", "tags":[]}], "url": "" }, { "studyId": "5e6a3d41c9065c475c32b3fe", "studyName": "RM411RMABDOMEN", "subjectName": "21N56F7T", "pathInDatalake": "blancagomez/21N56F7T_Neuroblastoma/RM411RMABDOMEN20150109", "series": [{"folderName": "serie1", "tags":[]}], "url": "" }, { "studyId": "5eeba960903aec091076c180", "studyName": "RM815RMDORSAL", "subjectName": "1GB90F75", "pathInDatalake": "blancagomez/1GB90F75_Neuroblastoma/RM815RMDORSAL20121123", "series": [{"folderName": "serie1", "tags":[]}], "url": "" }], "subjects": [{"subjectName": "17B76FEW", "eForm": {}}, {"subjectName": "21N56F7T", "eForm": {}}, {"subjectName": "1GB90F75", "eForm": {}}]}" \
"${DSS_ENDPOINT}/datasets"
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Content-Length: 61
Content-Type: application/json
{"url": "/api/datasets/efa2cba6-4a17-4612-8074-7e9eb9c9d7ca"}
Example of "external" dataset creation:
$ curl -i -X POST \
-H "Authorization: bearer $DSS_TOKEN" \
-F name="Maastricht Lung1" -F description="Test dataset from Maastricht University." \
-F clinical_data=@"NSCLC Radiomics Lung1.clinical.csv" \
"${DSS_ENDPOINT}/datasets?external=True"
HTTP/1.1 100 Continue
HTTP/1.1 201 Created
Content-Length: 61
Content-Type: application/json
{"url": "/api/datasets/3388a9c5-4ebb-45ba-93fc-7b54813f0cf2"}
The creation of a medium or big dataset can be a time consuming process, so it is asynchronous. The POST operation returns the code 201 after some checks over the input and the creation of the dataset in database. The rest of work (file system setup and hash calculation) is done by a kubernetes job and the progress can be retrieved with GET /datasets/{id}/creationStatus. While the dataset is being created the flag "draft" is true and can not be changed to false (it does not appear in the "dataset.editablePropertiesByTheUser"). When the flag "draft" is true, there is another flag "creating" which is true until the end of the creation process (i.e. while the retrieving of creationStatus makes sense).
GET /datasets
With the authorization header and some parameters accepted in the URL for pagination. If success, the code 200 will be returned and a JSON array in the body of the response. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message.
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/datasets/operation/listDatasets
Example:
$ curl -i -X GET \
-H "Authorization: bearer $DSS_TOKEN" \
"${DSS_ENDPOINT}/datasets?limit=30&skip=0"
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 728
[{"id": "f99017af-9015-4222-b064-77f3c1b49d8b", "name": "TestDataset3", "authorName": "test test", "creationDate": "2021-10-05 12:29:11.932542", "public": false, "invalidated": false, "studiesCount": 6, "subjectsCount": 3},
{"id": "00e821c4-e92b-48f7-a034-ba2df547e2bf", "name": "TestDataset2", "authorName": "test test", "creationDate": "2021-10-04 14:50:47.214108", "public": false, "invalidated": false, "studiesCount": 1, "subjectsCount": 1},
{"id": "efa2cba6-4a17-4612-8074-7e9eb9c9d7ca", "name": "TestDataset1", "authorName": "test test", "creationDate": "2021-10-04 14:42:37.725548", "public": false, "invalidated": false, "studiesCount": 1, "subjectsCount": 1}]
This is the same operation as the previous example but with the parameter searchString (case-insensitive).
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/datasets/operation/listDatasets
Example:
$ curl -i -X GET \
-H "Authorization: bearer $DSS_TOKEN" \
"${DSS_ENDPOINT}/datasets?searchString=dataset3"
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 182
[{"id": "f99017af-9015-4222-b064-77f3c1b49d8b", "name": "TestDataset3", "authorName": "test test", "creationDate": "2021-10-05 12:29:11.932542", "public": false, "invalidated": false, "studiesCount": 6, "subjectsCount": 3}]
GET /datasets/{id}
With the authorization header and some parameters accepted in the URL for pagination of studies in the dataset. Returns a JSON object. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message.
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/datasets/operation/getDataset
Example:
$ curl -i -X GET \
-H "Authorization: bearer $DSS_TOKEN" \
"${DSS_ENDPOINT}/datasets/f99017af-9015-4222-b064-77f3c1b49d8b?studiesLimit=30"
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 1506
{"id": "f99017af-9015-4222-b064-77f3c1b49d8b", "name": "TestDataset3", "previousId": null, "authorId": "a43d426c-11aa-41cb-ab15-616d68627c77", "authorName": "test test", "authorEmail": "[email protected]", "creationDate": "2021-10-05 22:29:11.932542", "description": "This is a dataset for testing.", "gid": 1, "public": false, "studiesCount": 6, "subjectsCount": 3,
"studies": [
{"studyId": "5e57a4356af19d299c17026d", "studyName": "GMIBG2DECUERPOENTERO", "subjectName": "17B76FEW", "url": ""},
{"studyId": "5e5629835938d32160636353", "studyName": "RM431RMRENAL", "subjectName": "17B76FEW", "url": ""},
{"studyId": "5e6a422939b892367c8a5c23", "studyName": "TCPEDITRICOABDOMINOPLVICOCONCONTRASTE", "subjectName": "17B76FEW", "url": ""},
{"studyId": "5e6b449a3144dc2bc0841efc", "studyName": "RM411RMABDOMEN", "subjectName": "21N56F7T", "url": ""},
{"studyId": "5e6a3d41c9065c475c32b3fe", "studyName": "RM411RMABDOMEN", "subjectName": "21N56F7T", "url": ""},
{"studyId": "5eeba960903aec091076c180", "studyName": "RM815RMDORSAL", "subjectName": "1GB90F75", "url": ""}
]}
PATCH /datasets/{id}
With the authorization header. If success, the code 200 will be returned. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message.
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/datasets/operation/modifyDataset
Example:
$ curl -i -X PATCH \
-H "Authorization: bearer $DSS_TOKEN" \
-H "Content-Type: application/json" \
-d '{"property": "public", "value": true}' \
"${DSS_ENDPOINT}/datasets/00e821c4-e92b-48f7-a034-ba2df547e2bf"
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html; charset=UTF-8
PUT /users/{userName}
With the authorization header and the user properties within the body in JSON format. If success, the code 201 will be returned. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message.
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/users/operation/createUser
Example authenticating previously with a service account:
$ DSS_TOKEN=$(curl -d "client_id=kubeauthorizer-pod" -d "client_secret=XXXX-XXXX-XXXX" -d "grant_type=client_credentials" \
"https://chaimeleon-eu.i3m.upv.es/auth/realms/CHAIMELEON/protocol/openid-connect/token" | jq -r ".access_token")
$ curl -i -X PUT -H "Authorization: bearer $DSS_TOKEN" -H "Content-Type: application/json" \
-d '{"uid": "d290f1ee-6c54-4b01-90e6-d701748f0851", "groups": ["data-scientists", "dataset-administrator"]}' \
"${DSS_ENDPOINT}/users/user1"
HTTP/1.1 201 Created
Content-Length: 0
Content-Type: text/html; charset=UTF-8
GET /users/{userName}
With the authorization header. Returns a JSON object. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message.
Details: https://chaimeleon-eu.i3m.upv.es/dataset-service/api-doc#tag/users/operation/getUser
Example:
$ curl -i -X GET -H "Authorization: bearer $DSS_TOKEN" "${DSS_ENDPOINT}/users/user1"
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 13
{"gid": 2002}
POST /datasetAccessCheck
With the authorization header and the access properties (userName and datasets required) within the body in JSON format. If success, the code 204 will be returned. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message. In case of fail due to any required dataset which is not available for the user, the code 403 is returned with a JSON array containing not available datasets.
Example authenticating previously with a service account:
$ DSS_TOKEN=$(curl -d "client_id=kubernetes-operator" -d "client_secret=XXXX-XXXX-XXXX" -d "grant_type=client_credentials" \
"https://chaimeleon-eu.i3m.upv.es/auth/realms/CHAIMELEON/protocol/openid-connect/token" | jq -r ".access_token")
$ curl -i -X POST -H "Authorization: bearer $DSS_TOKEN" -H "Content-Type: application/json" \
-d '{"userName": "user1", "datasets": ["00e821c4-e92b-48f7-a034-ba2df547e2bf", "f99017af-9015-4222-b064-77f3c1b49d8b"]}' \
"${DSS_ENDPOINT}/datasetAccessCheck"
HTTP/1.1 204 No Content
Content-Length: 0
POST /datasetAccess/{id}
With the authorization header and the access properties (userName, datasets required and tool info) within the body in JSON format. If success, the code 201 will be returned. If fail, a 40X code will be returned with a JSON object in the body containing also the code and the error message. In case of fail due to any required dataset which is not available for the user, the code 403 is returned.
Example:
$ curl -i -X POST -H "Authorization: bearer $DSS_TOKEN" -H "Content-Type: application/json" \
-d '{"userName": "user1", "datasets": ["f99017af-9015-4222-b064-77f3c1b49d8b"], "toolName": "desktop-tensorflow", "toolVersion": "0.3.1"}' \
"${DSS_ENDPOINT}/datasetAccess/c0bd6506-219b-4fc2-8fdb-3deb1d1a4ac2"
HTTP/1.1 201 Created
Content-Length: 0
POST /set-ui
To set the web UI static files from any URL with a ZIP package.
Example:
$ curl -i -H "devToken: SECRET-TOKEN" -d "http://158.42.154.23:19000/build.zip" "https://chaimeleon-eu.i3m.upv.es/dataset-service/api/set-ui"
There is a second method uploading directly the ZIP file in the body.
Example:
$ curl -i -H "devToken: SECRET-TOKEN" -F zip=@"build.zip" "https://chaimeleon-eu.i3m.upv.es/dataset-service/api/set-ui?method=fileInBody"
set IMAGE_NAME=harbor.chaimeleon-eu.i3m.upv.es/chaimeleon-services/dataset-service-backend
set /p IMAGE_TAG=< VERSION
docker build -t %IMAGE_NAME%:%IMAGE_TAG% .
docker login -u registryUser harbor.chaimeleon-eu.i3m.upv.es
docker push %IMAGE_NAME%:%IMAGE_TAG%
docker logout harbor.chaimeleon-eu.i3m.upv.es
cd k8s
You can make a private copy of the templates you want to adjust:
cp 0-ceph-sercret.yaml 0-ceph-sercret.private.yaml
cp 1-db-service.yaml 1-db-service.private.yaml
cp 2-dataset-service.yaml 2-dataset-service.private.yaml
cp 3-ingress.yaml 3-ingress.private.yaml
Configure: edit your private copies (set passwords, urls, etc.)
Create a namespace: kubectl create namespace dataset-service
Create the persistent volumes: kubectl apply -n dataset-service -f 0-pvcs.yaml
Create the ceph secret: kubectl apply -n dataset-service -f 0-ceph-secret.private.yaml
Create the service account, role and roleBinding: kubectl apply -n dataset-service -f 0-service-account.yaml
Create the database deployment and service: kubectl apply -n dataset-service -f 1-db-service.private.yaml
Create the main deployment and service: kubectl apply -n dataset-service -f 2-dataset-service.private.yaml
And finally the ingress: kubectl apply -n dataset-service -f 3-ingress.private.yaml
Deploy database with docker:
docker run -d -e POSTGRES_DB=db -e POSTGRES_USER=dssuser -e POSTGRES_PASSWORD=XXXXXX -p 5432:5432 --name my-postgres postgres:12
Once database is ready, you can run the main service with a local configuration file:
python .\start_dataset_service.py .\etc\dataset-service-local.yaml
You can explore database with psql:
docker exec -it my-postgres bash
psql db dssuser
\dt
select * from metadata;
\q
exit
Configuration values specification: see etc/dataset-service.default.yaml
First of all, default configuration values will be loaded from a file located in (the first that exists):
./etc/dataset-service.default.yaml/etc/dataset-service/dataset-service.default.yaml
You should change at least the password values... but please do not modify the default config file for that, it is useful as a template to always see the full configuration keys available with descriptive comments. To set your own configuration you should make a copy of the file:
cp dataset-service.default.yaml dataset-service.yaml
Then adjust the values of keys you want to change and next you can delete all the others (the default values will be taken).
The configuration file will be loaded from path (the first that exists):
./etc/dataset-service.yaml/etc/dataset-service/dataset-service.yaml
Optional: you can set the location of the configuration file in the first parameter of the execution line. Example:
python .\start_dataset_service.py .\etc\dataset-service-local.yaml
The keys defined in the configuration file takes precedence over the same keys which are defined in default configuration file.
Finally you can override in the same way some (or all) of the configuration keys with the environment variable DATASET_SERVICE_CONFIG.
- Using win cmd:
set DATASET_SERVICE_CONFIG={ db: { host: "mydbhost" } } - Using bash:
export DATASET_SERVICE_CONFIG={ db: { host: "mydbhost" } }
Please note it is JSON format this time, and takes precedence over all configuration files.
The capabilities of a user (which operations can do) are defined by the token received. There are two places in the token to see:
- the roles for the application (i.e. in 'resource_access.dataset-service.roles').
- the groups with prefix 'PROJECT-' Example:
{
"exp": ...,
"iat": ...,
...
"iss": ...,
"aud": ...,
"sub": ...,
...
"realm_access": {
"roles": [
...
]
},
"resource_access": {
"account": {
"roles": [
...
]
},
"dataset-service": {
"roles": [
"use_datasets",
"admin_datasets"
]
}
},
"groups": [
"data-scientists",
"PROJECT-CHAIMELEON"
],
...
"name": ...,
"preferred_username": ...,
"email": ...
}
These are the known roles:
- use_datasets (1): The user can use datasets (in addition to only see the metadata).
- admin_datasets (2): Also can create datasets and modify properties (and the state) of his/her own datasets.
- superadmin_datasets (3): Also can modify any dataset (not only his/her own datasets).
The name of each one can be customized in the configuration file. The number is just a hint, not part of the name: you can see them as permission levels, each level include the previous levels.
The groups with pattern PROJECT-{project} indicate the projects which the user has joined to.
The user will be able to see and use any public dataset and the non-public datasets of the projects which the user has joined to.
There are other special roles:
- admin_users: required for the operations in path '/user'.
- admin_datasetAccess: required for the operations in paths '/datasetAccess' and '/datasetAccessCheck'.
Each dataset has some flags (with value true or false) which define its state of visibility, editability and usability.
The flags are:
- draft
- public
- invalidated
According to value of the flags a dataset can be in one of these states:
- Draft: (draft = true, public = false, invalidated = false)
All datasets are created in this state.
Only the author can see and use the dataset in Draft state and some properties can be modified (name/title, description).
The draft mode can be useful for testing datasets because they are "private" to the author.
Possible actions:
- Release (draft -> false), goes to Released state.
- Invalidate (invalidated -> true), goes to Invalidated state. - Released: (draft = false, public = false, invalidated = false)
When released, the dataset can't be edited anymore and all the registered users in the project to which the dataset is assigned can see it and use it.
Possible actions:
- Publish (public -> true) goes to Published state.
- Invalidate (invalidated -> true), goes to Invalidated state. - Published: (draft = false, public = true, invalidated = false)
When published, the dataset can be seen and used by any registered user joined to other projects or even not joined to any project.
It can be seen (not used) by unregistered users.
If there is not a pidUrl yet for the dataset, the metadata and a small index of studies will be deposited publicly in Zenodo.org in order to obtain a DOI for the pidUrl.
Possible actions:
- Set to non-public (public -> false), returns to Released state.
- Invalidate (invalidated -> true), goes to Invalidated state. - Invalidated: (draft = true/false, public = true/false, invalidated = true)
Only appears in the list of the author, but anyone (who has the id or link) can see the details.
For example when the dataset have a PID and somebody goes to the detail from the paper where is included.
Anyway, a big label with the text "invalidated" should appear in details, and also in the list for the author.
Nobody can modify it nor use it, neither the author.
Possible actions:
- 'Reactivate' (invalidated -> false), goes to previous state (Draft, Released or Published).
All the actions can be performed only by the author or superadmin.
Some properties of dataset can be modified depending on the user and the current state of the dataset.
In order to simplify the client and not duplicate that logic,
there is a dynamic property named "editablePropertiesByTheUser" in the object returned by GET /api/datasets/{id} (the details of dataset).
The value of that property is a list of the properties (including flags) that can be modified with PATCH operation by the current user in the current state of dataset.
Only the author or superadmin can modify any of the properties or flags.
And these are the properties that can be modified (and when):
- draft (when draft = true)
- public (when draft = false)
- invalidated (always)
- name (when draft = true)
- description (when draft = true)
- licenseUrl (always)
- contactInfo (always)
- pidUrl (when draft = false)
Dataset-service is able to use the Zenodo API to automatically create a deposition of dataset metadata. The publication at Zenodo gives visibility to the dataset and allow us to obtain a DOI, which is a permanent reference (Permanent ID Url) to the dataset that can be included in publications. This way the dataset can fulfill the FAIR principles.
- On released datasets, the user can optionally edit the PID.
She/He can...
- set a custom URL, for example a DOI obtained by him/herself,
- or simply select to obtain automatically one from Zenodo.
- When a dataset is published, then a PID will be obtainded for it if it still does not have one.
- On invalidated datasets, the Zenodo deposition (if exists) will be closed (not listed in searches on Zenodo and showing the "closed" label for someone who access through an old reference).
Only metadata will be published in Zenodo, never the contents of dataset (images nor clinical data).
There is a section in the config file to configure the access to Zenodo:
zenodo:
url: "https://sandbox.zenodo.org/"
# The url of Zenodo service to submit depositions.
# Usually "https://sandbox.zenodo.org/" for testing, and "https://zenodo.org/" for production.
access_token: "XXXXXXXXXXXXXXXXXXXXXXX"
community: ""
grant: ""
That service provides a POST /datasets/eucaimSearch operation which fulfills the requirements of the Federated Search of the EUCAIM project and currently is integrated. So any published dataset (not invalidated) will be included in the results of that Federated Search.