Add import-metadata command

[funnell]

It would be very nice to have an import-metadata command for importing metadata from a .csv or .tsv sample sheet.

It seems common for users to write their own importer scripts using the SDK to help automate importing experiment metadata.
Currently, the only "official" sample sheet import mechanism is to upload a filled Excel submission template using the web API.
However, relying on the web UI upload can introduce some friction in situations where we are generating sample sheets programmatically, e.g. by querying an existing database.
Since it seems common to write importer scripts, we can save new users some time and reduce friction by providing an official import command!

If there was a way for the command to re-use the existing validation code used during the Excel submission template upload, then that would also be great!

[juanesarango]

@funnell If you would like to explore this further, this can be achieved by using the same api endpoints used in the frontend.

These are the steps:

1. Create submission: create a new isabl_api.Submission record by using POST /submissions endpoint.

// Payload example to /submissions
{
    "projects": [1, 10],
    "description": "This is a test",
    "tags": [{"name": "test", "pk": 4}]}
}

if successful, you will get the new created record in the response

2. Send file: Update the field submission_form of the new record by sending the excel filename as form-data, by using the PATCH /submissions/<pk>

3. Validate file: use the process endpoint POST submissions/process/<pk> with payload {commit: false} to check for errors.

If errors can be fixed:

4. Commit the file to create records in database: when it's free of errors, use the process endpoint POST submissions/process/<pk> with payload {commit: true} to check for errors.

If no file can't or won't be commited:

5. Delete unsuccessful submissions: if step 4 is never made, delete the submission you created DELETE submissions/process/<pk>

[nickp60]

Adding some R code for how I've implemented what you described, if anyone else finds this useful.

library(httr)
library(jsonlite)
library(dotenv)
dotenv::load_dot_env()


projects_GET <- httr::GET(url = paste0(url, "projects"), 
                          query = list(limit = 88),
                          add_headers(`Authorization` = paste("Token ", Sys.getenv("ISABL_API_TOKEN"))),
                          verbose()
)
project_list <- fromJSON(content(projects_GET,type="text"))$results

submission_init <- jsonlite::toJSON(
   list(
       projects = list(9),
       description = "Add shotgun samples from initial methods pilot" 
    ), auto_unbox = TRUE) 
make_submission <- httr::POST(url = paste0(url, "submissions"),body = submission_init,  encode = "json",
                              add_headers(
                                'Content-Type'= 'application/json',
                                `Authorization` = paste("Token ", Sys.getenv("ISABL_API_TOKEN"))),
                              verbose()
)
submission_ob <- fromJSON(content(make_submission,type="text"))
########### upload the file
patch_ob <- list(
  submission_form = "path_to_submission.xlsm"
)
patch_submission <- httr::PATCH(url = paste0(url, "submissions/", submission_ob$pk),
                                body = list(submission_form=upload_file(patch_ob$submission_form)),  encode = "multipart",
                                add_headers(
                                  'Content-Type'= 'multipart/form-data',
                                  `Authorization` = paste("Token ", Sys.getenv("ISABL_API_TOKEN"))),
                                verbose()
)
fromJSON(content(patch_submission,type="text"))


######### Validate
validate_submission <- httr::POST(url = paste0(url, "submissions/process/", submission_ob$pk),
                                  body = list("commit" = FALSE),  encode = "json",
                                  add_headers(
                                    'Content-Type'= 'application/json',
                                    `Authorization` = paste("Token ", Sys.getenv("ISABL_API_TOKEN"))),
                                  verbose()
)
fromJSON(content(validate_submission,type="text"))


### Commit validated submission
commit_submission <- httr::POST(url = paste0(url, "submissions/process/", submission_ob$pk),
                                body = list("commit" = TRUE),  encode = "json",
                                add_headers(
                                  'Content-Type'= 'application/json',
                                  `Authorization` = paste("Token ", Sys.getenv("ISABL_API_TOKEN"))),
                                verbose()
)
fromJSON(content(commit_submission,type="text"))