Merge pull request #90 from JuliaSprenger/wrapup

Add data manager and maintainer docs

docs/config_template.json

@@ -0,0 +1,8 @@
+{
+  "title": "<My Project Title>",
+  "structure": "structure.csv",
+  "customization": "customization.csv",
+  "validation": [],
+  "api_token": "<paste your API token here>",
+  "api_url": "<put the url of your metadata server here>",
+}

docs/data_manager.md

@@ -0,0 +1,39 @@
+## Instructions for Data Managers
+
+### Why use Diglabtools
+
+Diglabtools permits the definition of survey forms for metadata collection independently of a specific platform use to serve the form. At the moment DigLabtools support RedCap as well as ElabFTW, but more server backend can be integrated in the future.
+By using Diglabtools surveys can be built from a set of existing survey templates (see the [ElabFTW template parts](../../../elab_bridge/template_parts) and [RedCap template parts](../../../redcap_bridge/template_parts). Templates can be translated between the bridge modules using the [conversion module](../../../diglab_utils/conversion.py))
+Organizing the survey form specifications outside of a server environment has the advantage that a survey can be set up based on pure text files, which can easily be version controlled, e.g. using git. Using version control the generation of a survey form can be provenance tracked, as already implemented for the redcap_bridge.
+Another advantage of DigLabTools is that the use of templates permits standardization of survey forms across different projects, but splitting generic templates and project specific customizations.
+
+_Note:_ RedCap surveys specifications contain a superset of ElabFTW survey specifications. When converting from RedCap to ElabFTW some parts of the specifications will be ignored.
+
+
+### Creating a project
+
+The specifications of a project require 3 files:
+
+- `project.json`: This file defines the general settings of the project. It contains reference to the files defining the survey content (by convention called `structure.json` and `customization.json`), the credentials to connect to the server and potentially a list of survey templates against which the project can be validated.
+- `structure.tsv/json`: This file defines the fields present in the survey form. Currently this is in `tsv` format for RedCap projects and `json` for ElabFTW projects. In the simplest case of only using a survey template, this mostly consists of the name of the template to include
+- `customizations.tsv/json`: This file defines project specific customization of the survey fields included via the `structure.tsv/json`. This is e.g useful to provide a selection of values for a predefined field, like the names of the experiments for the `User` field.
+
+For more examples on how a project can be set up, see the projects used in the [elab tests](../../../elab_bridge/tests/testfiles_elab/TestProject) and [redcap tests](../../../redcap_bridge/tests/testfiles_redcap/TestProject) 
+
+To get from the project specification files listed above to a survey form on a server multiple steps are required
+
+1) [Server Project Setup] Generation of an empty project on the server
+2) [Local Project Setup] Generation of the project specification files above (see also the [user level setup](setup.md))
+3) [Survey Generation] Using the elab_bridge / redcap_bridge `project_control.setup_project()` function.
+
+The last step will perform multiple steps internally:
+- building of the general project survey form based on the `structure` specifications using the `project_building.build_project` function
+- customization of the project survey based on the `customization` specification using the `project_building.customize_project` function
+- upload of the project survey to the server using the `server_interface` module
+
+### Running tests
+
+Running the tests of DigLabTools requires `pytest` to be installed and generation of an empty project on the server to be used for upload and download testing.
+Instead of adding the server api token to the TestProject configuration file, create an environment variable named `ELAB_API_TOKEN` / `REDCAP_API_TOKEN` containing the coresponding token. This will be automatically used while running the tests.
+
+

docs/developer.md

@@ -0,0 +1,17 @@
+## Maintainer Documentation
+
+### Running tests on github actions
+
+For security reasons tests on github actions are not run automatically as soon as a new pull request is opened, but need to be labelled by a maintainer as `save_to_test`. Adding this label will retrigger the tests and show a link to the test run in the thread of the pull request.
+
+### Create a new release
+
+To make a new release of DigLabTools on github follow these steps:
+
+- Ensure all tests are passing
+- Check that the documentation is built correctly
+- Create a commit on github to bump the version number in main to the release version, e.g. `0.3.0`
+- Create a github tag pointing to that commit. This should automatically trigger the [github actions release workflow](https://github.com/INT-NIT/DigLabTools/blob/main/.github/workflows/release.yml) building and publishing the new version on PyPI.
+- Test the new release by installing it from PyPI
+- Create a commit on github to bump the version number to the next alpha version, e.g. `0.4.0-a.1`
+

docs/index.rst

@@ -11,6 +11,8 @@
    installation
    setup
    usage
+   data_manager
+   developer
 
 .. mdinclude:: ../README.md
 

docs/setup.md

@@ -7,13 +7,16 @@ To be able to access the metadata from a server an API token needs to be created
 #### ...on RedCap
 Login to your local RedCap server and go to the `API` menu on the left pane. Here you can generate a personalized access token for your user account. Copy the generated token, you will need to add it to your DigLab project configuration in the next step.
 
+![RedCap_API-token-generation.png](images/RedCap_API-token-generation.png)
+
 #### ...on eLabFTW
 Login to your local eLabFTW server and go to your `User Control Panel`. Switch to the `API KEYS` tab and generate a personalized API Key. Note that this key will only be displayed once. If you only plan to use DigLabTools to retrieve data from the metadata server, choose the `Read Only` permission. In case you also want to generate surveys via DigLabTools, choose `Read/Write` permissions.  Copy the generated key and use it as `api_token` in your DigLab project configuration in the next step.
 
+![ElabFTW_API-token-generation.png](images/ElabFTW_API-token-generation.png)
 
 ### Configure your DigLab project
 
-You need to create a project configuration `json` file on your computer to store the RedCap API token you just generated. Use your preferred text editor to create a `<project_name>.json` file with the following content:
+You need to create a project configuration `json` file on your computer to store the API token you just generated. You can download a template of that files [here](config_template.json) or use your preferred text editor to create a `<project_name>.json` file with the following content:
 
 ```
 {
@@ -26,5 +29,10 @@ You need to create a project configuration `json` file on your computer to store
 }
 ```
 
-Replace the `<...>` entries with the corresponding values for your project. The `structure`, `customization` and `validation` entries are required for project building and not needed to download the collected metadata.
-Note that the API URL should come in the `https://redcap.MY_INSTIUTION.org/api/` format for RedCap and `https://MY_INSTITUTION.elab.one/api/v2` format for eLabFTW .
+Replace the `<...>` entries with the corresponding values for your project. The `structure`, `customization` and `validation` entries are only required for project building and not needed to download the collected metadata.
+Note that the API URL format depends on the server you are using:
+
+| Server   | API URL Format                           |
+|----------|------------------------------------------|
+| RedCap   | `https://redcap.MY_INSTIUTION.org/api/`  |
+| ElabFTW  | `https://MY_INSTITUTION.elab.one/api/v2` |