Merge pull request #25 from rodekruis/repeat

add repeating group
rodekruis · Mar 13, 2024 · 2f6d5f9 · 2f6d5f9
2 parents 3a4c048 + bd4c134
commit 2f6d5f9
Show file tree

Hide file tree

Showing 2 changed files with 99 additions and 57 deletions.
diff --git a/README.md b/README.md
@@ -8,42 +8,55 @@ Synopsis: a [dockerized](https://www.docker.com/) [python](https://www.python.or
 
 Details: see [the docs](https://kobo-connect.azurewebsites.net/docs).
 
-## API Usage
-
-### EspoCRM
+## EspoCRM
 
 Using the [`kobo-to-espocrm`](https://kobo-connect.azurewebsites.net/docs#/default/kobo_to_espocrm_kobo_to_espocrm_post) endpoint, it is possible to save a Kobo submission as one or more entities in [EspoCRM](https://www.espocrm.com/). 
 
-#### Steps to connect Kobo to EspoCRM (see also screenshot below):
+### Basic setup
 
 1. Define which questions in the Kobo form need to be saved in which entity and field in EspoCRM.
 2. In EspoCRM,
-   1. Create a role (Administration>Roles), set `Access` to the target entity on `enabled`, with the permission on `yes` to `Create` (if you need to update records, also add `Read` and `Edit`)
-   2. Create an API user (Administration>API Users), give it a descriptive `User Name`, select the previously created role, make sure `Is Active` is checked and that `Authentication Method` is `API Key`. After saving, you will see a newly created API Key which is needed for the next step.
+   - Create a role (Administration>Roles), set `Access` to the target entity on `enabled`, with the permission on `yes` to `Create` (if you need to update records, also add `Read` and `Edit`).
+   - Create an API user (Administration>API Users), give it a descriptive `User Name`, select the previously created role, make sure `Is Active` is checked and that `Authentication Method` is `API Key`. After saving, you will see a newly created API Key which is needed for the next step.
 3. [Register a new Kobo REST Service](https://support.kobotoolbox.org/rest_services.html) for the Kobo form of interest and give it a descriptive name.
 4. Insert as `Endpoint URL`
 ```
 https://kobo-connect.azurewebsites.net/kobo-to-espocrm
 ```
 6. In Kobo REST Services, add under `Custom HTTP Headers`:
-   1. In `Name` add `targeturl` with in the `Value` the EspoCRM URL (for example, https://espocrminstancex.com)
-   2. In `Name` add `targetkey` with in the `Value` the (newly) created API Key (from EspoCRM API User)
+   - In `Name` add `targeturl` with in the `Value` the EspoCRM URL (for example, https://espocrminstancex.com).
+   - In `Name` add `targetkey` with in the `Value` the (newly) created API Key (from EspoCRM API User).
 9. For each question, add a `Custom HTTP Header` that specifies which Kobo questions responds to which entity and field EspoCRM:
-   1. The header `Name` (left) must correspond to the Kobo question **name**. (You can check the Kobo question name by going into edit mode of the form, open 'Settings' of the specific question and inspect the `Data Column Name`. Also, the Kobo question names can be found in the 'Data' table with previous submissions. This Kobo question name is different from the [Kobo question label](https://support.kobotoolbox.org/getting_started_xlsform.html#adding-questions) and can not contain spaces or symbols (except the underscore).)
-   2. The header value (right) must correspond to the EspoCRM entity **name**, followed by a dot (`.`), followed by the specific field **name**. Example: `Contact.name`. (EspoCRM name is different from the EspoCRM label, similar to the difference between Kobo question name and Kobo question label)
-
-#### Extra steps for adding Multi-Enums, Attachments and Updating records (not mandatory):
-- If you have a question of type `Select Many` (`select_multiple`) in Kobo and you want to save it in a field of type `Multi-Enum` in EspoCRM, add `multi.` before the Kobo question name in the header name (see screenshot below). 
-- If you need to send **attachments** (e.g. images) to to EspoCRM, add a `Custom HTTP Header` called `kobotoken` with your API token (see [how to get one](https://support.kobotoolbox.org/api.html#getting-your-api-token)).
-- If you need to **update** a pre-existing record:
-  - add a question of type `calculate` called `updaterecordby` in the kobo form, whcih will contain the value of the field which you will use to identify the record;
-  - add a `Custom HTTP Header` called `updaterecordby` with the name of the field that you will use to identify the record.
+   - The header `Name` (left) must correspond to the Kobo question **name**. (You can check the Kobo question name by going into edit mode of the form, open 'Settings' of the specific question and inspect the `Data Column Name`. Also, the Kobo question names can be found in the 'Data' table with previous submissions. This Kobo question name is different from the [Kobo question label](https://support.kobotoolbox.org/getting_started_xlsform.html#adding-questions) and can not contain spaces or symbols (except the underscore).).
+   - The header value (right) must correspond to the EspoCRM entity **name**, followed by a dot (`.`), followed by the specific field **name**. Example: `Contact.name`. (EspoCRM name is different from the EspoCRM label, similar to the difference between Kobo question name and Kobo question label).
 
+> [!IMPORTANT]  
+> If you need to send **attachments** (e.g. images) to EspoCRM, add a `Custom HTTP Header` called `kobotoken` with your API token (see [how to get one](https://support.kobotoolbox.org/api.html#getting-your-api-token)).
 
 <img src="https://github.com/rodekruis/kobo-connect/assets/26323051/06de75f3-d02d-4f9f-bb82-db6736542cf5" width="500">
 
+### Advanced setup: select many, repeat groups, etc.
+
+- If you have a question of type `Select Many` (`select_multiple`) in Kobo and you want to save it in a field of type `Multi-Enum` in EspoCRM, add `multi.` before the Kobo question name in the header name.
+  - Example header: `multi.multiquestion1`: `Entity.field1`
+- If you have a **repeating group** of questions in Kobo:
+  - you will need to save each repeated question in a different field in EspoCRM;
+  - in the header name:
+    - add `repeat.`, followed by the Kobo group;
+    - then add a number to specify the number of the repeated question (starting from 0);
+    - then add the name of the repeated question after the number;
+  - in the header value:
+    - as before, use the entity name, followed by a dot (`.`), followed by the field name in EspoCRM.
+  - Example headers:
+    - `repeat.repeatedgroup.0.repeatedquestion`: `Entity.field1`
+    - `repeat.repeatedgroup.1.repeatedquestion`: `Entity.field2`
+  - Not all repeated questions need to be filled in nor saved to EspoCRM.
+- If you need to **update** a pre-existing record:
+  - add a question of type `calculate` called `updaterecordby` in the kobo form, which will contain the value of the field which you will use to identify the record;
+  - add a `Custom HTTP Header` called `updaterecordby` with the name of the field that you will use to identify the record.
+
 
-### 121
+## 121
 
 Using the [`kobo-to-121`](https://kobo-connect.azurewebsites.net/docs#/default/kobo_to_121_kobo_to_121_post) endpoint, it is possible to save a Kobo submission as a Person Affected (PA) registration in the [121 Portal](https://www.121.global/).
 
@@ -58,7 +71,7 @@ Step by step:
 
 _Special Headers_:
 
-- The headers `url121` is required and corresponds the the url of the 121 instance (without trailing `/`, so e.g. https://staging.121.global)
+- The headers `url121` is required and corresponds the url of the 121 instance (without trailing `/`, so e.g. https://staging.121.global)
 - Headers `username121` and `password121`, corresponding to the 121 username and the 121 password respectively, must be included as well.
 - If `programid` is included as a (select one) question, the `XML Value` of the question in kobo needs to be the corresponding number in the 121 portal, the label can be something else, see below
   ![programId](https://github.com/rodekruis/kobo-connect/assets/39266480/1b0ccf53-2740-4432-b31e-d5cb57d2aac5)
@@ -72,7 +85,7 @@ See below for an example configuration, in which programId was not included as a
 #### Nota Bene
 The 121 API is currently throttled at 3000 submissions per minute. If you expect to go over this limit, please reach out the the 121 platform team.
 
-### Create headers endpoint
+## Create headers endpoint
 If you need to map a lot of questions, creating the headers manually is cumbersome. The `/create-kobo-headers` endpoint automates this. It expects 4 query parameters:
 - `system`: required, enum (options: 121, espocrm, generic)
 - `kobouser`: your kobo username
@@ -90,7 +103,7 @@ In the body you can pass all the headers you want to create as key value pairs,
 
 This endpoint assumes the IFRC kobo server (`https://kobonew.ifrc.org`)
 
-### Generic endpoint
+## Generic endpoint
 
 See [the docs](https://kobo-connect.azurewebsites.net/docs).
 

diff --git a/main.py b/main.py
@@ -57,7 +57,14 @@
 cosmos_container_client = cosmos_db.get_container_client('kobo-submissions')
 
 
+@app.get("/", include_in_schema=False)
+async def docs_redirect():
+    """Redirect base URL to docs."""
+    return RedirectResponse(url='/docs')
+
+
 def add_submission(kobo_data):
+    """Add submission to CosmosDB. If submission already exists and status is pending, raise HTTPException."""
     submission = {
         'id': str(kobo_data['_uuid']),
         'uuid': str(kobo_data['formhub/uuid']),
@@ -93,27 +100,8 @@ def update_submission_status(submission, status, error_message=None):
         )
 
 
-class system(str, Enum):
-    system_generic = "generic"
-    system_espo = "espocrm"
-    system_121 = "121"
-
-
-def required_headers(
-        targeturl: str = Header(),
-        targetkey: str = Header()):
-    return targeturl, targetkey
-
-
-def required_headers_121(
-        url121: str = Header(),
-        username121: str = Header(),
-        password121: str = Header()):
-    return url121, username121, password121
-
-
 def get_kobo_attachment(URL, kobo_token):
-    # Get attachment from kobo
+    """Get attachment from kobo"""
     headers = {'Authorization': f'Token {kobo_token}'}
     data_request = requests.get(URL, headers=headers)
     data = data_request.content
@@ -134,19 +122,14 @@ def get_attachment_dict(kobo_data):
 
 
 def clean_kobo_data(kobo_data):
+    """Clean Kobo data by removing group names and converting keys to lowercase."""
     kobo_data_clean = {k.lower(): v for k, v in kobo_data.items()}
     # remove group names
     for key in list(kobo_data_clean.keys()):
         new_key = key.split('/')[-1]
         kobo_data_clean[new_key] = kobo_data_clean.pop(key)
     return kobo_data_clean
 
-def clean_text(text):
-    # Normalize text to remove accents
-    normalized_text = unicodedata.normalize('NFD', text)
-    # Remove accents and convert to lowercase
-    cleaned_text = ''.join(c for c in normalized_text if not unicodedata.combining(c)).lower()
-    return cleaned_text
 
 def espo_request(submission, espo_client, method, action, params=None):
     """Make a request to EspoCRM. If the request fails, update submission status in CosmosDB."""
@@ -157,14 +140,14 @@ def espo_request(submission, espo_client, method, action, params=None):
         update_submission_status(submission, 'failed', e.detail)
 
 
-@app.get("/", include_in_schema=False)
-async def docs_redirect():
-    """Redirect base URL to docs."""
-    return RedirectResponse(url='/docs')
+def required_headers_espocrm(
+        targeturl: str = Header(),
+        targetkey: str = Header()):
+    return targeturl, targetkey
 
 
 @app.post("/kobo-to-espocrm")
-async def kobo_to_espocrm(request: Request, dependencies=Depends(required_headers)):
+async def kobo_to_espocrm(request: Request, dependencies=Depends(required_headers_espocrm)):
     """Send a Kobo submission to EspoCRM."""
 
     kobo_data = await request.json()
@@ -183,11 +166,10 @@ async def kobo_to_espocrm(request: Request, dependencies=Depends(required_header
     attachments = get_attachment_dict(kobo_data)
 
     # check if records need to be updated
-    update_record, update_record_payload = False, {}
+    update_record_payload = {}
     if 'updaterecordby' in request.headers.keys():
         if 'updaterecordby' in kobo_data.keys():
             if kobo_data['updaterecordby'] != "" and kobo_data['updaterecordby'] is not None:
-                update_record = True
                 update_record_entity = request.headers['updaterecordby'].split('.')[0]
                 update_record_field = request.headers['updaterecordby'].split('.')[1]
                 update_record_payload[update_record_entity] = {
@@ -200,10 +182,18 @@ async def kobo_to_espocrm(request: Request, dependencies=Depends(required_header
     payload, target_entity = {}, ""
     for kobo_field, target_field in request.headers.items():
 
-        multi = False
+        kobo_value, multi, repeat, repeat_no, repeat_question = "", False, False, 0, ""
+
+        # determine if kobo_field is of type multi or repeat
         if "multi." in kobo_field:
             kobo_field = kobo_field.split(".")[1]
             multi = True
+        if "repeat." in kobo_field:
+            split = kobo_field.split(".")
+            kobo_field = split[1]
+            repeat_no = int(split[2])
+            repeat_question = split[3]
+            repeat = True
 
         # check if kobo_field is in submission
         if kobo_field not in kobo_data.keys():
@@ -218,11 +208,22 @@ async def kobo_to_espocrm(request: Request, dependencies=Depends(required_header
         else:
             continue
 
+        # get kobo_value based on kobo_field type
         if multi:
             kobo_value = kobo_data[kobo_field].split(" ")
+        elif repeat:
+            if 0 <= repeat_no < len(kobo_data[kobo_field]):
+                kobo_data[kobo_field][repeat_no] = clean_kobo_data(kobo_data[kobo_field][repeat_no])
+                if repeat_question not in kobo_data[kobo_field][repeat_no].keys():
+                    continue
+                kobo_value = kobo_data[kobo_field][repeat_no][repeat_question]
+            else:
+                continue
         else:
             kobo_value = kobo_data[kobo_field]
-        kobo_value_url = kobo_data[kobo_field].replace(" ", "_")
+
+        # process individual field; if it's an attachment, upload it to EspoCRM
+        kobo_value_url = str(kobo_value).replace(" ", "_")
         if kobo_value_url not in attachments.keys():
             payload[target_entity][target_field] = kobo_value
         else:
@@ -282,6 +283,23 @@ async def kobo_to_espocrm(request: Request, dependencies=Depends(required_header
     update_submission_status(submission, 'success')
     return JSONResponse(status_code=200, content=target_response)
 
+########################################################################################################################
+
+
+def clean_text(text):
+    # Normalize text to remove accents
+    normalized_text = unicodedata.normalize('NFD', text)
+    # Remove accents and convert to lowercase
+    cleaned_text = ''.join(c for c in normalized_text if not unicodedata.combining(c)).lower()
+    return cleaned_text
+
+
+def required_headers_121(
+        url121: str = Header(),
+        username121: str = Header(),
+        password121: str = Header()):
+    return url121, username121, password121
+
 
 @app.post("/kobo-to-121")
 async def kobo_to_121(request: Request, dependencies=Depends(required_headers_121)):
@@ -355,6 +373,15 @@ async def kobo_to_121(request: Request, dependencies=Depends(required_headers_12
 
     return JSONResponse(status_code=response.status_code, content=target_response)
 
+########################################################################################################################
+
+
+class system(str, Enum):
+    system_generic = "generic"
+    system_espo = "espocrm"
+    system_121 = "121"
+
+
 @app.post("/create-kobo-headers")
 async def create_kobo_headers(json_data: dict, system: system, kobouser: str, kobopassword: str, koboassetId: str, hookId: str = None):
     """Utility endpoint to automatically create the necessary headers in Kobo. \n
@@ -405,9 +432,11 @@ def remove_keys(data, keys_to_remove):
     else:
         return JSONResponse(content={"message": "Failed to post data to the target endpoint"}, status_code=response.status_code)
 
+########################################################################################################################
+
 
 @app.post("/kobo-to-generic")
-async def kobo_to_generic(request: Request, dependencies=Depends(required_headers)):
+async def kobo_to_generic(request: Request):
     """Send a Kobo submission to a generic API.
      API Key is passed as 'x-api-key' in headers."""