Merge pull request #3 from rodekruis/dev

minor improvements

README.md

@@ -23,28 +23,47 @@ To run the pipeline locally
     pip install poetry
     poetry install --no-interaction
     ```
-3. Run the pipeline : `python nrt_rainfall_pipeline.py --extract --transform --send`
+3. Edit the configuration in `config\config.yaml`
+   ```
+   - name: <country-iso3>
+     days-to-observe: <val> # number of most recent days to observe rainfall e.g. 14
+     alert-on-threshold: <val> # threshold to send to EspoCRM e.g. 50
+     shapefile-area: <name>.geojson # shapefile of areas (. geojson) where the zonal stats bases on
+     espo-area:  # entity storing areas code (and id)
+       entity: <entity-name>
+       field: <id-field-name>
+     espo-destination: # entity to send alerts to
+       entity: <entity-name>
+       field: <rainfall-field-name>
+   ```
+4. Run the pipeline : `python nrt_rainfall_pipeline.py --extract --transform --send`
     ```
     Usage: nrt_rainfall_pipeline.py [OPTIONS]
 
     Options:
     --country TEXT  country ISO3
-    --extract       extract rainfall data
-    --transform     calculate rainfall data into pre-defined
-    --send          send to IBF app
-    --dateend       specify date until which the data should be extracted
-    --help          Show this message and exit.
+    --extract       extract NRT rainfall raster data
+    --transform     calculate rainfall data in pre-defined administrative areas
+    --send          send to EspoCRM
+    --dateend       specify a customed latest date YYYY-mm-dd until which the data should be extracted, by default it is the date before today
+    --help          Show this message and exit
     ```
 
-Payload sent to EspoCRM:
-
+__Note:__ Payload sent to EspoCRM
+```
     {
         "status": "onhold",
         "type": "heavyrainfall", 
         "source": "GPM",
         "<espo-area-field>": "<id-field-name>",
         "<espo-destination-field>": "<rainfall-field-name>"
     }
+```
+Where:
+- `<espo-area-field>`: is the value of `espo-area`'s `field` in the config
+- `<id-field-name>`: is to be automatically filled in the pipeline
+- `<espo-destination-field>`: is the value of `espo-destination`'s `field` in the config
+- `<rainfall-field-name>`: is to be automatically filled in the pipeline
 
 ## Adding new country
 1. Prepare shapefile
@@ -65,17 +84,5 @@ Payload sent to EspoCRM:
     | `type`    | `enum` | `heavyrainfall` |
     | `source`   | `enum` or `text` | `GPM` |
 - Make sure the entity for area is linked with this one
-4. Add the new country to the `config\config.yaml` below the existing one:
-   ```
-   - name: <country-iso3>
-     days-to-observe: 14  # number of most recent days to observe rainfall
-     alert-on-threshold: 50  # threshold to send to EspoCRM
-     shapefile-area: <name>.geojson # shapefile of areas (. geojson) where the zonal stats bases on
-     espo-area:  # entity storing areas code (and id)
-       entity: <entity-name>
-       field: <id-field-name>
-     espo-destination: # entity to send alerts to
-       entity: <entity-name>
-       field: <rainfall-field-name>
-   ```
+4. Add the new country to the `config\config.yaml`, see Step 3 in __Basic usage__:
 5. Test and adjust settings if needed.

config/config.yaml

@@ -1,11 +1,11 @@
 countries:
   - name: CMR
-    days-to-observe: 14  # number of most recent days to observe rainfall
+    days-to-observe: 4  # number of most recent days to observe rainfall
     alert-on-threshold: 50  # threshold to send to EspoCRM
     shapefile-area: cmr_district_sante_2022.geojson # shapefile of areas (.geojson) where the zonal stats bases on
     espo-area:  # entity storing areas code and id
       entity: CHealthDistrict
-      field: CHealthDistrictId
+      field: cHealthDistrictId
     espo-destination: # entity to send alerts to
       entity: CClimaticHazard
       field: averageRainfall

nrt_rainfall_pipeline.py

@@ -8,17 +8,17 @@
 @click.command()
 @click.option("--country", help="country ISO3", default="CMR")
 @click.option("--extract", help="extract NRT rainfall raster data", default=False, is_flag=True)
-@click.option("--transform", help="calculate rainfall per admin", default=False, is_flag=True)
-@click.option("--send", help="send to EspoCRM", default=False, is_flag=True)
+@click.option("--transform", help="calculate rainfall data in pre-defined administrative areas", default=False, is_flag=True)
+@click.option("--send", help="specify a customed latest date YYYY-mm-dd until which the data should be extracted, by default it is the date before today", default=False, is_flag=True)
 @click.option("--save", help="save to storage", default=False, is_flag=True)
 @click.option(
     "--dateend",
-    help="date start in YYYY-mm-dd",
+    help="date end in YYYY-mm-dd",
     default=(datetime.now(timezone.utc)-timedelta(days=2)).strftime("%Y-%m-%d"),
 )
 
 def run_nrt_rainfall_pipeline(
-    country, extract, transform, send, save, dateend#, debug
+    country, extract, transform, send, save, dateend
 ):
     dateend = datetime.strptime(dateend, "%Y-%m-%d")
     pipe = Pipeline(

nrt_rainfall_pipeline/transform.py

@@ -111,7 +111,6 @@ def __prepare_data_for_espo(self, stats):
             new_d[destination_field] = new_d.pop('median')
             new_d.update(additional_data)
             stats_list.append(new_d)
-            print('stats_list: ', stats_list)
 
         threshold = self.settings.get_country_setting(self.country, "alert-on-threshold")
         filtered = self.__filter_dict(stats_list, destination_field, threshold)