Merge pull request #1 from rodekruis/dev

Dev

.github/workflows/update-image-dev.yaml

@@ -1,4 +1,4 @@
-name: Update pipeline prod
+name: Update pipeline dev
 
 on:
   push:

README.md

@@ -1,3 +1,32 @@
 # Near real-time rainfall pipeline
 
-TBD
+Near real-time rainfall monitoring. Part of the project RIPOSTE for Cameroon Red Cross Society.
+
+## Description
+The pipeline roughly consists of three steps:
+
+- Extract daily data on rainfall measurement of the past days (e.g. last 14 days) from [NOAA's GPM](https://gpm.nasa.gov/data/directory)
+- Transform the data into pre-defined areas (health districts) and calculate average rainfall of the past days. Then determine which area has its average rainfall higher than pre-defined thresholds.
+- Send this data as alert to the EspoCRM for the NS.
+
+## Basic Usage
+To run the pipeline locally
+
+1. Fill in the secrets in .env.example and rename the file to .env; in this way, they will be loaded as environment variables
+2. Install requirements
+    ```
+    pip install poetry
+    poetry install --no-interaction
+    ```
+3. Run the pipeline : `python nrt_rainfall_pipeline.py --extract --transform --send`
+    ```
+    Usage: nrt_rainfall_pipeline.py [OPTIONS]
+
+    Options:
+    --country TEXT  country ISO3
+    --extract       extract rainfall data
+    --transform     calculate rainfall data into pre-defined
+    --send          send to IBF app
+    --dateend       specify date until which the data should be extracted
+    --help          Show this message and exit.
+    ```

nrt_rainfall_pipeline.py

@@ -14,7 +14,7 @@
 @click.option(
     "--dateend",
     help="date start in YYYY-mm-dd",
-    default=(datetime.now(timezone.utc)-timedelta(days=1)).strftime("%Y-%m-%d"),
+    default=(datetime.now(timezone.utc)-timedelta(days=2)).strftime("%Y-%m-%d"),
 )
 
 def run_nrt_rainfall_pipeline(

nrt_rainfall_pipeline/pipeline.py

@@ -39,7 +39,7 @@ def run_pipeline(
         transform: bool = True,
         send: bool = True,
         save: bool =True,
-        dateend: datetime = datetime.now(timezone.utc)-timedelta(days=1),
+        dateend: datetime = datetime.now(timezone.utc)-timedelta(days=2),
         # debug: bool = False
     ):
         """Run the flood data pipeline"""