Merge pull request #49 from nsidc/docker

Dockerize & continuously deliver docker images

.github/workflows/build-and-publish.yml

@@ -0,0 +1,24 @@
+name: "Build & publish artifacts"
+
+on:
+  push:
+    branches:
+      - "main"
+      - "v2"  # TODO: Remove before merge!
+    tags:
+      - "v[0-9]+.[0-9]+.[0-9]+*"
+
+concurrency:
+  group: "${{ github.workflow }}-${{ github.ref }}"
+  cancel-in-progress: true
+
+env:
+  # Many color libraries just need this to be set to any value, but at least
+  # one distinguishes color depth, where "3" -> "256-bit color".
+  FORCE_COLOR: 3
+
+jobs:
+  build-and-publish-docker-image:
+    name: "Build and publish Docker image"
+    uses: "nsidc/.github/.github/workflows/build-and-publish-container-image.yml@main"
+    secrets: "inherit"

Dockerfile

@@ -0,0 +1,26 @@
+FROM mambaorg/micromamba:1.5.8-alpine3.20
+
+WORKDIR /app
+
+# Activate the conda environment during build process
+ARG MAMBA_DOCKERFILE_ACTIVATE=1
+
+COPY ./conda-lock.yml .
+
+# NOTE: `-p` is important to install to the "base" env
+RUN micromamba install -y \
+    -p /opt/conda \
+    -f conda-lock.yml \
+  && micromamba clean --all --yes
+
+
+# Install source
+COPY ./pyproject.toml .
+COPY ./antarctica_today ./antarctica_today
+COPY ./qgis ./qgis
+
+# TODO: `pip install --editable .`
+ENV PYTHONPATH=.
+
+# Extend the default micromamba entrypoint to use our CLI
+ENTRYPOINT ["/usr/local/bin/_entrypoint.sh", "python", "antarctica_today"]

compose.yml

@@ -0,0 +1,11 @@
+services:
+  cli:
+    image: "nsidc/antarctica_today:local"
+    build: "."
+    user: "root"
+    volumes:
+      - "./Tb/:/app/Tb/"
+      - "./data/:/app/data/"
+      - "./database/:/app/database/"
+      - "./plots/:/app/plots/"
+      - "./baseline_datasets/:/app/baseline_datasets/:ro"

doc/operation.md

@@ -39,14 +39,14 @@ threshold binaries provided by Tom Mote. These are checked in to this repository
 
 ## 1. Download NSIDC-0080
 
-Download NSIDC-0080 granules into the `Tb/` directory:
+Download NSIDC-0080 granules:
 
 ```bash
 PYTHONPATH=.
 python antarctica_today download-tb
 ```
 
-> [!IMPORTANT]
+> [!NOTE]
 > For data before 2022-01-10, data have already been processed through step 2 and are
 > available as binary ".bin" files in this repo's `/data/daily_melt_bin_files`
 > directory. This data was generated from `NSIDC-0001` and `NSIDC-0007` datasets.
@@ -56,19 +56,18 @@ python antarctica_today download-tb
 > downloads that raw data.
 
 
+#### Creates data:
+
+* `.nc` files in `Tb/` directory
+
+
 ## 2. Generate all the daily melt binary files
 
-Generate new data in `data/daily_melt_bin_files/`:
 
 ```bash
 PYTHONPATH=.
 python antarctica_today generate-daily-melt
 ```
-> [!IMPORTANT]
-> Binaries provided in this repo's `data/daily_melt_bin_files/` dir with pre-2016 dates
-> are already calibrated by Tom Mote and don't need to be generated. Remember from the
-> note in the previous step: pre-generated data goes through to 2022-01-10.
-
 
 <details><summary>🛠️ _TODO_</summary>
 I receive a large number of warnings like:
@@ -82,30 +81,27 @@ Why?
 </details>
 
 
-## 3. Generate the database
+#### Creates data:
 
-This step creates, primarily, four pickle files:
+* `.bin` files in `data/daily_melt_bin_files/` directory for dates on or after
+  2022-01-10
 
-* `daily_cumulative_melt_averages.pickle`
-* `daily_melt_pixel_averages.pickle`
-* `database/v3_1979-present_gap_filled.pickle`
-* `database/v3_1979-present_raw.pickle`
+> [!IMPORTANT]
+> Binaries provided in this repo's `data/daily_melt_bin_files/` dir with pre-2016 dates
+> are already calibrated by Tom Mote and don't need to be generated. Remember from the
+> note in the previous step: pre-generated data goes through to 2022-01-10.
 
-Additionally:
 
-* `.csv` files will be created in `database/` directory
-* `.tif` files will be created in `data/mean_climatology/` directory
-* `.tif` files will be created in `data/annual_*_geotifs` directories
+## 3. Generate the database
 
+> [!NOTE]
+> This command may take up to tens of minutes.
 
 ```bash
 PYTHONPATH=.
 python antarctica_today preprocess
 ```
 
-> [!NOTE]
-> This command may take up to tens of minutes.
-
 <details><summary>🛠️ _TODO_</summary>
 
 - [ ] Why is the next section called "Initializing"? Are there multiple pickle files?
@@ -124,11 +120,21 @@ python antarctica_today preprocess
 </details>
 
 
-### Database initialization (?)
+#### Creates data:
+
+* `database/v3_1979-present_gap_filled.pickle`
+* `database/v3_1979-present_raw.pickle`
+* `database/gap_fill_data/daily_cumulative_melt_averages.pickle`
+* `database/gap_fill_data/daily_melt_pixel_averages.pickle`
+* `.csv` files in `database/` directory
+* `.tif` files in `data/mean_climatology/` directory
+* `.tif` files in `data/annual_*_geotifs/` directories
 
 
+### Database initialization (?)
+
 <details><summary>🛠️ _TODO_</summary>
-Is this step necessary? It seems like new files aren't being created when this step is
+Is this step necessary? New files aren't being created when this step is
 run.
 </details>
 
@@ -174,7 +180,7 @@ python antarctica_today/update_data.py
 > [!NOTE]
 > This command may take up to tens of minutes.
 
-This will go through the entire database and produce summary maps and plots for every year on record in the `plots/` directory.
+This will go through the entire database and produce summary maps and plots for every year on record.
 
 ```bash
 PYTHONPATH=.
@@ -186,3 +192,22 @@ python antarctica_today process
 - [ ] After this step, `git status` shows changed files. Should they be committed?
 
 </details>
+
+
+#### Creates data:
+
+* `.png` files in `plots/` subdirectories
+
+
+## Running in Docker
+
+This repository includes a `compose.yml` configuration which enables running this code
+with Docker. For example, the `download-tb` command can be run as follows:
+
+```bash
+docker compose run cli download-tb
+```
+
+> [!WARNING]
+> By default, outputs will be written as `root`! You can override the user (TODO: how?)
+> to match your desired production user.