Merge branch 'master' of github.com:USDA-ARS-NWRC/awsm_test_cases

README.md

@@ -1,27 +1,28 @@
 # AWSM Test Cases
 This repository contains instructions for installing the Docker container of AWSM and running short test cases. The instructions involve installing the supporting software and the AWSM system. The test cases include three model tests:
-1. Tuolumne River Basin three day run during WY 2016, which includes an lidar snow depth update from the Airborne Snow Observatory
-2. Boise River Basin three month run during WY 2017 during a historic snowmelt event
-3. Reynolds Mountain East in the Reynolds Creek Experimental Watershed for the entire WY 2006 with wind redistribution of precipitation
+1. Reynolds Mountain East in the Reynolds Creek Experimental Watershed for the entire WY 2006 with wind redistribution of precipitation
+2. Boise River Basin three month simulation during WY 2017 for a historic snowmelt event
+3. Tuolumne River Basin two day simulation during WY 2016, which includes an lidar snow depth update from Airborne Snow Observatory
 
 ## Table of contents
 - [AWSM Test Cases](#awsm-test-cases)
   - [Table of contents](#table-of-contents)
   - [Repository contents](#repository-contents)
 - [Downloading this repository](#downloading-this-repository)
 - [Docker setup](#docker-setup)
-  - [Compute resources](#compute-resources)
   - [Installing Docker](#installing-docker)
   - [Installing `docker-compose`](#installing-docker-compose)
+  - [Compute resources for docker](#compute-resources-for-docker)
 - [NetCDF viewers](#netcdf-viewers)
   - [Windows](#windows)
-  - [Mac](#mac)
-  - [Linux](#linux)
+  - [Mac and Linux](#mac-and-linux)
 - [Running the test cases](#running-the-test-cases)
   - [Reynolds Mountain East, Idaho (RME)](#reynolds-mountain-east-idaho-rme)
   - [Boise River Basin, Idaho (BRB)](#boise-river-basin-idaho-brb)
   - [Tuolumne River Basin, California](#tuolumne-river-basin-california)
-- [Juptyer Lab for analysis](#juptyer-lab-for-analysis)
+- [Jupyter Lab for analysis](#jupyter-lab-for-analysis)
+  - [Linux and Mac](#linux-and-mac)
+  - [Windows](#windows-1)
 
 ## Repository contents
 - **brb/:** all files necessary to run the Boise River Basin test case
@@ -30,6 +31,8 @@ This repository contains instructions for installing the Docker container of AWS
 - **output/:** contains output from the test cases
 - **README.md:** documentation on running the test cases
 - **awsm_docker:** python script wrapping the docker compose calls for each test case
+- **docker-compose.yml:** Jupyter Lab analysis
+- **docker-compose.windows.yml:** Jupyter Lab analysis for Windows
 
 
 # Downloading this repository
@@ -49,23 +52,15 @@ Our modeling system is isolated in a Docker container in order to run consistent
 
 In order to run AWSM, you will need to install Docker and understand the basics of running a Docker container. For a more in depth information, read https://docs.docker.com/engine/docker-overview/. 
 
->**NOTE: Docker requires that you are a system administrator for your workstation or have sudo privileges.**
-
-<!-- Instructions for installing the supporting software are included in the [Linux and Mac set-up instructions](#linux-and-mac-set-up-instructions) section and the [Windows set-up instructions](#windows-set-up-instructions) section. -->
-
-## Compute resources
-Docker is only allowed limited compute resources from your computer. On Windows and Mac, you will need to provide more resources to Docker for AWSM to function properly. We recommend at a minimum:
-* 2 cores
-* 16GB of RAM
-* 20GB of hard drive space if running all test cases
+>**NOTE: Docker requires that you are a system administrator for your computer or have sudo/administrative privileges.**
 
 ## Installing Docker
 
 * [Window installation instructions](https://docs.docker.com/docker-for-windows/install/)
 * [Mac installation instructions](https://docs.docker.com/docker-for-mac/install/)
 * [Ubuntu installation instruction](https://docs.docker.com/install/linux/docker-ce/ubuntu/) and many other Linux distributions are supported
 
->**NOTE: Windows/Mac users need to ensure that Docker Desktop is running and configured prior to running test cases**
+>**NOTE: Windows/Mac users need to ensure that Docker Desktop is running and configured prior to running test cases. Enabling shared drives is required to run the test cases.**
 
 ## Installing `docker-compose`
 
@@ -75,118 +70,128 @@ For ease of use, we use `docker-compose` to handle the mounting of folders betwe
 * Mac already includes `docker-compose`
 * Linux [latest install instructions](https://docs.docker.com/compose/install/)
 
+## Compute resources for docker
+The AWSM test cases have been optimized to limit the computational burden but increasing the computational resources will make the simulations run faster. On Windows and Mac, docker is only allowed limited compute resources from your computer and you will need to provide more resources to Docker for AWSM to function properly. We recommend at a minimum the following computer specifications:
+* 2 cores
+* 16GB of RAM
+* 20GB of hard drive space if running all test cases
+
 # NetCDF viewers
 
-The output files from [SMRF](https://github.com/USDA-ARS-NWRC/smrf) and [PySnobal](https://github.com/USDA-ARS-NWRC/pysnobal) are in NetCDF format. In order to view these files, you will need a NetCDF viewer. 
+The output files from [SMRF](https://github.com/USDA-ARS-NWRC/smrf) and [PySnobal](https://github.com/USDA-ARS-NWRC/pysnobal) are in NetCDF format. In order to view these files, you will need a NetCDF viewer. The following are a small collection of viewers that we have used on the different operating systems but are by no means the only available viewers.
 
 ## Windows
-Panoply is a NetCDF, HDF, and GRIB viewer maintained by NASA, and is a good option for **Windows** users. Install instruction can be found at https://www.giss.nasa.gov/tools/panoply/.
+[Panoply](https://www.giss.nasa.gov/tools/panoply/) is a NetCDF, HDF, and GRIB viewer maintained by NASA, and is a good option for **Windows** users.
 
-## Mac
-NCView is a lightweight NetCDF viewer that allows for quick visualization of NetCDF files. It is maintained by UC San Diego. Further documentation can be found at http://meteora.ucsd.edu/~pierce/ncview_home_page.html.
-
-The easiest method of install on a Mac is to use Homebrew.
-```
-brew install ncview
-```
-
-## Linux
-To install Ncview from the command line on Ubuntu:
-```
-sudo apt-get install ncview
-```
+## Mac and Linux
+[Ncview](http://meteora.ucsd.edu/~pierce/ncview_home_page.html) is a lightweight NetCDF viewer that allows for quick visualization of NetCDF files and is maintained by UC San Diego.
 
 # Running the test cases
 
-There are three tests cases in this `awsm_test_cases` repo. They are the Tuolumne
-River Basin, Boise River Basin and Reynolds Mountain East. In order to run these cases using the AWSM Docker image, the correct folders must be mounted to the Docker image, and the AWSM configuration file must be passed to the Docker image.
+There are three tests cases in this `awsm_test_cases` repo. They are Reynolds Mountain East, Boise River Basin and Tuolumne. In order to run these cases using the AWSM Docker image, the correct folders must be mounted to the Docker image, and the AWSM configuration file must be passed to the Docker image.
 
-For these test cases, the folder sharing is handled through a docker-compose file (located in each test case folder), and a Python script that will call the docker-compose routine behind the scenes.
+For these test cases, the folder sharing is handled through a docker-compose file (located in each test case folder), and a Python script that will create and call the docker-compose routine behind the scenes. The docker compose commands are printed to the terminal to run outside of the Python script.
 
 The `output` folder structure for AWSM was developed to organize multiple runs over various water years and basins. The base output structure for these test cases are
 ```
 output/<basin>/devel/<year>/<run_name>/<data | logs | runs><run_date_range>
 ```
 
-The model results will be available in the `run` directory, simply open the netCDF files with a viewer to quickly visualize the results. `snow.nc` will have the snowpack state variables like `specific_mass` or `snow_depth` and the `em.nc` will have all the snowpack energetics like `swi` or `net_rad`.
+The model results will be available in the `run` directory, simply open the netCDF files with a viewer to quickly visualize the results. `snow.nc` will have the snowpack state variables like `specific_mass` or `snow_depth` and the `em.nc` will have all the snowpack energetics like `swi` or `net_rad` if the simulation output those variables.
 
 ## Reynolds Mountain East, Idaho (RME)
 
 The RME test case is for the complete water year 2006 for a small snow dominated headwater catchment within the larger Reynolds Creek Experimental Watershed (RCEW). Specifics for this model run:
-* 10 meter resolution, hourly
+* 10 meter resolution, hourly timestep
 * 0.27 km<sup>2</sup> research watershed
 * 28 meterological stations across RCEW
-* Wind redistribution of precipitation base on [Winstral and Marks 2002](https://doi.org/10.1002/hyp.1238)
+* Wind redistribution of precipitation based on [Winstral and Marks 2002](https://doi.org/10.1002/hyp.1238)
+* Model type is `smrf_ipysnobal` meaning `SMRF` and `pysnobal` run at the same time utilizing the in memory queue
 
 From the command prompt on **Linux** or **Mac**, run
 ```
-./awsm_docker --case rme
+python ./awsm_docker --case rme
 ```
 On **Windows**, run the PowerShell command
 ```
 python .\awsm_docker --case rme
 ```
 
-> **NOTE:** will take approximately 45 minutes to 1.5 hours to complete.
+> **NOTE:** will take approximatel 1 to 2 hours to complete.
 
 ## Boise River Basin, Idaho (BRB)
 
-The BRB test case is for a significant rain on snow event in water year 2017 where the second highest inflows to the Lucky Peak Reservoir was recorded.
-* 100 meter resolution, hourly
+The BRB test case is for a significant rain on snow event in water year 2017 where the second highest inflows to the Lucky Peak Reservoir were recorded.
+* 100 meter resolution, hourly timestep
 * 7,000 km<sup>2</sup> watershed
 * 58 meterological stations
-* Model spin up performed to produce a model state file for restart model
+* Model spin up previously performed to produce a model state file to restart model
 * 3 month model run
+* Model type is `smrf_ipysnobal` meaning `SMRF` and `pysnobal` run at the same time utilizing the in memory queue
 
 
 From the command prompt on **Linux** or **Mac**, run
 ```****
-./awsm_docker --case brb
+python ./awsm_docker --case brb
 ```
 On **Windows**, run the PowerShell command
 ```
 python .\awsm_docker --case brb
 ```
 
-> **NOTE:** will take approximately 2 to 4 hours to complete, dependent on computer resources.
+> **NOTE:** will take approximately 2 to 8 hours to complete, dependent on computer resources and if threading is turned on or off.
+
+> **WARNING:** Due to the domain size of the BRB, the simulation will consume significant memory. If the computer does not have 16GB minimum allocated to Docker, you can turn off threading by setting `threading: False` in the configuration file under `[system]`. Otherwise `AWSM` will quit unexpectedly.
 
 ## Tuolumne River Basin, California
 
 The Tuolumne test case is for April 15 through April 17 in WY 2016. It contains a 50 meter Airborne Snow Observatory snow depth product that automatically updates the model state and shows the model results in a summary report.
 
-* 50 meter resolution, hourly
+* 50 meter resolution, hourly timestep
 * 1,400 km<sup>2</sup> watershed
 * 21 meterological stations
-* 3 day model run, first runs SMRF and then runs `ipysnobal`
+* 2 day model run, first runs SMRF and then runs `ipysnobal`
 * 50 meter snow depth update in netCDF file format
+* The simulation will first run `SMRF` then run `ipysnobal`
 * Automated report generated to show lidar flight update change to model state
 
 
 From the command prompt on **Linux** or **Mac**, run
 ```
-./awsm_docker --case tuol
+python ./awsm_docker --case tuol
 ```
 On **Windows**, run the PowerShell command
 ```
 python .\awsm_docker --case tuol
 ```
 
-> **NOTE:** will take approximately 2 to 2.5 hours to complete.
+> **NOTE:** will take approximately 10 to 30 minutes to complete.
 
 The generated report will be in the `report` directory under `output/tuolumne`.
 
 
-# Juptyer Lab for analysis
+# Jupyter Lab for analysis
 
 All the analysis for the manuscript figures are performed in Jupter Lab included in this repository. A `docker-compose.yml` will pull the image `jupyter/scipy-notebook` which is a pre-built docker image containing everything needed for analysis. Only the BRB and RME test case are analyzed as the figures for Tuolumne were generated with `snowav` during the run.
 
-To start the docker image in Linux:
+Once the Jupyter Lab is started, go to `http://localhost:10000` where you will be prompted for a token. The token will have appeared in the terminal, copy and paste into the form and submit.
+
+
+## Linux and Mac
+To start Jupyter Lab in Linux:
 ```
 CURRENT_UID=$(id -u) docker-compose up
 ```
 
-To start the docker image in Mac:
+## Windows
+The Jupyter Lab needs the current user id to ensure that any files created or modified are owned by the user. Windows is slightly different in that the `CURRENT_UID` is just a placeholder and does not matter. To start the docker image in Windows, either:
 
-To start the docker image in Windows:
+```
+docker-compose -f docker-compose.windows.yml up
+```
+or
+```
+SET CURRENT_UID=1000
+docker-compose up
+```
 
-Once started, go to `http://localhost:10000` where you will be prompted for a token. The token will have appeared in the terminal, copy and paste into the form and submit.

awsm_docker

@@ -55,7 +55,8 @@ def run():
         os.remove('./{}/topo/maxus.nc'.format(fpc))
         shutil.move('./{}/topo/maxus_30window.nc'.format(fpc),
                     './{}/topo/maxus.nc'.format(fpc))
-    print(maxus_call)
+        print(maxus_call)
+
     # run system
     run_call = ('docker-compose'
                     ' -f ./{}/docker-compose.yml'

brb/docker-compose.yml

@@ -2,7 +2,7 @@ version: '3'
 
 services:
    awsm:
-      image: usdaarsnwrc/awsm:0.9.13
+      image: usdaarsnwrc/awsm:0.9.26
       volumes:
         - .:/data/input
         - ../output:/data/output

docker-compose.windows.yml

@@ -0,0 +1,12 @@
+version: '3'
+
+services:
+   jupyter:
+      image: jupyter/scipy-notebook:7f1482f5a136
+      volumes:
+        - .:/home/jovyan/work
+      ports:
+        - 10000:8888
+      environment:
+        - JUPYTER_ENABLE_LAB=yes
+      user: root

rme/docker-compose.yml

@@ -2,7 +2,7 @@ version: '3'
 
 services:
    awsm:
-      image: usdaarsnwrc/awsm:0.9.13
+      image: usdaarsnwrc/awsm:0.9.26
       volumes:
         - .:/data/input
         - ../output:/data/output