Add dockerfile setup

.gitignore

@@ -120,4 +120,8 @@ venv.bak/
 dmypy.json
 
 # Pyre type checker
-.pyre/
+.pyre/
+
+## repository specific
+/data/imagery
+/data/solar.db

README.md

@@ -8,23 +8,59 @@ I've chosen to use SQLite and SQLAlchemy for persisting the search locations and
 
 # Quickstart
 
-To install requirements, run:
+If you'd like to build and work out of the pre-existing Docker container, jump to the Docker container section just below. Regardless of the setup method, once you're all done you should be able to run:
 
-`pip install -r requirements.txt`
+`python run_entire_process.py --city <city to search> --state <state>`
 
-One of the requirements is rtree, which requires you also build libspatialindex, instructions [here](http://toblerity.org/rtree/install.html) (currently only required if you want to add filtering of existing OSM panels to your MapRoulette task to add the found panels to OSM)
+And the whole suite of scripts should run, eventually outputting a MapRoulette challenge geoJSON for your city. (And leaving you with a sqlite database of these locations)
 
-Also, currently [this DeepSolar repo](https://github.com/typicalTYLER/DeepSolar) must be present at ../DeepSolar (relative to this repo) and pre-trained weights must be present in the checkpoint directory.
+Please create an [issue](https://github.com/typicalTYLER/SolarPanelDataWrangler/issues/new) if you have any trouble with this quickstart!
 
-Lastly your Mapbox API key must be in your environment variables as MAPBOX_ACCESS_TOKEN="MY_ACCESS_TOKEN"
+## Manual Setup
 
-With all that taken care of, you can simply run: 
+### Conda Environment
 
-`python run_entire_process.py --city <city to search> --state <state>`
+One of the requirements is rtree, which requires you to install libspatialindex, instructions [here](http://toblerity.org/rtree/install.html).
 
-And the whole suite of scripts should run, eventually outputting a MapRoulette challenge geoJSON for your city. (And leaving you with a sqlite database of these locations)
+To install the environment, choose either the `setup/environment_cpu.yml` and `setup/environment_gpu.yml`. If you have a GPU that you intend to use, you'll need to setup the relevant packages / drivers (e.g. cuDNN / CUDA, NVIDIA diver, etc.) before installing the conda environment. Once you've chosen a YML file, you can create the environment via something like the following:
 
-Please create an [issue](https://github.com/typicalTYLER/SolarPanelDataWrangler/issues/new) if you have any trouble with this quickstart!
+```
+conda create --name spdw --file setup/environment_cpu.yml
+```
+
+*Note*: These environments are built on `conda=4.6.11`, `python=3.6.8`, and `tensorflow=1.12.0`.
+
+### DeepSolar repository
+
+Currently, [this DeepSolar repo](https://github.com/typicalTYLER/DeepSolar) must be present at ../DeepSolar (relative to this repo) and pre-trained weights must be present in the `ckpt` directory inside of the `DeepSolar` repository.
+
+### MapBox Token
+
+Your Mapbox API key must be in your environment variables as MAPBOX_ACCESS_TOKEN="MY_ACCESS_TOKEN".
+
+## Docker Setup
+
+Within the `setup` directory is a `build_docker_images.sh` script that can be used to automatically setup a docker container to develop out of.
+
+### No GPU Build 
+
+To build a docker image that uses the no GPU conda environment, first install docker. Next, choose a username that you would like to use inside the container, and from **within the setup directory**, run:
+
+```
+bash build_docker_images.sh --docker_user [specify username here]
+```
+
+Once the image is built, you should be able to work inside a container created from the image just as if you were logged into a remote instance.
+
+### GPU Build 
+
+To build a docker image that uses  the GPU conda environment, first install nvidia-docker (version 2.0). Next, make sure that you have the appropriate GPU driver installed for your system and for the version of tensorflow that will be used (`1.12.0`) as well as the versions of CUDA and cuDNN (CUDA 9.0 and cuDNN7. Once that is done, choose a username that you would like to use inside the container, and from **within the setup directory**, run:
+
+```
+bash build_docker_images.sh --docker_user [specify username here] --gpu_build
+```
+
+Once the image is built, you should be able to work inside a container created from the image just as if you were logged into a remote instance.
 
 # Overview
 
@@ -45,4 +81,4 @@ maproulette.py contains functionality to turn positive classifications (above a
 
 # Contributing
 
-Feel free to sign up for and submit pull requests for one of the [existing issues](https://github.com/typicalTYLER/SolarPanelDataWrangler/issues) if you want to contribute! I'm also down to add other Open Climate Fix collaborators as collaborators on this repo. Also feel free to create issues if you are having trouble with anything in this repo.
+Feel free to sign up for and submit pull requests for one of the [existing issues](https://github.com/typicalTYLER/SolarPanelDataWrangler/issues) if you want to contribute! I'm also down to add other Open Climate Fix collaborators as collaborators on this repo. Also feel free to create issues if you are having trouble with anything in this repo.

run_entire_process.py

@@ -50,9 +50,9 @@
     polygon = process_city_shapes.simplify_polygon(polygon)
 
     if not args.no_geojsonio:
-        # Open the simplified polygon in a web window to double check correctness
-        geojsonio.display(geopandas.GeoSeries([polygon]).to_json())
-        input("A geojson.io window has been opened with your simplified search polygon, press enter to continue if it "
+        # Create a link to geojsonio for the polygon to double check correctness
+        print(geojsonio.make_url(geopandas.GeoSeries([polygon]).to_json()))
+        input("A geojson.io link has been created with your simplified search polygon, press enter to continue if it "
               "looks okay. If it doesn't, implement a way to edit your polygon and feed it directly to this script :)")
 
     print("Calculating the coordinates of the imagery grid contained within this polygon.")

setup/Dockerfile

@@ -0,0 +1,40 @@
+FROM spdw/base
+LABEL maintainer="Sean Sall <[email protected]>"
+
+ARG fname_environment_yml
+ARG conda_version
+ARG user
+ARG branch
+
+USER $user
+
+RUN mkdir $HOME/repos && \
+    cd $HOME/repos && \
+    git clone https://github.com/typicalTYLER/SolarPanelDataWrangler.git && \
+    git clone https://github.com/typicalTYLER/DeepSolar.git
+
+USER root
+RUN cd /opt && \
+    wget http://download.osgeo.org/libspatialindex/spatialindex-src-1.8.5.tar.gz && \
+    tar -xvf spatialindex-src-1.8.5.tar.gz && \
+    cd spatialindex-src-1.8.5 && \
+    ./configure; make; make install
+
+USER $user
+RUN cd $HOME/repos/SolarPanelDataWrangler &&\
+    git checkout $branch && \
+    cd $HOME/repos/SolarPanelDataWrangler/setup && \
+    conda install conda=$conda_version && \
+    conda env create -f $fname_environment_yml && \
+    cd $HOME
+
+RUN mkdir -p ~/.config/matplotlib && \
+    echo "backend: Agg" > ~/.config/matplotlib/matplotlibrc
+
+RUN cd $HOME/repos/DeepSolar && \
+    mkdir ckpt && \
+    cd ckpt && \
+    wget https://s3-us-west-1.amazonaws.com/roofsolar/inception_classification.tar.gz && \
+    wget https://s3-us-west-1.amazonaws.com/roofsolar/inception_segmentation.tar.gz && \
+    tar xzf inception_classification.tar.gz && \
+    tar xzf inception_segmentation.tar.gz

setup/Dockerfile.base

@@ -0,0 +1,42 @@
+ARG base_image=ubuntu:16.04
+FROM ${base_image}
+LABEL maintainer="Sean Sall <[email protected]>"
+
+ARG conda_version
+ARG user
+
+ENV CONDA_DIRPATH /opt/conda
+ENV PATH $CONDA_DIRPATH/bin:$PATH
+ENV USER_UID 1000
+
+RUN apt-get update && apt-get install -y \
+   bzip2 \
+   cmake \
+   g++ \
+   git \
+   graphviz \
+   libgl1-mesa-glx \
+   libhdf5-dev \
+   rtorrent \
+   sudo \
+   tmux \
+   vim \
+   wget
+
+RUN mkdir -p $CONDA_DIRPATH && \
+    cd $CONDA_DIRPATH && \
+    wget https://repo.continuum.io/miniconda/Miniconda3-${conda_version}-Linux-x86_64.sh && \
+    chmod u+x Miniconda3-${conda_version}-Linux-x86_64.sh && \
+    ./Miniconda3-${conda_version}-Linux-x86_64.sh -b -f -p $CONDA_DIRPATH && \
+    conda install conda=4.6.11 && \
+    conda install python=3.6.8 && \
+    rm Miniconda3-${conda_version}-Linux-x86_64.sh
+
+RUN useradd -m -s /bin/bash -N -u $USER_UID $user && \
+    echo "$user:$user" | chpasswd && adduser $user sudo && \
+    chown -R $user $CONDA_DIRPATH && \
+    echo "$user    ALL=NOPASSWD: ALL" > /etc/sudoers.d/$user && \
+    echo ". /opt/conda/etc/profile.d/conda.sh" >> /home/$user/.bashrc
+
+WORKDIR /home/$user
+USER $user

setup/build_docker_images.sh

@@ -0,0 +1,156 @@
+#!/bin/bash
+#
+# build_docker_images.sh builds the necessary docker images for setting up the 
+# spdw environment and repository, using the Dockerfile.base and Dockerfile 
+# files in this directory.
+
+function usage {
+    echo "Usage: build_docker_image.sh  [--branch] [--conda_version version] [--docker_user username]"
+    echo "                              [--fpath_custom_dockerfile fpath] [--gpu_build] "
+    echo "                              [--rebuild_image (base|final|custom)]"
+    echo ""
+    echo "   --branch                            Branch to use when building, defaults to master."
+    echo "   --conda_version                     Conda version for the Docker images - defaults "
+    echo "                                       to 4.5.11, which is compatible with the repository "
+    echo "                                       YMLs."
+    echo ""
+    echo "    --docker_user                      Username for the Docker images. "
+    echo ""
+    echo "    --fpath_custom_dockerfile fpath:   Build a custom Docker image from a provided "
+    echo "                                       Dockerfile after the Dockerfile.base and "
+    echo "                                       Dockerfile image builds, meant to allow for "
+    echo "                                       further customization of a user's environment. "
+    echo "                                       This Dockerfile must use setup/Dockerfile as "
+    echo "                                       the base image."
+    echo ""
+    echo "    --gpu_build                        Build the base image (setup/Dockerfile.base) using "
+    echo "                                       nvidia/cuda9.0-cudnn7-runtime-ubuntu16.04 as the "
+    echo "                                       base image, instead of ubuntu:16.04 (the default)."
+    echo ""
+    echo "    --rebuild_image                    Rebuild this image and any subsequent images that "
+    echo "                                       use this one as base. Useful if you know that "
+    echo "                                       something inside has changed (e.g. the conda "
+    echo "                                       environment) but Docker won't be able to register "
+    echo "                                       that change. Valid options are (base, final, custom)."
+    exit 1
+}
+
+while [ ! $# -eq 0 ]
+do 
+    case "$1" in
+            --branch)
+                BRANCH=$2
+                shift 2
+                ;;
+            --conda_version)
+                CONDA_VERSION=$2
+                shift 2
+                ;;
+            --docker_user)
+                DOCKER_USER=$2
+                shift 2
+                ;;
+            --fpath_custom_dockerfile)
+                FPATH_CUSTOM_DOCKERFILE=$2
+                shift 2
+                ;;
+            --gpu_build)
+                GPU_BUILD=true
+                shift 1
+                ;;
+            --rebuild_image)
+                REBUILD_IMAGE=$2
+                shift 2
+                ;;
+            --help)
+                usage
+                shift
+                ;;
+    esac
+done
+
+if [ -z "$DOCKER_USER" ]; then
+    echo "--docker_user flag must be specified!"
+    exit 1
+fi
+
+if [ -z "$BRANCH" ]; then
+    echo "--branch not specified, using the default of master."
+    BRANCH=master
+fi
+
+if [ -z "$CONDA_VERSION" ]; then
+    echo "--conda_version not specified, using the default of 4.5.11."
+    CONDA_VERSION=4.5.11
+fi
+
+if [ ! -z "$REBUILD_IMAGE" ]; then
+    if ! [[ "$REBUILD_IMAGE" =~ ^(base|final|custom)$ ]]; then
+        echo "--rebuild_image option \"$REBUILD_IMAGE\" is not one of the "
+        echo "accepted options (base, final, or custom). If you'd like to "
+        echo "delete and remove one of the images, please specify one of "
+        echo "these options."
+        exit 1
+    fi
+
+    if [[ "$REBUILD_IMAGE" == "base" ]]; then
+        echo "--rebuild_image equal to \"base\"... will delete any existing, "
+        echo "spdw/base, spdw/final, and "
+        echo "spdw/custom images to build them anew."
+        DELETE_BASE=true
+        DELETE_FINAL=true
+        DELETE_CUSTOM=true
+    elif [[ "$REBUILD_IMAGE" == "final" ]]; then
+        echo "--rebuild_image equal to \"final\"... will delete any existing, "
+        echo "spdw/final and spdw/custom images to build "
+        echo "them anew."
+        DELETE_FINAL=true
+        DELETE_CUSTOM=true
+    elif [[ "$REBUILD_IMAGE" == "custom" ]]; then
+        echo "--rebuild_image equal to \"custom\"... will delete the "
+        echo "spdw/custom image to build it anew."
+        DELETE_CUSTOM=true
+    fi
+
+    BASE_IMAGE_EXISTS=$(docker images -q spdw/base)
+    FINAL_IMAGE_EXISTS=$(docker images -q spdw/final)
+    CUSTOM_IMAGE_EXISTS=$(docker images -q spdw/custom)
+
+    if [[ "$DELETE_BASE" == "true" ]] && [[ ! -z $BASE_IMAGE_EXISTS ]]; then
+        docker image rm spdw/base
+    fi
+    if [[ "$DELETE_FINAL" == "true" ]] && [[ ! -z $FINAL_IMAGE_EXISTS ]]; then
+        docker image rm spdw/final
+    fi
+    if [[ "$DELETE_CUSTOM" == "true" ]] && [[ ! -z $CUSTOM_IMAGE_EXISTS ]]; then
+        docker image rm spdw/custom
+    fi
+fi
+
+BASE_IMAGE="ubuntu:16.04"
+FNAME_ENVIRONMENT_YML="environment_cpu.yml"
+if [[ ! -z "$GPU_BUILD" ]]; then
+    BASE_IMAGE="nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04"
+    FNAME_ENVIRONMENT_YML="environment_gpu.yml"
+fi
+
+echo "Creating images with docker username $DOCKER_USER and miniconda "
+echo "version $CONDA_VERSION..."
+
+docker build --build-arg user=$DOCKER_USER \
+             --build-arg conda_version=$CONDA_VERSION \
+             --build-arg base_image=$BASE_IMAGE \
+             -t spdw/base --file ./Dockerfile.base ./
+
+docker build --build-arg user=$DOCKER_USER \
+             --build-arg branch=$BRANCH \
+             --build-arg conda_version=$CONDA_VERSION \
+             --build-arg fname_environment_yml=$FNAME_ENVIRONMENT_YML \
+             -t spdw/final --file ./Dockerfile ./
+
+if [ ! -z "$FPATH_CUSTOM_DOCKERFILE" ]; then
+    echo "Building custom Docker image based off of "
+    echo "$FPATH_CUSTOM_DOCKERFILE ..."
+    docker build --build-arg user=$DOCKER_USER \
+                 -t spdw/custom --file $FPATH_CUSTOM_DOCKERFILE ./
+fi