Skip to content

Latest commit

 

History

History
103 lines (85 loc) · 2.55 KB

README-NCI.md

File metadata and controls

103 lines (85 loc) · 2.55 KB
Raw

Landshark on the NCI

This README is a quick guide to getting the landshark library up and running in a PBS batch environment that has MPI support. This setup is common in HPC systems such as the NCI (raijin).

The instructions below should apply to both single- and multi-node runs on the NCI. Just set ncpus in the PBS directives in the job submission script accordingly (e.g. ncpus=32 for 2 nodes).

The instructions assume you are using bash shell.

Pre-installation

These instructions currently only work with gcc and not the Intel compiler. Note that on NCI it appears python is compiled against gcc anyway.

  1. Unload the icc compiler from the terminal:
$ module unload intel-cc
$ module unload intel-fc
  1. Load the modules requried for installation and running:
module load python3/3.6.2
module load tensorflow/1.8-cudnn7.1-python3.6
module load gdal/2.2.2 git/2.9.5 gcc/4.9.0 openmpi/3.1.0

(Alternatively, you may wish to add the above lines to your ~/.profile (or ~/ .bashrc))

  1. Now add the following lines to the end of your ~/.profile (or ~/.bashrc):
export PATH=$HOME/.local/bin:$PATH
export VIRTUALENVWRAPPER_PYTHON=/apps/python3/3.6.2/bin/python3
export PYTHONPATH=/home/547/sxb547/.local/lib/python3.6/site-packages:$PYTHONPATH
export LC_ALL=en_AU.UTF-8
export LANG=en_AU.UTF-8
source $HOME/.local/bin/virtualenvwrapper.sh
  1. Install virtualenv and virtualenvwrapper by running the following command on the terminal:
$ pip3 install  --user virtualenv virtualenvwrapper
  1. Refresh your environment by reloading your profile:
$ source ~/.profile
# or 
$ source ~/.bashrc

Installation

  1. Create a new virtualenv for landshark:
$ mkvirtualenv --system-site-packages landshark
  1. Make sure the virtualenv is activated:
$ workon landshark
  1. Clone the landshark repo into your home directory:
$ cd ~
$ git clone [email protected]:data61/landshark.git
  1. Install landshark:
$ cd 
$ pip install -e .[dev]
  1. Once installation has completed, you can run the tests to verify everything has gone correctly:
$ make test

Follow the above by running the integration tests

$ make integration

Updating the Code

To update the code, first make sure you are in the landshark virtual environment:

$ workon landshark

Next, pull the latest commit from the master branch, and install:

$ cd ~/landshark
$ git pull origin
$ pip install -e .[dev]

If the pull and the installation complete successfully, the code is ready to run!