From 7e8774af7b1a1f5923c96b946889835d3bddb711 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Wed, 28 Feb 2018 09:37:45 -0500 Subject: [PATCH 01/28] Update BuildDigits.md Modified to include my experience installing DIGITS on Ubuntu server 16.04 with full instructions --- docs/BuildDigits.md | 188 +++++++++++++++++++++++++++++--------------- 1 file changed, 124 insertions(+), 64 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index d0e7c19a1..6060f50dc 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -1,93 +1,153 @@ -# Building DIGITS +# Building DIGITS for Ubuntu Server 16.04 -The following instructions will walk you through building the latest version of DIGITS from source. -**These instructions are for installation on Ubuntu 14.04 and 16.04.** - -Alternatively, see [this guide](BuildDigitsWindows.md) for setting up DIGITS and Caffe on Windows machines. - -Other platforms are not officially supported, but users have successfully installed DIGITS on Ubuntu 12.04, CentOS, OSX, and possibly more. -Since DIGITS itself is a pure Python project, installation is usually pretty trivial regardless of the platform. -The difficulty comes from installing all the required dependencies for Caffe, Torch7 , Tensorflow, and configuring the builds. -Doing so is your own adventure. +I've built Ubuntu Server instances for DIGITS multiple times and have compiled a list of steps that seems to work well for me. Configuring all the dependencies and installing all the prerequisites can be tricky so I thought I'd share it in one place with some notes. ## Prerequisites -You need an NVIDIA driver ([details and instructions](InstallCuda.md#driver)). +You need an NVIDIA driver to begin with of course ([details and instructions](InstallCuda.md#driver)). -Run the following commands to get access to some package repositories: -```sh -# For Ubuntu 14.04 -CUDA_REPO_PKG=http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_8.0.61-1_amd64.deb -ML_REPO_PKG=http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1404/x86_64/nvidia-machine-learning-repo-ubuntu1404_4.0-2_amd64.deb +## Install Ubuntu 16.04 64-bit Server +I do the typical installation using LVM to encrypt the whole drive. No automatic updates. Basic installation packages plus SSH server and really nothing else (at first). -# For Ubuntu 16.04 +### Basic updates/upgrades after installation - modify for your editor of choice +``` +sudo apt-get update +sudo apt-get upgrade +sudo apt-get install emacs +``` +### I like to try and request a specific IP. Insert the following lines into dhclient.conf. If you can get a static IP even better. +``` +emacs /etc/dhcp/dhclient.conf + #try to get a specific ip; dhclient -r -v to request again if it fails on startup + send dhcp-requested-address XXX.XX.XXX.XXX; +``` +### Install CUDA which will pull the latest Nvidia driver. I have always pulled 8-0 which is good for DIGITS 6. +``` +cd /tmp CUDA_REPO_PKG=http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb ML_REPO_PKG=http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb - -# Install repo packages wget "$CUDA_REPO_PKG" -O /tmp/cuda-repo.deb && sudo dpkg -i /tmp/cuda-repo.deb && rm -f /tmp/cuda-repo.deb wget "$ML_REPO_PKG" -O /tmp/ml-repo.deb && sudo dpkg -i /tmp/ml-repo.deb && rm -f /tmp/ml-repo.deb - -# Download new list of packages sudo apt-get update +sudo apt-get install cuda-8-0 +sudo apt-mark hold cuda-8-0 ``` - -## Dependencies - -Install some dependencies with Deb packages: -```sh -sudo apt-get install --no-install-recommends git graphviz python-dev python-flask python-flaskext.wtf python-gevent python-h5py python-numpy python-pil python-pip python-scipy python-tk +### The above, in my experience, pulls the 390 version of the Nvidia drivers. If not, you can do the following. ``` - -Follow [these instructions](BuildCaffe.md) to build Caffe (**required**). - -Follow [these instructions](BuildTorch.md) to build Torch7 (*suggested*). - -Follow [these instructions](BuildTensorflow.md) to build Tensorflow (*suggseted*). - -## Download source - -```sh -# example location - can be customized -DIGITS_ROOT=~/digits -git clone https://github.com/NVIDIA/DIGITS.git $DIGITS_ROOT +sudo apt-get install nvidia-390 +apt-mark hold nvidia-390 ``` - -Throughout the docs, we'll refer to your install location as `DIGITS_ROOT` (`~/digits` in this case), though you don't need to actually set that environment variable. - -## Python packages - -Several PyPI packages need to be installed: -```sh -sudo pip install -r $DIGITS_ROOT/requirements.txt +### Since we are using Nvidia, don't use Nouveau. Make a few changes to not try to boot to the GUI but command line instead. +``` +sudo emacs /etc/modprobe.d/blacklist-nouveau.conf + blacklist nouveau + blacklist lbm-nouveau + alias nouveau off + alias lbm-nouveau off +sudo emacs /etc/default/grub + GRUB_CMDLINE_LINUX_DEFAULT="text" + GRUB_CMDLINE_LINUX="text" +sudo systemctl set-default multi-user.target +sudo reboot now +``` +### After reboot, double check. The first command should show stuff, the second not (if successful). +``` +lsmod | grep nvidia +lsmod | grep nouveau ``` -# [Optional] Enable support for plug-ins +## Dependencies - I choose to put everything in /usr/local but this is up to personal preference. +### Protobuf - in some cases I had trouble with git:// endpoints so I modified to always use https in the second line. +``` +sudo apt-get install autoconf automake libtool curl make g++ git python-dev python-setuptools unzip +git config --global url."https://".insteadOf git:// +export PROTOBUF_ROOT=/usr/local/protobuf +sudo git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT -b '3.2.x' +cd $PROTOBUF_ROOT +sudo ./autogen.sh +sudo ./configure +sudo make "-j$(nproc)" +sudo make install +sudo ldconfig +cd python +sudo python setup.py install --cpp_implementation +``` -DIGITS needs to be installed to enable loading data and visualization plug-ins: +### Caffe. I use Cudnn version 5 though I think it will work with version 6. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. ``` -sudo pip install -e $DIGITS_ROOT +sudo apt-get install libcudnn5 +sudo apt-mark hold libcudnn5 +sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev +emacs ~/.bashrc + export CAFFE_ROOT=/usr/local/caffe +source ~/.bashrc +sudo git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT -b 'caffe-0.15' +pip install -r $CAFFE_ROOT/python/requirements.txt +cd $CAFFE_ROOT +sudo mkdir build +cd build +sudo cmake .. +sudo make -j"$(nproc)" +sudo make install ``` -# Starting the server +### Torch +``` +sudo apt-get install --no-install-recommends git software-properties-common +export TORCH_ROOT=/usr/local/torch +git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive +cd $TORCH_ROOT +sudo ./install-deps +sudo ./install.sh -b +sudo apt-get install --no-install-recommends libhdf5-serial-dev liblmdb-dev +source ~/.bashrc +sudo su +source /usr/local/torch/install/bin/torch-activate +luarocks install tds +luarocks install "https://raw.github.com/deepmind/torch-hdf5/master/hdf5-0-0.rockspec" +luarocks install "https://raw.github.com/Neopallium/lua-pb/master/lua-pb-scm-0.rockspec" +luarocks install lightningmdb 0.9.18.1-1 LMDB_INCDIR=/usr/include LMDB_LIBDIR=/usr/lib/x86_64-linux-gnu + +``` -```sh -./digits-devserver +### Tensorflow. Using 1.2.1 currently per DIGITS recommendations but I've worked with 1.4 as well. Some of this depends on the compute capability of your card. +``` +pip install tensorflow-gpu==1.2.1 ``` -Starts a server at `http://localhost:5000/`. +### DIGITS. Note that I've given control to the local user for the jobs directory and digits log file. This depends on what user will be running the DIGITS server. +``` +DIGITS_ROOT=/usr/local/digits +sudo git clone https://github.com/NVIDIA/DIGITS.git $DIGITS_ROOT +pip install -r $DIGITS_ROOT/requirements.txt +sudo pip install -e $DIGITS_ROOT +sudo apt-get install python-tk +sudo mkdir /usr/local/digits/digits/jobs +sudo chown : /usr/local/digits/digits/jobs +sudo touch /usr/local/digits/digits/digits.log +sudo chown : /usr/local/digits/digits/digits.log ``` -$ ./digits-devserver --help -usage: __main__.py [-h] [-p PORT] [-d] [--version] -DIGITS development server +## Now you should have everything installed and can start DIGITS using the digits-devserver executable. Good idea to test here to be sure. I like to create a startup job as follows. -optional arguments: - -h, --help show this help message and exit - -p PORT, --port PORT Port to run app on (default 5000) - -d, --debug Run the application in debug mode (reloads when the - source changes and gives more detailed error messages) - --version Print the version number and exit +``` +sudo emacs /lib/systemd/system/digits.service + [Unit] + Description=Start Digits + + [Service] + User= + Environment=CAFFE_ROOT=/usr/local/caffe + WorkingDirectory=/usr/local/digits + ExecStart=/bin/bash digits-devserver + Restart=always + RestartSec=30 + + [Install] + WantedBy=multi-user.target +systemctl daemon-reload +systemctl enable digits +systemctl start digits ``` # Getting started From afbd304d0717a27f161cf53b6e4a6bb3b5742e1b Mon Sep 17 00:00:00 2001 From: rwfilice Date: Thu, 1 Mar 2018 10:01:22 -0500 Subject: [PATCH 02/28] Update BuildDigits.md updated cudnn install instructions --- docs/BuildDigits.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 6060f50dc..5253af8af 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -73,10 +73,12 @@ cd python sudo python setup.py install --cpp_implementation ``` -### Caffe. I use Cudnn version 5 though I think it will work with version 6. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. +### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.1 for cuda 8.0 as follows. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. ``` -sudo apt-get install libcudnn5 -sudo apt-mark hold libcudnn5 +# download ubuntu cudnn packages from https://developer.nvidia.com/rdp/cudnn-download to match your version of CUDA and Ubuntu +sudo dpkg -i libcudnn7_7******* #whatever version +sudo dpkg -i libcudnn7_7-dev +sudo dpkg -i libcudnn7_7-doc #if you want docs and code samples sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe From 1bc211ce25fdefa0147a0b1a5fa49bdc35423273 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 5 Mar 2018 16:25:05 -0500 Subject: [PATCH 03/28] Add files via upload Modified the method in which the bounding boxes are passed to the html template such that they are stored in a sorted array rather than an inherently unsorted javascript object. Under the previous version with the unsorted javascript object, the key for the bounding boxes below the image would be incorrectly labeled because the label and color relied on the index in a presumed sorted object. --- .../view/boundingBox/app_begin_template.html | 20 +++++++++---------- digits/extensions/view/boundingBox/view.py | 14 +++++++++---- .../view/boundingBox/view_template.html | 12 ++++++----- 3 files changed, 27 insertions(+), 19 deletions(-) diff --git a/digits/extensions/view/boundingBox/app_begin_template.html b/digits/extensions/view/boundingBox/app_begin_template.html index c59ef717e..4d976e414 100644 --- a/digits/extensions/view/boundingBox/app_begin_template.html +++ b/digits/extensions/view/boundingBox/app_begin_template.html @@ -40,7 +40,8 @@ }; // Add a bounding box draw method to the canvas - CanvasRenderingContext2D.prototype.draw_bbox = function(bbox, color, opacity) { + CanvasRenderingContext2D.prototype.draw_bbox = function(bbox, color, opacity) { + console.log(bbox); this.globalAlpha = opacity; this.fillStyle = color; this.fillRect(bbox[0], @@ -147,7 +148,8 @@ // Draw the image and the bounding boxes with line_width and desaturated by desaturation. $scope.draw_bboxes = function(image, bboxes) { - var canvas = document.getElementById($scope.canvas_id); + console.log("in draw bboxes"); + var canvas = document.getElementById($scope.canvas_id); var context = canvas.getContext('2d'); // first set the size of the canvas @@ -171,14 +173,12 @@ // rectangle fill opacity var opacity = $rootScope.storage.opacity / 100.0; - // draw the bounding boxes - var keys = Object.keys(bboxes); - keys.sort(); - for (var k = 0; k < keys.length; k++) { - var color = $scope.get_color(k); - for (var i = 0; i < bboxes[keys[k]].length; i++) { - context.draw_bbox(bboxes[keys[k]][i], color, opacity); - } + // draw the bounding boxes + console.log(bboxes); + for (var b = 0; b < bboxes.length; b++) { + var color = $scope.get_color(b); + var key = Object.keys(bboxes[b])[0]; + context.draw_bbox(bboxes[b][key], color, opacity); } } diff --git a/digits/extensions/view/boundingBox/view.py b/digits/extensions/view/boundingBox/view.py index 0d26102d4..fad30cc86 100644 --- a/digits/extensions/view/boundingBox/view.py +++ b/digits/extensions/view/boundingBox/view.py @@ -114,15 +114,21 @@ def process_data( # create arrays in expected format keys = inference_data.keys() - bboxes = dict(zip(keys, [[] for x in range(0, len(keys))])) + bboxes = [] for key, outputs in inference_data.items(): + added = False # last number is confidence - bboxes[key] = [list(o) for o in outputs if o[-1] > 0] - self.bbox_count += len(bboxes[key]) + for o in outputs: + if o[-1]>0: + bboxes.append({key: list(o)}) + self.bbox_count += 1 + added = True + if not added: + bboxes.append({key: "null"}) image_html = digits.utils.image.embed_image_html(image) return { 'image': image_html, - 'bboxes': bboxes, + 'bboxes': sorted(bboxes), 'index': self.image_count, } diff --git a/digits/extensions/view/boundingBox/view_template.html b/digits/extensions/view/boundingBox/view_template.html index 467a3aae7..808c40754 100644 --- a/digits/extensions/view/boundingBox/view_template.html +++ b/digits/extensions/view/boundingBox/view_template.html @@ -15,11 +15,13 @@ style="max-width:100%;" resize>

- - - - {[key]} - + + + + + {[key]} + +

From 6cd3c76b19d0d1d4cff509e5c469ec8462089fd3 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 5 Mar 2018 16:58:56 -0500 Subject: [PATCH 04/28] Update app_begin_template.html Removing debug lines --- digits/extensions/view/boundingBox/app_begin_template.html | 3 --- 1 file changed, 3 deletions(-) diff --git a/digits/extensions/view/boundingBox/app_begin_template.html b/digits/extensions/view/boundingBox/app_begin_template.html index 4d976e414..43f892b11 100644 --- a/digits/extensions/view/boundingBox/app_begin_template.html +++ b/digits/extensions/view/boundingBox/app_begin_template.html @@ -41,7 +41,6 @@ // Add a bounding box draw method to the canvas CanvasRenderingContext2D.prototype.draw_bbox = function(bbox, color, opacity) { - console.log(bbox); this.globalAlpha = opacity; this.fillStyle = color; this.fillRect(bbox[0], @@ -148,7 +147,6 @@ // Draw the image and the bounding boxes with line_width and desaturated by desaturation. $scope.draw_bboxes = function(image, bboxes) { - console.log("in draw bboxes"); var canvas = document.getElementById($scope.canvas_id); var context = canvas.getContext('2d'); @@ -174,7 +172,6 @@ var opacity = $rootScope.storage.opacity / 100.0; // draw the bounding boxes - console.log(bboxes); for (var b = 0; b < bboxes.length; b++) { var color = $scope.get_color(b); var key = Object.keys(bboxes[b])[0]; From c32b3e13ac4fd136837607dd93da94b7d8a45ece Mon Sep 17 00:00:00 2001 From: rwfilice Date: Thu, 8 Mar 2018 14:00:53 -0500 Subject: [PATCH 05/28] Update BuildDigits.md --- docs/BuildDigits.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 5253af8af..845235ed2 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -77,8 +77,8 @@ sudo python setup.py install --cpp_implementation ``` # download ubuntu cudnn packages from https://developer.nvidia.com/rdp/cudnn-download to match your version of CUDA and Ubuntu sudo dpkg -i libcudnn7_7******* #whatever version -sudo dpkg -i libcudnn7_7-dev -sudo dpkg -i libcudnn7_7-doc #if you want docs and code samples +sudo dpkg -i libcudnn7-dev +sudo dpkg -i libcudnn7-doc #if you want docs and code samples sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe From 402f94ad087cb7228dea1a692e0b9c59805e908c Mon Sep 17 00:00:00 2001 From: rwfilice Date: Thu, 8 Mar 2018 14:36:54 -0500 Subject: [PATCH 06/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 845235ed2..1027ab988 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -97,7 +97,7 @@ sudo make install ``` sudo apt-get install --no-install-recommends git software-properties-common export TORCH_ROOT=/usr/local/torch -git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive +sudo git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive cd $TORCH_ROOT sudo ./install-deps sudo ./install.sh -b From f4ff58003343bf8aa30ecb71132b4e2f934e4825 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Thu, 8 Mar 2018 15:08:27 -0500 Subject: [PATCH 07/28] Update BuildDigits.md --- docs/BuildDigits.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 1027ab988..1af0f8876 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -105,6 +105,7 @@ sudo apt-get install --no-install-recommends libhdf5-serial-dev liblmdb-dev source ~/.bashrc sudo su source /usr/local/torch/install/bin/torch-activate +git config --global url."https://".insteadOf git:// #because you need to do this as root as well for the following luarocks install tds luarocks install "https://raw.github.com/deepmind/torch-hdf5/master/hdf5-0-0.rockspec" luarocks install "https://raw.github.com/Neopallium/lua-pb/master/lua-pb-scm-0.rockspec" From 4b19be4dbc6e74ada3d3cd7eccd694faf2c9bbfa Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 11:31:54 -0500 Subject: [PATCH 08/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 1af0f8876..786952da8 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -17,7 +17,7 @@ sudo apt-get install emacs ``` ### I like to try and request a specific IP. Insert the following lines into dhclient.conf. If you can get a static IP even better. ``` -emacs /etc/dhcp/dhclient.conf +sudo emacs /etc/dhcp/dhclient.conf #try to get a specific ip; dhclient -r -v to request again if it fails on startup send dhcp-requested-address XXX.XX.XXX.XXX; ``` From 2e67539f3f55cbdab4995a1afc50b0d553bf9d5d Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 11:44:12 -0500 Subject: [PATCH 09/28] Update BuildDigits.md --- docs/BuildDigits.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 786952da8..464b5439f 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -24,18 +24,17 @@ sudo emacs /etc/dhcp/dhclient.conf ### Install CUDA which will pull the latest Nvidia driver. I have always pulled 8-0 which is good for DIGITS 6. ``` cd /tmp -CUDA_REPO_PKG=http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb +sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub +CUDA_REPO_PKG=http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb ML_REPO_PKG=http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/nvidia-machine-learning-repo-ubuntu1604_1.0.0-1_amd64.deb wget "$CUDA_REPO_PKG" -O /tmp/cuda-repo.deb && sudo dpkg -i /tmp/cuda-repo.deb && rm -f /tmp/cuda-repo.deb wget "$ML_REPO_PKG" -O /tmp/ml-repo.deb && sudo dpkg -i /tmp/ml-repo.deb && rm -f /tmp/ml-repo.deb sudo apt-get update -sudo apt-get install cuda-8-0 -sudo apt-mark hold cuda-8-0 -``` -### The above, in my experience, pulls the 390 version of the Nvidia drivers. If not, you can do the following. -``` -sudo apt-get install nvidia-390 -apt-mark hold nvidia-390 +sudo apt-get install cuda-9-0 +sudo apt-mark hold cuda-9-0 +wget https://developer.nvidia.com/compute/cuda/9.0/Prod/patches/2/cuda-repo-ubuntu1604-9-0-local-cublas-performance-update-2_1.0-1_amd64-deb +sudo dpkg -i cuda-repo-ubuntu1604-9-0-local-cublas-performance-update-2_1.0-1_amd64-deb +sudo apt-mark hold nvidia-390 ``` ### Since we are using Nvidia, don't use Nouveau. Make a few changes to not try to boot to the GUI but command line instead. ``` @@ -62,7 +61,7 @@ lsmod | grep nouveau sudo apt-get install autoconf automake libtool curl make g++ git python-dev python-setuptools unzip git config --global url."https://".insteadOf git:// export PROTOBUF_ROOT=/usr/local/protobuf -sudo git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT -b '3.2.x' +sudo git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT cd $PROTOBUF_ROOT sudo ./autogen.sh sudo ./configure @@ -73,12 +72,12 @@ cd python sudo python setup.py install --cpp_implementation ``` -### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.1 for cuda 8.0 as follows. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. +### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. ``` # download ubuntu cudnn packages from https://developer.nvidia.com/rdp/cudnn-download to match your version of CUDA and Ubuntu -sudo dpkg -i libcudnn7_7******* #whatever version -sudo dpkg -i libcudnn7-dev -sudo dpkg -i libcudnn7-doc #if you want docs and code samples +sudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb +sudo dpkg -i libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb +sudo dpkg -i libcudnn7-doc_7.0.5.15-1+cuda9.0_amd64.deb sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe From ac93192fe66368fabc68c657c90d40da434818a3 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 11:46:41 -0500 Subject: [PATCH 10/28] Update BuildDigits.md --- docs/BuildDigits.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 464b5439f..9689e75de 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -1,6 +1,6 @@ # Building DIGITS for Ubuntu Server 16.04 -I've built Ubuntu Server instances for DIGITS multiple times and have compiled a list of steps that seems to work well for me. Configuring all the dependencies and installing all the prerequisites can be tricky so I thought I'd share it in one place with some notes. +I've built Ubuntu Server instances for DIGITS multiple times and have compiled a list of steps that seems to work well for me. Configuring all the dependencies and installing all the prerequisites can be tricky so I thought I'd share it in one place with some notes. It does depend somewhat on the compute capability of your Nvidia card. ## Prerequisites @@ -13,7 +13,7 @@ I do the typical installation using LVM to encrypt the whole drive. No automatic ``` sudo apt-get update sudo apt-get upgrade -sudo apt-get install emacs +sudo apt-get install emacs #sorry vim fanboys ``` ### I like to try and request a specific IP. Insert the following lines into dhclient.conf. If you can get a static IP even better. ``` @@ -21,7 +21,7 @@ sudo emacs /etc/dhcp/dhclient.conf #try to get a specific ip; dhclient -r -v to request again if it fails on startup send dhcp-requested-address XXX.XX.XXX.XXX; ``` -### Install CUDA which will pull the latest Nvidia driver. I have always pulled 8-0 which is good for DIGITS 6. +### Install CUDA which will pull the latest Nvidia driver. I am currently using 9.0 with DIGITS 6.1. ``` cd /tmp sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub @@ -72,13 +72,13 @@ cd python sudo python setup.py install --cpp_implementation ``` -### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. +### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. I'm also now pulling the latest version of caffe (curently 0.16). Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. ``` # download ubuntu cudnn packages from https://developer.nvidia.com/rdp/cudnn-download to match your version of CUDA and Ubuntu sudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb sudo dpkg -i libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb sudo dpkg -i libcudnn7-doc_7.0.5.15-1+cuda9.0_amd64.deb -sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev +sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe source ~/.bashrc @@ -112,9 +112,9 @@ luarocks install lightningmdb 0.9.18.1-1 LMDB_INCDIR=/usr/include LMDB_LIBDIR=/u ``` -### Tensorflow. Using 1.2.1 currently per DIGITS recommendations but I've worked with 1.4 as well. Some of this depends on the compute capability of your card. +### Tensorflow. Using 1.5 currently. Some of this depends on the compute capability of your card - I think must be at least 3.5 for tensorflow 1.5. ``` -pip install tensorflow-gpu==1.2.1 +pip install tensorflow-gpu==1.5 ``` ### DIGITS. Note that I've given control to the local user for the jobs directory and digits log file. This depends on what user will be running the DIGITS server. From 6e344feb4c50286e2ec7f7216b11a62b09f5b4cd Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 11:48:12 -0500 Subject: [PATCH 11/28] Update BuildDigits.md --- docs/BuildDigits.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 9689e75de..3be3003ec 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -78,6 +78,14 @@ sudo python setup.py install --cpp_implementation sudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb sudo dpkg -i libcudnn7-dev_7.0.5.15-1+cuda9.0_amd64.deb sudo dpkg -i libcudnn7-doc_7.0.5.15-1+cuda9.0_amd64.deb +# test your cudnn installation +cd /tmp +cp -r /usr/src/cudnn_samples_v7/ . +cd cudnn_samples_v7/mnistCUDNN/ +make clean +make +./mnistCUDNN #should say "Test Passed!" +rm -rf /tmp/cudnn_samples_v7 sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe From d5be0e2a7696f3b7fdb3e9ad6719d3360057d998 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 14:15:15 -0500 Subject: [PATCH 12/28] Update BuildDigits.md --- docs/BuildDigits.md | 44 ++++++++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 20 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 3be3003ec..54d404ec0 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -15,12 +15,14 @@ sudo apt-get update sudo apt-get upgrade sudo apt-get install emacs #sorry vim fanboys ``` + ### I like to try and request a specific IP. Insert the following lines into dhclient.conf. If you can get a static IP even better. ``` sudo emacs /etc/dhcp/dhclient.conf #try to get a specific ip; dhclient -r -v to request again if it fails on startup send dhcp-requested-address XXX.XX.XXX.XXX; ``` + ### Install CUDA which will pull the latest Nvidia driver. I am currently using 9.0 with DIGITS 6.1. ``` cd /tmp @@ -36,6 +38,7 @@ wget https://developer.nvidia.com/compute/cuda/9.0/Prod/patches/2/cuda-repo-ubun sudo dpkg -i cuda-repo-ubuntu1604-9-0-local-cublas-performance-update-2_1.0-1_amd64-deb sudo apt-mark hold nvidia-390 ``` + ### Since we are using Nvidia, don't use Nouveau. Make a few changes to not try to boot to the GUI but command line instead. ``` sudo emacs /etc/modprobe.d/blacklist-nouveau.conf @@ -49,6 +52,7 @@ sudo emacs /etc/default/grub sudo systemctl set-default multi-user.target sudo reboot now ``` + ### After reboot, double check. The first command should show stuff, the second not (if successful). ``` lsmod | grep nvidia @@ -61,15 +65,15 @@ lsmod | grep nouveau sudo apt-get install autoconf automake libtool curl make g++ git python-dev python-setuptools unzip git config --global url."https://".insteadOf git:// export PROTOBUF_ROOT=/usr/local/protobuf -sudo git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT +sudo -H git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT cd $PROTOBUF_ROOT -sudo ./autogen.sh -sudo ./configure -sudo make "-j$(nproc)" -sudo make install -sudo ldconfig +sudo -H ./autogen.sh +sudo -H ./configure +sudo -H make "-j$(nproc)" +sudo -H make install +sudo -H ldconfig cd python -sudo python setup.py install --cpp_implementation +sudo -H python setup.py install --cpp_implementation ``` ### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. I'm also now pulling the latest version of caffe (curently 0.16). Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. @@ -86,29 +90,29 @@ make clean make ./mnistCUDNN #should say "Test Passed!" rm -rf /tmp/cudnn_samples_v7 -sudo apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev +sudo -H apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe -source ~/.bashrc -sudo git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT -b 'caffe-0.15' -pip install -r $CAFFE_ROOT/python/requirements.txt +source ~/.bashrc #you may need to source or log out and log back in again to get pip added to your PATH +sudo -H git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT +sudo -H pip install -r $CAFFE_ROOT/python/requirements.txt #you seem to need root access at least for some of these libraries cd $CAFFE_ROOT -sudo mkdir build +sudo -H mkdir build cd build -sudo cmake .. -sudo make -j"$(nproc)" -sudo make install +sudo -H cmake .. +sudo -H make -j"$(nproc)" +sudo -H make install ``` ### Torch ``` -sudo apt-get install --no-install-recommends git software-properties-common +sudo -H apt-get install --no-install-recommends git software-properties-common export TORCH_ROOT=/usr/local/torch -sudo git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive +sudo -H git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive cd $TORCH_ROOT -sudo ./install-deps -sudo ./install.sh -b -sudo apt-get install --no-install-recommends libhdf5-serial-dev liblmdb-dev +sudo -H ./install-deps +sudo -H ./install.sh -b +sudo -H apt-get install --no-install-recommends libhdf5-serial-dev liblmdb-dev source ~/.bashrc sudo su source /usr/local/torch/install/bin/torch-activate From 6698c0664252116444a7211165b23104260bd852 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 15:18:00 -0500 Subject: [PATCH 13/28] Update BuildDigits.md --- docs/BuildDigits.md | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 54d404ec0..85b40dd63 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -106,13 +106,16 @@ sudo -H make install ### Torch ``` -sudo -H apt-get install --no-install-recommends git software-properties-common +sudo apt-get install --no-install-recommends git software-properties-common export TORCH_ROOT=/usr/local/torch -sudo -H git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive +sudo git clone https://github.com/torch/distro.git $TORCH_ROOT --recursive cd $TORCH_ROOT -sudo -H ./install-deps -sudo -H ./install.sh -b -sudo -H apt-get install --no-install-recommends libhdf5-serial-dev liblmdb-dev +sudo su +./install-deps +export TORCH_NVCC_FLAGS="-D__CUDA_NO_HALF_OPERATORS__" #something with cuda 9.0? needs to be in root environment +./install.sh -b +exit #exit root +sudo apt-get install --no-install-recommends libhdf5-serial-dev liblmdb-dev source ~/.bashrc sudo su source /usr/local/torch/install/bin/torch-activate @@ -121,20 +124,26 @@ luarocks install tds luarocks install "https://raw.github.com/deepmind/torch-hdf5/master/hdf5-0-0.rockspec" luarocks install "https://raw.github.com/Neopallium/lua-pb/master/lua-pb-scm-0.rockspec" luarocks install lightningmdb 0.9.18.1-1 LMDB_INCDIR=/usr/include LMDB_LIBDIR=/usr/lib/x86_64-linux-gnu - +exit #exit root ``` ### Tensorflow. Using 1.5 currently. Some of this depends on the compute capability of your card - I think must be at least 3.5 for tensorflow 1.5. ``` pip install tensorflow-gpu==1.5 +#test tensorflow if you want +python + import tensorflow as tf + hello = tf.constant('Hello, TensorFlow!') + sess = tf.Session() + print(sess.run(hello)) #should get "Hello, TensorFlow!" ``` ### DIGITS. Note that I've given control to the local user for the jobs directory and digits log file. This depends on what user will be running the DIGITS server. ``` DIGITS_ROOT=/usr/local/digits sudo git clone https://github.com/NVIDIA/DIGITS.git $DIGITS_ROOT -pip install -r $DIGITS_ROOT/requirements.txt -sudo pip install -e $DIGITS_ROOT +sudo -H pip install -r $DIGITS_ROOT/requirements.txt +sudo -H pip install -e $DIGITS_ROOT sudo apt-get install python-tk sudo mkdir /usr/local/digits/digits/jobs sudo chown : /usr/local/digits/digits/jobs From f64d8f9091b49bfa3e162be16dc2a21ff3806533 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 15:36:57 -0500 Subject: [PATCH 14/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 85b40dd63..a48fc1337 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -65,7 +65,7 @@ lsmod | grep nouveau sudo apt-get install autoconf automake libtool curl make g++ git python-dev python-setuptools unzip git config --global url."https://".insteadOf git:// export PROTOBUF_ROOT=/usr/local/protobuf -sudo -H git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT +sudo -H git clone https://github.com/google/protobuf.git $PROTOBUF_ROOT -b '3.2.x' cd $PROTOBUF_ROOT sudo -H ./autogen.sh sudo -H ./configure From 4d369d1660994ae018750a789c9b16c4fbbfff95 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 17:05:08 -0500 Subject: [PATCH 15/28] Update BuildDigits.md --- docs/BuildDigits.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index a48fc1337..cf150e68b 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -168,9 +168,9 @@ sudo emacs /lib/systemd/system/digits.service [Install] WantedBy=multi-user.target -systemctl daemon-reload -systemctl enable digits -systemctl start digits +sudo systemctl daemon-reload +sudo systemctl enable digits +sudo systemctl start digits ``` # Getting started From 225af90aae5b031f7ad8faf8e997b3eb033f7791 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 17:09:47 -0500 Subject: [PATCH 16/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index cf150e68b..4e089245a 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -60,7 +60,7 @@ lsmod | grep nouveau ``` ## Dependencies - I choose to put everything in /usr/local but this is up to personal preference. -### Protobuf - in some cases I had trouble with git:// endpoints so I modified to always use https in the second line. +### Protobuf - in some cases I had trouble with git:// endpoints so I modified to always use https in the second line. 3.2.x is what appears to be compatible with DIGITS - newer versions gave me Caffe errors. ``` sudo apt-get install autoconf automake libtool curl make g++ git python-dev python-setuptools unzip git config --global url."https://".insteadOf git:// From 38d1887454868f1a7ccf00048b52b798d9442f37 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 22:35:14 -0500 Subject: [PATCH 17/28] Update BuildDigits.md --- docs/BuildDigits.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 4e089245a..428266ed5 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -124,6 +124,7 @@ luarocks install tds luarocks install "https://raw.github.com/deepmind/torch-hdf5/master/hdf5-0-0.rockspec" luarocks install "https://raw.github.com/Neopallium/lua-pb/master/lua-pb-scm-0.rockspec" luarocks install lightningmdb 0.9.18.1-1 LMDB_INCDIR=/usr/include LMDB_LIBDIR=/usr/lib/x86_64-linux-gnu +luarocks install "https://raw.githubusercontent.com/ngimel/nccl.torch/master/nccl-scm-1.rockspec" exit #exit root ``` From 6acd81c8cbf3d51d707f022e5745a01e230c94ab Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 22:48:07 -0500 Subject: [PATCH 18/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 428266ed5..e96ea582d 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -94,7 +94,7 @@ sudo -H apt-get install --no-install-recommends build-essential cmake git gfortr emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe source ~/.bashrc #you may need to source or log out and log back in again to get pip added to your PATH -sudo -H git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT +sudo -H git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT -b 'caffe-0.15' sudo -H pip install -r $CAFFE_ROOT/python/requirements.txt #you seem to need root access at least for some of these libraries cd $CAFFE_ROOT sudo -H mkdir build From 5a28777c138b67689dcc970c995d4e5acce215c2 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 22:49:43 -0500 Subject: [PATCH 19/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index e96ea582d..41547029d 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -76,7 +76,7 @@ cd python sudo -H python setup.py install --cpp_implementation ``` -### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. I'm also now pulling the latest version of caffe (curently 0.16). Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. +### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. Detectnet requires caffe version 0.15. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. ``` # download ubuntu cudnn packages from https://developer.nvidia.com/rdp/cudnn-download to match your version of CUDA and Ubuntu sudo dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb From b8db757cae3e51bcc326f088e60aa3a04fae90bb Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 23:08:47 -0500 Subject: [PATCH 20/28] Update BuildDigits.md --- docs/BuildDigits.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 41547029d..07bafc81f 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -162,6 +162,7 @@ sudo emacs /lib/systemd/system/digits.service [Service] User= Environment=CAFFE_ROOT=/usr/local/caffe + Environment=TORCH_ROOT=/usr/local/torch WorkingDirectory=/usr/local/digits ExecStart=/bin/bash digits-devserver Restart=always From 679c426ed4fbbd987be35bfbc9fbafa02c8e6408 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Fri, 9 Mar 2018 23:24:22 -0500 Subject: [PATCH 21/28] Update BuildDigits.md --- docs/BuildDigits.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 07bafc81f..66b5165bd 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -90,10 +90,12 @@ make clean make ./mnistCUDNN #should say "Test Passed!" rm -rf /tmp/cudnn_samples_v7 -sudo -H apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev +sudo -H apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe source ~/.bashrc #you may need to source or log out and log back in again to get pip added to your PATH +sudo -H pip install pip --upgrade +sudo -H pip install numpy --upgrade sudo -H git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT -b 'caffe-0.15' sudo -H pip install -r $CAFFE_ROOT/python/requirements.txt #you seem to need root access at least for some of these libraries cd $CAFFE_ROOT From 11749a54ad2327d65ebd8326206158d55780e9fe Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 12:21:50 -0400 Subject: [PATCH 22/28] Update BuildDigits.md --- docs/BuildDigits.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 66b5165bd..73e18bba7 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -53,10 +53,12 @@ sudo systemctl set-default multi-user.target sudo reboot now ``` -### After reboot, double check. The first command should show stuff, the second not (if successful). +### After reboot, double check. The first command should show stuff, the second not (if successful). The check packages and upgrade. ``` lsmod | grep nvidia lsmod | grep nouveau +sudo apt-get update +sudo apt-get ugprade ``` ## Dependencies - I choose to put everything in /usr/local but this is up to personal preference. From c8de7865ddf060264e70d08038a63f9b08f948c5 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 12:33:37 -0400 Subject: [PATCH 23/28] Update BuildDigits.md --- docs/BuildDigits.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 73e18bba7..b11cf44a7 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -76,6 +76,7 @@ sudo -H make install sudo -H ldconfig cd python sudo -H python setup.py install --cpp_implementation +protoc --version #should say "libprotoc 3.2.1" ``` ### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. Detectnet requires caffe version 0.15. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. From 7e59cb29d0f9d9a32f4dd2c1ed78226a4a60d327 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 12:34:38 -0400 Subject: [PATCH 24/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index b11cf44a7..6969e9827 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -76,7 +76,7 @@ sudo -H make install sudo -H ldconfig cd python sudo -H python setup.py install --cpp_implementation -protoc --version #should say "libprotoc 3.2.1" +protoc --version #should say "libprotoc 3.2.0" ``` ### Caffe. Update: the ubuntu libcudnn libraries are not found when you try and build Caffe. It's possible you could set environment variables to find them, but now I'm building from scratch using cudnn v. 7.0.5 for cuda 9.0 as follows. Detectnet requires caffe version 0.15. Note that you need to add an environment variable to the .bashrc file for whatever user will be running DIGITS so it knows where Caffe is. From f9310a5b2716c9e608b9e81fff24f207941c39f1 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 12:52:50 -0400 Subject: [PATCH 25/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 6969e9827..9692313b8 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -93,7 +93,7 @@ make clean make ./mnistCUDNN #should say "Test Passed!" rm -rf /tmp/cudnn_samples_v7 -sudo -H apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pip python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev +sudo -H apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe source ~/.bashrc #you may need to source or log out and log back in again to get pip added to your PATH From 4b76c205fdb38c50b8079b32bd61aaa7a2b24fc4 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 12:53:10 -0400 Subject: [PATCH 26/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 9692313b8..018c5e6c5 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -96,7 +96,7 @@ rm -rf /tmp/cudnn_samples_v7 sudo -H apt-get install --no-install-recommends build-essential cmake git gfortran libatlas-base-dev libboost-filesystem-dev libboost-python-dev libboost-system-dev libboost-thread-dev libgflags-dev libgoogle-glog-dev libhdf5-serial-dev libleveldb-dev liblmdb-dev libopencv-dev libsnappy-dev python-all-dev python-dev python-h5py python-matplotlib python-numpy python-opencv python-pil python-pydot python-scipy python-skimage python-sklearn libnccl-dev libboost-regex-dev libturbojpeg libopenblas-dev emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe -source ~/.bashrc #you may need to source or log out and log back in again to get pip added to your PATH +source ~/.bashrc sudo -H pip install pip --upgrade sudo -H pip install numpy --upgrade sudo -H git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT -b 'caffe-0.15' From 4c41907cb65fbca22fad8ab3dff65c4ca0622dc0 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 12:58:23 -0400 Subject: [PATCH 27/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index 018c5e6c5..d28ad8be9 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -98,9 +98,9 @@ emacs ~/.bashrc export CAFFE_ROOT=/usr/local/caffe source ~/.bashrc sudo -H pip install pip --upgrade -sudo -H pip install numpy --upgrade sudo -H git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT -b 'caffe-0.15' sudo -H pip install -r $CAFFE_ROOT/python/requirements.txt #you seem to need root access at least for some of these libraries +sudo -H pip install numpy --upgrade cd $CAFFE_ROOT sudo -H mkdir build cd build From 66ea097c063e8a6d32566e27c7b2fd95f6592c53 Mon Sep 17 00:00:00 2001 From: rwfilice Date: Mon, 12 Mar 2018 13:43:29 -0400 Subject: [PATCH 28/28] Update BuildDigits.md --- docs/BuildDigits.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/BuildDigits.md b/docs/BuildDigits.md index d28ad8be9..e0096b7e3 100644 --- a/docs/BuildDigits.md +++ b/docs/BuildDigits.md @@ -135,7 +135,7 @@ exit #exit root ### Tensorflow. Using 1.5 currently. Some of this depends on the compute capability of your card - I think must be at least 3.5 for tensorflow 1.5. ``` -pip install tensorflow-gpu==1.5 +sudo -H pip install tensorflow-gpu==1.5 #test tensorflow if you want python import tensorflow as tf