Skip to content

Latest commit

 

History

History
374 lines (279 loc) · 14.4 KB

dl_env_setup.md

File metadata and controls

374 lines (279 loc) · 14.4 KB
Raw

Objective

Provide a comprehensive guide for installing a Linux distro alongside Windows (dual boot) and setting up all necessary Nvidia drivers/libaries. It also follows a virtual enviroment paradigm for handling Python packages and creating environments with different independecies like different Python, CUDA, PyTorch/TensorFlow/NumPy/SciPy/matplotlib etc. versions.

Environment Info

 This guide was tested on the following environment:
  • PC
    • Laptop: HP Omen 16-b1007nv
    • CPU: Intel i7-12700H
    • GPU: NVIDIA GeForce RTX 3070 Ti Laptop GPU
    • RAM: 32 GB
    • Disk: 1 TB
    • OS: Windows 11 Business (21H2), Ubuntu Desktop 22.04.3 LTS

Dual-boot Windows 11 and Ubuntu 22.04.3 LTS on your PC

Preliminary optional step: Disable Secure Boot

While UEFI Secure Boot is supported by Ubuntu, some issues still may rise like not be able to install some 3rd party kernel modules that are not signed. To disable Secure Boot, first make sure to find your BitLocker recovery key. Then follow these steps:

  1. Open the Start menu and open the power menu in the bottom right corner. Then, hold Shift on your keyboard and click Restart. After the PC restarts you will be taken to a blue screen with the following options:
    • Continue
    • Use a device
    • Troubleshoot
    • Turn off your PC
  2. Here, choose Troubleshoot, followed by Advanced options.
  3. Select UEFI Firmware Settings and then Restart.
  4. This will take you to your PC's BIOS settings, where you can turn off Secure Boot. Every BIOS is a little different, so you may have to look around. In the HP Omen laptop that we are using, navigate to the BIOS Setup and then from the upper left main menu select Boot Options. There, you can disable Secure Boot by moving the slider to the left. Lastly, click Exit and make sure that the Save Changes and Exit option is selected before choosing Yes.

Step 1: Create bootable USB drive

 For this step you'll need to have a USB drive that you can install Ubuntu from. To turn your flash drive into installation media, everything on it will be erased, so make sure you've backed up anything you might need.
  1. First download the latest version of Ubuntu Desktop LTS (Long-term Support) from the official website.
  2. Next, you'll need a tool that creates bootable USB drives from ISO files. I am using Rufus for this purpose.
  3. Run Rufus and insert the flash drive you want to use as installation media. Then, click Select and choose the ISO file you downloaded. All the options will be filled in automatically.
  4. Click Start, then click OK in the prompt that shows up.

Step 2: Creating a partition to dual-boot Linux

Next, you'll need to create a second partition on your drive for Linux.

  1. Right-click your Start menu icon (or press Windows key + X on your keyboard) and choose Disk Management. You'll see a list of your drives and partitions.
  2. Right-click your primary partition (the one labeled as Windows C:) and choose Shrink Volume...
  3. Specify the amount of space you want to remove from the partition. This will be limited by the files you already have stored on it, and the amount you enter will be the space you have for your Linux installation. In my case, I decided to allocate 500 GB for the Linux partition so I set the amount of space to shring in MB to 512000.
  4. The space you chose will be deducted from your partition, and it will be listed as unallocated space in the Disk Management window. You can leave it as is and close the window.

Step 3: Installing Ubuntu to dual-boot with Windows 11

Now you're ready to install Linux on your empty partition. If you removed the USB installation media from your PC, insert it again (remove other flash drives), then follow these steps:

  1. Open the Start menu and then click the power button and — while holding Shift on your keyboard — click Restart.
  2. Click Use a device, then choose the USB flash drive you have inserted and your PC will boot from it.
  3. You'll now be in the Ubuntu boot menu. Press Enter to boot into Ubuntu.
  4. Ubuntu and other Linux operating systems let you try it out by booting from the USB drive without installing it. To install Linux on your empty partition, click Install Ubuntu.
  5. Follow the setup experience by first choosing your keyboard layout.
  6. At the Updates and other software section choose to Download updates while isntalling Ubuntu and to Install third-party software for graphics and Wi-Fi hardware and additional media formats, where you will need to set a password for Secure Boot.
  7. Next, at the Installation type select Something else and click Continue. At the next window select the unallocated space that you have created for install Linux (it should be named as free space). Hit Right Click and select add to create the primary partition for Ubuntu. Set the partition's size (I set it to 526871 MB) and the mount point (in my case I set to `/`) and then click OK. The newly created partition should be listed now. Repeat the process by using the rest of the free space partition (10000 MB) in order to create a swap partition. Finally, click Install Now to continue with installation process.
  8. Next you will select your timezone and enter your name and computer's name along with a login password.
  9. Proceed with Ubuntu installation.

Install NVIDIA GPU drivers
  1. Open the command terminal and search for NVIDIA drivers using the apt command:
apt-cache search 'nvidia-driver-' | grep '^nvidia-driver-[[:digit:]]*' | sort -k 3 -t '-'
  1. Next, use the following two commands as you must apply all pending security updates:
sudo apt update
sudo apt upgrade
  1. Now, lets install the latest NVIDIA driver (535 as of September 12th, 2023):
sudo apt install nvidia-driver-535 nvidia-dkms-535
  1. Then, reboot your machine:
sudo reboot
  1. To verify the NVIDIA drivers installation you can use the NVIDIA System Management Interface (SMI)
nvidia-smi

You can also check the version of the installed drivers:

nvidia-smi --query-gpu=driver_version --format=csv

Setup Conda

Conda is an open source package management system and environment management system that runs on Windows, macOS, and Linux. For this guide will be using Miniconda which is a free minimal installer for conda. For installing Miniconda follow the next steps:

  1. Open a command terminal and execute these four commands to quickly and quietly install the latest 64-bit version of the installer and then clean up afterwards. To install a different version or architecture of Miniconda for Linux, change the name of the .sh installer in the wget command.
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
  1. After installing, initialize your newly-installed Miniconda. The following commands initialize for bash and zsh shells:
~/miniconda3/bin/conda init bash
~/miniconda3/bin/conda init zsh

Setup PyTorch/TensorFlow

Install PyTorch

For the purpose of this guideline I will be using the latest version of PyTorch (stable 2.0.1 as of September 12th, 2023)

  1. First, create and activate a separate conda environmet for PyTorch:
conda create -n torch2 python=3.10
conda activate torch2
  1. Then, run the following command to install PyTorch with the appropriate version of CUDA:
conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
  1. To ensure that PyTorch was installed correctly,verify the installation by running the following sample PyTorch code:
python -c "import torch; print(f'{torch.__version__=}'); x = torch.rand(5, 3); print(f'{x=}')"
  1. Additionally, to check if your GPU driver and CUDA/ROCm is enabled and accessible by PyTorch, run the following commands to return whether or not the GPU driver is enabled:
python -c "import torch; print(f'{torch.cuda.is_available()=}'); x = torch.rand(5, 3); x = x.cuda(); print(f'{x.device}')"

Install TensorFlow

This guide is for the latest stable version of TensorFlow (v2.13.0 as of September 12th, 2023).

  1. Create a new conda environment named tf with the following command:
conda create --name tf python=3.9

You can deactivate and activate it with the following commands.

conda deactivate
conda activate tf

Make sure it is activated for the rest of the installation.

  1. Then, install CUDA and cuDNN with conda and pip:
conda install -c conda-forge cudatoolkit=11.8.0
pip install nvidia-cudnn-cu11==8.6.0.163
  1. Next configure the system paths:
mkdir -p $CONDA_PREFIX/etc/conda/activate.d
echo 'CUDNN_PATH=$(dirname $(python -c "import nvidia.cudnn;print(nvidia.cudnn.__file__)"))' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
echo 'export LD_LIBRARY_PATH=$CUDNN_PATH/lib:$CONDA_PREFIX/lib/:$LD_LIBRARY_PATH' >> $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh

The system paths will be automatically configured when you reactivate this conda environment.

  1. Install TensorFlow with pip:
pip install tensorflow==2.13.*
  1. Verify the TensorFlow setup:
python3 -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

If a tensor is returned, you've installed TensorFlow successfully.

  1. Verify the GPU setup:
python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"

If a list of GPU devices is returned, you've installed TensorFlow successfully.

Optional steps

Setup PyCharm

For code development you can use the popular PyCharm IDE

  1. Download PyCharm from here.

  2. Unpack PyCharm using:

tar -xzvf pycharm*

  1. Then, follow the installation instructions found in pycharm-<version>/Install-Linux-tar.txt.

  2. Finally, remove the PyCharm archive:

rm pycharm-*.tar.gz

Install Gnome-tweaks

You may notice, especially in monitors with high resolution, e.g., 3840 x 2160 (4K) or 2560 x 1440 (WQHD), Ubuntu tends to not scale its UI optimally. Thus, fonts and icons may appear very small and not easy to read. Although, Ubunut 22.04.3 allows to scale its UI through the display settings, I prefer to use gnome-tweaks instead. In order to install this tool you will need to go through the following steps:

  1. Make sure that you have the universe repository enabled on your Ubuntu system:
sudo add-apt-repository universe
  1. Next, execute the following command to install Tweak Tool on your Ubuntu 22.04 system:
sudo apt install gnome-tweaks

  1. Finally, run gnome-tweaks through your command terminal.

  2. From the menu on the left select Fonts and then adjust the Scaling Factor accordingly.

Install bpytop

bpytop is a resource monitor that shows usage and stats for processor, memory, disks, network and processes.

  1. Activate the base conda environment:
conda activate
  1. Install bpytop using pip:
pip install bpytop --upgrade
  1. Run bpytop in your command terminal.

Install nvitop

nvitop is an interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.

  1. Activate the base conda environment:
conda activate
  1. Install nvitop using pip:
pip install --upgrade nvitop
  1. Run nvitop in your command terminal.

Resources