create_conda_env.md

Creating a conda environment on a Virtual Machine (notebooks or otherwise)

The instructions for creating conda environments on EC2 virtual machines are exactly the same as those for Sagemaker notebooks. The reason is that notebooks have conda pre-installed and so don't play well with changing the conda environment within the notebook. Thus, if you are using a notebook, follow all steps from within a terminal, which you can access from the launcher.

1. Create a conda environment

Install mamba

curl -L -O https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh
bash Mambaforge-$(uname)-$(uname -m).sh -b -p $HOME/mambaforge

Export to Path

export PATH="$HOME/mambaforge/bin:$PATH"

From within the notebook you can add Mamba to path with

import os
os.environ["PATH"] += os.pathsep + os.environ["HOME"]+"/Mambaforge/bin"

Create and activate the environment

mamba create -n vcftools -c bioconda vcftools ipykernel -y

You can also create the environment using a yaml file like this: mamba env create -f environment.yml

source activate vcftools

2. Create a custom kernel for your notebook instance

Create the kernel using ipykernel

python -m ipykernel install --user --name=vcftools

If you get a no module named ipykernel, then run pip3 install ipykernel.

Open the kernel

Now you can switch to the kernel either from the launcher

Or, from the top right from within the notebook.

Add bin to path

On AWS we need to add one extra step to add the environment bin to our kernel environment. Although we have a separate kernel, the conda environment does not copy over correctly.

From within the notebook run

os.environ["PATH"] += os.pathsep + os.environ["HOME"]+"/mambaforge/envs/vcftools/bin/"

Notice in the screenshot that vcftools is not available until we add the environment bin to PATH. If you were to then switch back to a previous conda environment, like base, source activate base, vcftools won't be available anymore.