pyEM.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# py-EM\n",
    "\n",
    "### a Python module to interact with SerialEM to enable automated Transmission Electron Microscopy\n",
    "\n",
    "## Table of contents\n",
    "1. [Installation instructions](#installation)\n",
    "\n",
    "  - How to use this jupyter notebook\n",
    "  \n",
    "  - [Install necessary dependencies in Python](#install1)\n",
    "  \n",
    "  - [install, update py-EM](#install2) \n",
    "\n",
    "\n",
    "2. [Getting started](#howtos)\n",
    " \n",
    "  - [Tutorial 1: Sort a Navigator](#tutorial1)\n",
    "  \n",
    "  - [Tutorial 2: Import pixel coordinates](#tutorial2)\n",
    "  \n",
    "  - [Tutorial 3: Duplicate Navigator entries](#tutorial3) \n",
    "  \n",
    "\n",
    "3. [Setting up Integration into SeralEM's GUI](https://git.embl.de/schorb/pyem/blob/master/doc/serialemtools.md)\n",
    "\n",
    "4. [Setting up KNIME](https://git.embl.de/schorb/pyem/blob/master/doc/knime.md)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation instructions<a name=\"installation\"></a>\n",
    "\n",
    "### Prepare your PC for the use of Python and py-EM\n",
    "\n",
    "- first we will install Python and then use it to fetch the packages needed for emTools to work.\n",
    "- Install Anaconda from [here](https://www.anaconda.com/download/ \"Download Anaconda\"). EmTools has been tested to work with both Python 2 and 3. Version 3 is recommended as 2.7 is approaching the end of its support frame. Choose the appropriate OS.\n",
    "- Start the Anaconda [Navigator](http://docs.anaconda.com/anaconda/user-guide/getting-started/ \"Getting started with Anaconda\").\n",
    "- as explained in [section 4](http://docs.anaconda.com/anaconda/user-guide/getting-started/#run-python-in-a-jupyter-notebook \"Run Python in a Jupyter Notebook\"), install and start a Jupyter notebook.\n",
    "- Download this [tutorial file](https://git.embl.de/schorb/pyem/raw/master/pyEM.ipynb?inline=false) (use \"save link as...\" if it is displayed in your browser), open it in your Jupyter session and follow the instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "### How to use this jupyter notebook\n",
    "\n",
    "In this document there are text blocks and code blocks that can be edited and executed.\n",
    "In order to run a code block, click into it to activate and then press `CTRL+ Enter`. There is a Python logo at the top right of the notebook that indicates whether there are still operations running.\n",
    "\n",
    "### Install necessary dependencies in Python<a name=\"install1\"></a>\n",
    "\n",
    "The following code block uses Anaconda to fetch necessary packages. Start the code by pressing `CTRL+ Enter` inside the code block.\n",
    "\n",
    "On Windows, if you installed Anaconda for all users, you need to run jupyter through Anaconda Navigator with administrative rights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# this will load the package needed for python package management\n",
    "import conda.cli\n",
    "\n",
    "# this will install the dependencies\n",
    "conda.cli.main('conda', 'install',  '-y', 'numpy', 'scipy', 'pandas', 'scikit-image', 'git', 'pip')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## install/update py-EM <a name=\"install2\"></a>\n",
    "\n",
    "Now we can install py-EM using pip, another package manager for Python. Start by pressing `CTRL+ Enter` inside the code block. \n",
    "\n",
    "To update to the current version, simply execute this code block anytime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this will load the pip package needed for python package management\n",
    "from pip._internal import main as pipmain\n",
    "\n",
    "# this will install the current version of py-EM\n",
    "pipmain(['install','git+https://git.embl.de/schorb/pyem'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "\n",
    "\n",
    "## Getting started <a name=\"howtos\"></a>\n",
    "\n",
    "\n",
    "\n",
    "### Tutorial 1: Sort a navigator file<a name=\"tutorial1\"></a>\n",
    "\n",
    "Have a look at the files in the directory [\"`tutorials/Tutorial1-sorting`\"](https://git.embl.de/schorb/pyem/tree/master/tutorials/Tutorial1-sorting). If you open the navigator file `nav1.nav` in SerialEM, you will get the following:\n",
    "\n",
    "#### the Navigator\n",
    "![The navigator](https://git.embl.de/schorb/pyem/raw/master/doc/images/nav1_1.png)\n",
    "\n",
    "If you double-click the `Map` item, you will see this:\n",
    "#### the Map\n",
    "![The cells](https://git.embl.de/schorb/pyem/raw/master/doc/images/nav1.png)\n",
    "\n",
    "We have a series of points for each cell. However, they are not listed such that all points lying within one cell are acquired in a row.\n",
    "\n",
    "Let's use emTools to create a sorted Navigator file.\n",
    "\n",
    "\n",
    "#### Reading in the Navigator\n",
    "\n",
    "First, we will import the py-EM module and thus make its functions available for the Python script.\n",
    "Then we will read in the Navigator file. When you have read and understood what each line of the code below does, press `CTRL + Enter` inside the code block to execute it.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the py-EM module and make its functions available\n",
    "import pyEM as em\n",
    "\n",
    "# load the navigator file\n",
    "navfile = 'tutorials/Tutorial1-sorting/nav1.nav'\n",
    "navlines = em.loadtext(navfile)\n",
    "\n",
    "# now we have a list of strings for each row of the file, let's look at the first 10.\n",
    "print('first 10 lines of the navigator: \\n')\n",
    "print(navlines[0:10])\n",
    "print('\\n----\\n')\n",
    "\n",
    "\n",
    "# let's convert them into a list of dictionaries with one entry for each navigator Item\n",
    "allitems = em.fullnav(navlines)\n",
    "# Let look at one item\n",
    "print('Item number 7 from the navigator file: \\n')\n",
    "print(allitems[6])\n",
    "print('\\n----\\n')\n",
    "\n",
    "# Each entry for this item is associated with a dedicated dictionary key. For example the Stage Coordinates\n",
    "print('Stage coordinates of item number 4: \\n')\n",
    "print(allitems[3]['StageXYZ'])\n",
    "print('\\n----\\n')\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "#### Now, let's find out how to order a navigator list using the function `ordernav`.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new list of navigator entries using the ordernav function\n",
    "newnav = em.ordernav(allitems)\n",
    "\n",
    "print('Let\\'s look at the first points in the list...')\n",
    "print(newnav[1]['# Item'])\n",
    "print(newnav[2]['# Item'])\n",
    "print(newnav[3]['# Item'])\n",
    "print(newnav[4]['# Item'])\n",
    "print(newnav[5]['# Item'])\n",
    "print(newnav[6]['# Item'])\n",
    "print(newnav[7]['# Item'])\n",
    "\n",
    "# and now we need to write the ordered navigator file.\n",
    "\n",
    "# create new file by copying the header of the input file\n",
    "newnavf = navfile[:-4] + '_sorted.nav'\n",
    "nnf = open(newnavf,'w')\n",
    "nnf.write(\"%s\\n\" % navlines[0])\n",
    "nnf.write(\"%s\\n\" % navlines[1])\n",
    "\n",
    "# fill the new file   \n",
    "for nitem in newnav: \n",
    "    out = em.itemtonav(nitem,nitem['# Item'])\n",
    "    for item in out: nnf.write(\"%s\\n\" % item)\n",
    "            \n",
    "nnf.close()\n",
    "\n",
    "# done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "and this is what we get...\n",
    "\n",
    "![The sorted navigator](https://git.embl.de/schorb/pyem/raw/master/doc/images/nav1_sorted.png)\n",
    "\n",
    "\n",
    "----\n",
    "\n",
    "#### Batch process parameters of navigator items\n",
    "\n",
    "Let's now also try and see if we can change some of the general parameters of these navigator items, like for example z-height, color or the 'Acquire' flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new file by copying the header of the input file\n",
    "newnavf = navfile[:-4] + '_playground.nav'\n",
    "nnf = open(newnavf,'w')\n",
    "nnf.write(\"%s\\n\" % navlines[0])\n",
    "nnf.write(\"%s\\n\"% navlines[1])\n",
    "\n",
    "# create a new list to collect ouput navigator items.\n",
    "outnav=list()\n",
    "\n",
    "# add the map. This we don't want to modify\n",
    "outnav.append(newnav[0])\n",
    "\n",
    "# Then modify and add the rest...\n",
    "\n",
    "for item in newnav[1:] :\n",
    "    item['StageXYZ'][2] = '31.415'\n",
    "    item['Color'] = ['3']\n",
    "    item['Acquire'] = ['1']\n",
    "    outnav.append(item)\n",
    "\n",
    "# fill the new file   \n",
    "for nitem in outnav: \n",
    "    out = em.itemtonav(nitem,nitem['# Item'])\n",
    "    for item in out: nnf.write(\"%s\\n\" % item)\n",
    "            \n",
    "nnf.close()\n",
    "\n",
    "# done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "and we get...\n",
    "\n",
    "![The modified navigator](https://git.embl.de/schorb/pyem/raw/master/doc/images/nav1_playground.png)\n",
    "\n",
    "\n",
    "yellow points at the new height that are set to be acquired.\n",
    "\n",
    "----\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Tutorial 2: Import pixel coordinates<a name=\"tutorial2\"></a>\n",
    "\n",
    "This tutorial will give an idea on how to import pixel cooedinates to a map.\n",
    "\n",
    "In SerialEM, open a new empty Navigator by choosing `Navigator > Open` in the menu.\n",
    "\n",
    "Then we import a map, this can be any kind of image, maybe from a light microscope if you're doing CLEM. Let's use the SerialEM logo from [this directory](https://git.embl.de/schorb/pyem/tree/master/tutorials/Tutorial2-coordinates). Therefore, click `Navigator > Import Map` and choose a target registration, this is simply the index of the coordinate frame.\n",
    "\n",
    "![Import Map](https://git.embl.de/schorb/pyem/raw/master/doc/images/import_map.png)\n",
    "![Import Map1](https://git.embl.de/schorb/pyem/raw/master/doc/images/import_map2.png)\n",
    "\n",
    "The pixel coordinates of the holes on the top left are stored in [this file](https://git.embl.de/schorb/pyem/tree/master/tutorials/Tutorial2-coordinates/holes.txt). Namely:\n",
    "\n",
    "| x   | y  |\n",
    "| --- |---:|\n",
    "| 13|13|\n",
    "|32|13|\n",
    "|51|13|\n",
    "|13|32|\n",
    "|32|32|\n",
    "|13|51|\n",
    "\n",
    "\n",
    "We now would like to populate our Navigator with those.\n",
    "\n",
    "In order to tell emTools which map to process, we mark it for acquisition.\n",
    "\n",
    "![Acquire Map](https://git.embl.de/schorb/pyem/raw/master/doc/images/acquire.png)\n",
    "\n",
    "Then save the navigator as `nav.nav`.\n",
    "\n",
    "Now, let's import those into our navigator.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the py-EM module and make its functions available\n",
    "import pyEM as em\n",
    "\n",
    "# load the navigator file\n",
    "navfile = 'tutorials/Tutorial2-coordinates/nav.nav'\n",
    "navlines = em.loadtext(navfile)\n",
    "\n",
    "allitems = em.fullnav(navlines)\n",
    "\n",
    "# find the navigator items with active 'Acquire' flag\n",
    "# first find items that have an 'Acquire' entry\n",
    "acq = filter(lambda item:item.get('Acquire'),allitems)\n",
    "acq = list(filter(lambda item:item['Acquire']==['1'],acq))\n",
    "\n",
    "# in this case, we have only one item\n",
    "mapitem = acq[0]\n",
    "\n",
    "# find the Registration of the map as the points shall go into the same\n",
    "regis = mapitem['Regis']\n",
    "\n",
    "# print('The target registration is '+ regis[0])\n",
    "\n",
    "# import the coordinates\n",
    "importlines =  em.loadtext('tutorials/Tutorial2-coordinates/holes.txt')\n",
    "\n",
    "# create a navigator\n",
    "newitems=list()\n",
    "\n",
    "#remove the Acquire flag from the map item\n",
    "mapitem['Acquire'] = '0'\n",
    "\n",
    "newitems.append(mapitem)\n",
    "\n",
    "\n",
    "# fill the list with new point items\n",
    "for idx,line in enumerate(importlines):\n",
    "    \n",
    "    #fill the relevant fields of the navigator item to create a point with the given label\n",
    "    point = em.pointitem('Point '+str(idx+1),regis)    \n",
    "    \n",
    "    #add the information on which map to place the point\n",
    "    point['DrawnID'] = mapitem['MapID']\n",
    "    \n",
    "    #split the coordinates by the comma separator\n",
    "    coords = line.split(',')    \n",
    "          \n",
    "    # we need to change the y coordinate because of the convention that SerialEM uses\n",
    "    ymapsize = int(mapitem['MapWidthHeight'][1])\n",
    "    \n",
    "    ynew = ymapsize - int(coords[1])\n",
    "    \n",
    "    coords[1] = str(ynew)\n",
    "    \n",
    "    #add the coordinates, we need an additional z-value for the CoordsInMap entry\n",
    "    point['PtsX'] = [coords[0]]\n",
    "    point['PtsY'] = [coords[1]]\n",
    "    \n",
    "    coords.append('0')\n",
    "    point['CoordsInMap'] = coords\n",
    "    \n",
    "    newitems.append(point)\n",
    "\n",
    "# create new file by copying the header of the input file\n",
    "newnavf = navfile[:-4] + '_with_points.nav'\n",
    "nnf = open(newnavf,'w')\n",
    "nnf.write(\"%s\\n\" % navlines[0])\n",
    "nnf.write(\"%s\\n\" % navlines[1])\n",
    "\n",
    "# fill the new file   \n",
    "for nitem in newitems:\n",
    "    out = em.itemtonav(nitem,nitem['# Item'])\n",
    "    for item in out: nnf.write(\"%s\\n\" % item)\n",
    "            \n",
    "nnf.close()\n",
    "\n",
    "# done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "and with this we get as a result:\n",
    "![import coordinates](https://git.embl.de/schorb/pyem/raw/master/doc/images/importcoords.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "----\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Tutorial 3: Duplicate Navigator entries<a name=\"tutorial3\"></a>\n",
    "\n",
    "This tutorial will give an idea on how to duplicate navigator entries into a new registration. This is needed when propagating items to the following section.\n",
    "\n",
    "In SerialEM, set the items you like to duplicate to \"Acquire\".\n",
    "\n",
    "![Acquire Map](https://git.embl.de/schorb/serialem_tools/raw/master/doc/images/acquire.png)\n",
    "\n",
    "Then run the following script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the py-EM module and make its functions available\n",
    "import pyEM as em\n",
    "\n",
    "# load the navigator file\n",
    "navfile = 'E:\\\\Data\\\\Isabella\\\\4091_181216\\\\grid-1\\\\s3_sorted.nav'\n",
    "navlines = em.loadtext(navfile)\n",
    "\n",
    "allitems = em.fullnav(navlines)\n",
    "\n",
    "\n",
    "# we will duplicate all items tagged with 'Acquire'\n",
    "# this is the default selection when using duplicate_items\n",
    "\n",
    "dup_items = em.duplicate_items(allitems)\n",
    "\n",
    "# create new file by copying the header of the input file\n",
    "newnavf = navfile[:-4] +'_duplicates.nav'\n",
    "nnf = open(newnavf,'w')\n",
    "nnf.write(\"%s\\n\" % navlines[0])\n",
    "nnf.write(\"%s\\n\" % navlines[1])\n",
    "\n",
    "# fill the new file   \n",
    "for nitem in dup_items:\n",
    "    out = em.itemtonav(nitem,nitem['# Item'])\n",
    "    for item in out: nnf.write(\"%s\\n\" % item)\n",
    "            \n",
    "nnf.close()\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}