Correspondence to: [email protected]
This tool was designed to compare two segmentation maps in the efforts to design an unbaised ground truth segmentation. The app comparison starts by using the argmax(Intersection \ Union) for each object in the two segementations. Each object is then displayed to the user on at a time so that they can compare the two segmentation approaches. The user can select one of the two algorithm results or draw their own annotation for each cell. The user can also add cells as they see fit.
There are two ways to launch the tool.
- Through MATLAB:
- check out the repository
- navigate to the repo. Open the
SegTool.mlapp
. The app designer will open. - In the task bar click the green run arrow.
- NOTE: This is also the window to edit the app. There are two views the
Desgn View
and theCode View
. You can toggle the views in the top right of the app designer window.
- NOTE: This is also the window to edit the app. There are two views the
- Install the tool and open from the icon.
- download the installer.exe located here
- double click on the installer.exe to launch it
- NOTE: If the installer fails this is typically b/c of a failed MATLAB runtime install. Download and install MATALB Runtime_R2020a_win64 separately, then retry the
SegTool
installer.
- 1. Description
- 2. Getting Started
- 3. Contents
- 4. Background
- 5. File Structure
- 6. Usage
- 7. Saving Output
- 8. Combing Ground Truth Results
- 9. Settings
Robust cell segmentation in mIF imagery is vital for assessing the performance of different biomarkers in the TME. However, commonly available software for segmentation is both inefficient and inaccurate. We aim to develop new algorithms for cell segmentation, either through machine learning or neural network methods. In order to train a neural network and assess the accuracy of any new approach to segmentation we will need a large dataset of ‘ground truth’ annotated cell objects. The manual annotation of every cell in hundreds of images required for a comprehensive ‘ground truth’ dataset is an exhaustive and time consuming process.
Here we present a tool to aid in the development of such a ‘ground truth’ dataset for mIF images to alleviate some of these travails. The tool was designed to compare cells identified by two segmentation algorithms and prompts users to select which segmentation algorithm had better performance. In the initial stage, the tool displays matching cell pairs from both alogrithms, determined by per pixel overlap, to the user one at a time. The user is blinded to which segmentation algorithm was used and can add or remove cells with proximity to the current cell. In a final stage, users can look over the entire image to edit the segmentation either by paning the image or by using a smaller moving window.
The tool was initially designed to compare cells from a super-pixel approach to segmentation, developed by Dr. Seyoun Park, to cells identified by the so-called ‘multi-pass’ inForm cell segmentation used as part of the initial AstroPath publication. As a result we use 'superpixel' and 'inform_data' (or abbreviations SP and IF) to denote the two algorithms throughout. However, any pair of multi-layer tiff label matrix outputs can be used for editing/comparing segmentation by changing values in the settings window - see section 9 for more details.
This document begins with a description on the workflow steps take in the UI, then designates the necessary directory structure of the image outputs, provides a step-by-step instruction for using the tool, and finally defines the comparison output files.
The program relies that the images files are set up in a unique file structure which is modeled after the Clinical Specimen directory structure, in order to find the corresponding inform images. In summary, there should be two adjacent directories for a given slide as follows:
Each folder should contain their own respective ‘Component_Tiffs’ folder, the images for each algorithm should reside in their respective folder.
As a more distinct definition for this structure we first split the directory tree into 4 main parts which will be labeled “root”, “slideID”, ”SP_tree”, and ”IF_tree”.
<root>\<slideID>\<SP_tree>\<SP_Image> OR <root>\<slideID>\<IF_tree>\<IF_Image>
EX. “\bki04\Segmentation\TMAs\Liver_TMA_145_23_01.30.2020\superpixel\Component_Tiffs\Liver_TMA_145_23_01.30.2020_[6435,55763]_component_data_seg.tif”
<root>: “\\bki04\Segmentation\TMAs”
<slideID>: “Liver_TMA_145_23_01.30.2020”
<SP _tree>: “superpixel\Component_Tiffs”
<SP _Image>: “Liver_TMA_145_23_01.30.2020_[6435,55763]_component_data_seg.tif”
For the corresponding inForm image, replace the <SP_tree> with the <IF_tree> as follows:
EX. “\bki04\Segmentation\TMAs\Liver_TMA_145_23_01.30.2020\inform_data\Component_Tiffs\Liver_TMA_145_23_01.30.2020_[6435,55763]_component_data_w_seg.tif”
<root>: “\\bki04\Segmentation\TMAs”
<slideID>: “Liver_TMA_145_23_01.30.2020”
<IF_tree>: “inform_data\Component_Tiffs”
<IF_Image>: “Liver_TMA_145_23_01.30.2020_[6435,55763]_component_data_w_seg.tif”
Note: Be sure to use the component_data_w_seg.tif for the IF image or the software will throw an error.
- Launch the program either by double clicking on the icon:
or by locating the SegmentationTool.exe (see above in Getting Started). - Click on the ‘Load new image’ button:
- A windows file explorer should open, navigate to either the <SP_tree> or <IF_tree>folder and open the <SP_Image> or <IF_Image> of interest.
- If the file structure was set up appropriately, the program will find the corresponding segmentation output image, otherwise it will throw an error.
- The segmentation overlap in the images will be computed or, if it exists, the corresponding overlap image and .csv file comparison will be loaded for the image.
- The first cell for comparison will appear in the UI as a set of four images.
- The top images will contain only the DAPI signal, the bottom images will contain the DAPI and the Membrane signal
- On the left will be used to display the machine learning or superpixel segmentation and the right will be used to display the inform segmentation
- Select to jump to the final review stage (described in 6.10). Note that this can be selected before an image is loaded, if an image is loaded you may have to select 'Next Cell' for additional action to take place.
Once the first cell appears, the figure title will be populated with the image name and the cell pair count.
- Select one of the 7 options in the classification box on the right panel (only one can be selected at a time and when selected the option will turn green):
- ‘A’: the image segmentation on the left or ‘A’ side panel is more correct
- ‘B’: the image segmentation on the right or ‘B’ side panel is more correct
- ‘A=B’: both segmentations are correct
- ‘Neither’: This cell is thrown out for some other reason, both segmentations failed and a new cell cannot be drawn on
- ‘Artifact’: The segmentation is a result of image artifact and not an actual cell
- : this option allows for drawing a segmentation on the ‘B’ panel DAPI only image. (Drawing explained below)
- : this option allows for drawing a segmentation on the ‘B’ DAPI + Membrane image. (Drawing explained below)
- Select ‘Next cell’
- : this goes back one cell in the numeric ordering (shown in the figure header at the top of the page)
- : this moves forward a single cell and a single cell only in the numeric ordering. This differs from the ‘Next cell’ button by ignoring which cells have already been checked off.
- ‘Jump to cell’: this button jumps to the cell pair entered in the input box beside it. This can be used to ask for confirmation or review the segmentation of a given cell pair.
- The brightness and the contrast of the DAPI and Membrane can be scaled separately. Select the marker of interest, then vary the appropriate parameter with the slider. Only one option can be selected at a time. When the option is selected the button turns green.
- : Toggles the segmentation on and off for all four image stamps. The segmentation show is the original segmentation for either ‘A’ or ‘B’ respectively.
- : Toggles the already applied segmentation on and off for all four image stamps. This shows the reviewed cells or the cells in the joint overlap group (if it has not been removed).
Any combination of these options can be applied to a give cell pair.
- : adds a review flag to this cell. When the segmentation is finished and handed off for statistics or review by another person they can review these cells for edits.
- : Indicates that the cell pair is part of a multi nucleated cell
- : allows the user to add a new cell, see below
- : allows the user to select cells in the top left panel and remove them. Click again to stop removing cells.
- : Moves the cell above or below other cells (when drawing images in the additional windows, this is set after the cell is drawn)
Opens the ‘Add cell window’, the main app will be shaded in and not usable when this window opens. The new window will look like this:
- The first time this option is selected a dialog with directions appears over the window. Click 'OK' to continue.
- Click on and wait for it to turn green in that UI to draw a cell.
- There are two methods for drawing (see section 6.7 for more drawing help)
- left click and hold, then drag the annotation around the cell, and close the cell by right clicking
- left click, release and left click again to add 'waypoints' around the cell, and right click to close the cell.
- To add a new cell, click on again.
- The segmentation and display settings are all the same asin the main app but there are a few additional options:
- To redraw a cell you will first need to delete it by selecting the trash can icon, then selecting that cell.
- The number of cells drawn are recorded in the window header
- Once finished adding cells, select either to accept all drawn cells or to reject drawn cells. A confirmation dialog will open either way.
When one of the drawing options is selected
- Move the cursor over the corresponding image. For the main app window (when selecting between A and B):
- There are two options for drawing the segmentation.
- Click and drag
- Left click and hold
- then drag around the cell to draw (while holding the left mouse button)
- Release the left click and right click to end the segmentation
- Create waypoints
- Left click once on the segmentation
- release
- Then move the mouse and left click again
- Do this all the way around the cell, creating ‘waypoints’
- right click on the mouse to end the segmentation
- Click and drag
- If one of the two segmentations are correct, select that segmentation before rejecting or drawing a new segmentation.
- Since the cells are usually ordered by location, segmentation on adjacent cells may show up later on. Often this means that over-segmented or under-segmented examples are directly next to each other. If one of the approaches correctly defines the over-segmented cell, it is safe to reject the second cell that appears or select the correct version again.
- For example:
When the overlaps are computed, some of the cells are computationally determined to be the same cells, these are the so-called ‘highly overlapping pairs’. It is assumed that these objects, because of their agreement are correctly identifying the cell of interest. A random sample of these pairs are filter back into the cells for review to assess the viability of this criteria for each image. If 20% of these sampled pairs have disagreement (were not defined as ‘A=B’) then this dialog will appear and the rest of these highly overlapped pairs are added back in for the user to review. The cell objects are also removed from the applied segmentation.
- When all cells in an image are reviewed, a prompt will open telling the user that all cell pairs have been review and that it is now time to review the whole image. The prompt will ask the user if the segmentation should be cleaned up. This will run an algorithm that detects if cells in the final segmentation results are occluding each other and removes these 'duplicate' cells.
- A new window will open, as below, which shows the whole image with similar drawing features as the ‘Add cell’ window. You will be able to see the applied segmentation on the whole image and identify any cells that may have been missed by both algorithms.
- Edit annotations by using the trash can icon to remove missegmented cells and the drawing utility to draw new ones. (6.7 has helpful hints on how to draw new cells)
- Select to accept all cells or to reject them.
- : toggle the applied segmentation mask on and off.
- : masks out the area where cells are annotated in black.
- : masks out regions not annotated in a transparent grey.
- : When selected in the 'Display' panel will mask all the cells flagged. When selected in the lower panel, will allow the user to click on a cell and flag it.
- : When selected in the 'Display' panel will mask all the cells marked as multi-nucleated. When selected in the lower panel, will allow the user to click on a cell and mark it as multi-nucleated.
- : Use the X and Y values to move across the image in a grid of 200 x 200 pixels. Regions outside the tile will be grayed out. Set X or Y to 0 to turn off.
There are a few ways to save progress.
- The first way is to select the ‘Save Table’ button at the bottom of the right panel:
- The second way is during the closing of the app. Close the app by clicking the ‘X’ in the upper right corner. A new closing dialog will appear:
- ‘Save & Close’: saves progress and closes the app
- ‘Don’t Save & Close’: will not save progress and closes the app
- ‘Cancel’ or clicking the ‘X’ from this box will cancel the closing dialog and return to the app for segmentation
- The UI will also save the current result if a new image is loaded into the UI When the tool saves, a saving progress bar will open over the UI. The UI will not be usable while the save is operating and will appear greyed out. Wait for the UI to finish saving before closing or logging off.
The tool saves two files, both files are labeled with the file indication ‘comparison_seg_data’ after the image name. The first file is a csv file with 14 column headers, the second is a tiff file with 2 image layers.
- Csv file:
- This file contains information on each cell object in the image
- Column description:
- IF_cellid
- Description: Numeric cellid from the IF_cellid label matrix
- Data type: Uint16
- IF_X_centroid
- Description: X value for the centroid of the IF cell
- Data type: Float32
- IF_Y_centroid
- Description: Y value for the centroid of the IF cell
- Data type: Float32
- SP_paired_w_IF
- Description: the corresponding SP cellid that is paired to this IF cellid
- Data type: Uint16
- SP_cellid
- Description: Numeric cellid from the SP_cellid label matrix
- Data type: Uint16
- SP_X_centroid
- Description: X value for the centroid of the SP cell
- Data type: Float32
- SP_Y_centroid
- Description: Y value for the centroid of the SP cell
- Data type: Float32
- IF_paired_w_SP
- Description: the corresponding IF cellid that is paired to this SP cellid
- Data type: Uint16
- pairid
- Description: the new unique cellid given to cell pairs in the table
- Data type: Uint16
- IF_level
- Description: the corresponding image segmentation type for the InForm segmentation a cell comes from.
- Data type: Uint8
- Opts:
- 1: immune cell segmentation layer
- 2: tumor cell segmentation layer
- joint_overlap
- Description: joint fractional overlap, from multiplying the fractional overlaps of each type or IF_frac * SP_frac
- Data type: Float32
- class_selection
- Description: which segmentation is saved in the final result
- Data type: Uint8
- Opts:
- (0): not yet defined
- (1): Inform type
- (2): super pixel type
- (3): drawn on
- (4): joint overlap over .8 and set NOT to review
- (5): reviewed as both
- (6): added \ new cell
- (-1): neither segmentation chosen and not drawn
- (-2): artifact
- cell_check
- Whether or not that cell will be reviewed
- Data type: Uint8
- Opts:
- (0): do not review (see (4) in class selection)
- (1): review
- (2): joint overlap over .8 and set to review Note: Cells can be repeated for each corresponding cell they overlap over 10% with, so all overlapped cells are included.
- IF_cellid
- TIFF file
- This file contains 2 label matrices to be used in conjunction with the csv file to produce the final segmentation result or for display in the UI
- Data type: Uint16
- Opts:
- Cell objects that were draw in the UI, values correspond to the pairid’s in the table above
- The applied segmentation matrix, a binary mask only used to create a visual display
To rebuild the label matrix one must read in the superpixel segmentation mask and the inform segmentation mask. Select the cells for each pairid using corresponding cell selection value. For IF or SP use the cellid values to located to cells. For drawn cells, use layer 1 of the ‘comparison_seg_data.tif’ image. Currently this is done in numeric pairid order, meaning that if a cell later in the pairs, overlaps with another, it will override the previous pairid.
The segmentation status for annotators is kept in the Segmentation_eval.xlsx document in the notes folder of the repo. Numeric ids can be found in the NumericIDs.csv file. To combine results run gather_simil()
from the QC repo directory on the image directory.
gather_simil(wd, P, imname, N)
Input:
wd
: the directory to the trainingimages where each annotator has a subfoler: 'SegmentationImages_PP' (where PP is the person's intials).P
: cell array containing the extensions after the '_' on annotators annotation folder for each annotator that has completed the image of interest- ex. for SegmentationImages_SS it would be {'SS'}
imname
: the image name up to the image coordinates- ex. 'Liver_TMA_145_23_01.30.2020_[6435,55763]'
N
: cell array containing the numeric ID from the NumericIDs.csv file which will be added as an extension to the final image
Output: a combined image in a wd\..\upkeep\Results folder with the segmentation label images named by the image names and appended with _comparison_seg_data_final_01. Also includes a .csv file with the MultiNuc flag status indicating if the user flagged the cell as being multi-nucleated.
Examples for running this code are found in the t1.m, t2.m, and t3.m files where each of the previous results were run and generated.
To cut the large images into smaller images using the cut_big_image
function in the QC folder. Which takes in a directory with the superpixel and component_tiff subfolders as described in section 6 and an image name as desribed for gather_simil
.
A settings window can be opened by selecting the gears button . One of the options in this window is the 'Segmentation Shading', this can be changed between 0-1 to increase or reduce the shading alpha of the segmentation masks. The rest of the settings indicate variables important when loading the image.
There are specific data type settings for the two segmentations being compared:
- input data: the name of the folder in the folder tree from section 5.
- boundary format: is the label matrix input a mask or a boundary label matrix.
- table abbreviation: indicates the abbreviation applied to the output tables, note that once an image set comparison has started the code specifies the names as column headers. To reload old results these abbreviations must be consistent with the previously used abbreviations.
- boundary layers: a comma separated list of the boundary layers to use in the multi-layer tiff inputs with segmentation. These layers are combined into a single layer. Cells overlapping will be overridden by successive layers so that there is a single layer segmentation mask.
We can also specify if the segmentation/annotations of interest is the membrane or the nucleus and which layers of the image to use for visualization.