- Discuss your ideas @@ -174,6 +178,8 @@
- From installer to install it without having to use mamba (supported platforms: Windows and Linux, only for CPU users).
Installation
You can find more information on the installation and how to troubleshoot it in the FAQ section.
+From mamba
mamba is a drop-in replacement for conda, but much faster. @@ -250,8 +256,8 @@
From installer
We also provide installers for Linux and Windows:
-
-
- Linux -
- Windows +
- Linux +
- Windows @@ -320,6 +326,8 @@
- Segment Anything was trained with a fixed image size of 1024 x 1024 pixels. Inputs that do not match this size will be internally resized to match it. Hence, applying Segment Anything to a much larger image will often lead to inferior results, because it will be downsampled by a large factor and the objects in the image become too small.
-To address this image we implement tiling: cutting up the input image into tiles of a fixed size (with a fixed overlap) and running Segment Anything for the individual tiles.
-You can activate tiling by passing the parameters
tile_shape, which determines the size of the inner tile andhalo, which determines the size of the additional overlap. --
-
- If you're using the
micro_samGUI you can specify the values for thehaloandtile_shapevia theTile X,Tile Y,Halo XandHalo Yby clicking onEmbeddings Settings.
- - If you're using a python script you can pass them as tuples, e.g.
tile_shape=(1024, 1024), halo=(128, 128). See also the wholeslide_annotator example.
- - If you're using the command line functions you can pass them via the options
--tile_shape 1024 1024 --halo 128 128
- - Note that prediction with tiling only works when the embeddings are cached to file, so you must specify an
embedding_path(-ein the CLI).
- - You should choose the
halosuch that it is larger than half of the maximal radius of the objects your segmenting.
-
- - If you're using the
- The applications pre-compute the image embeddings produced by SegmentAnything and (optionally) store them on disc. If you are using a CPU this step can take a while for 3d data or timeseries (you will see a progress bar with a time estimate). If you have access to a GPU without graphical interface (e.g. via a local computer cluster or a cloud provider), you can also pre-compute the embeddings there and then copy them to your laptop / local machine to speed this up. You can use the command
micro_sam.precompute_embeddingsfor this (it is installed with the rest of the applications). You can specify the location of the precomputed embeddings via theembedding_pathargument. --
-
- If you use the GUI to save or load embeddings, simply specify an
embeddings save path. Existing embeddings are loaded from the specified path or embeddings are computed and the path is used to save them.
-
- - If you use the GUI to save or load embeddings, simply specify an
- Most other processing steps are very fast even on a CPU, so interactive annotation is possible. An exception is the automatic segmentation step (2d segmentation), which takes several minutes without a GPU (depending on the image size). For large volumes and timeseries segmenting an object in 3d / tracking across time can take a couple settings with a CPU (it is very fast with a GPU). -
- You can also try using a smaller version of the SegmentAnything model to speed up the computations. For this you can pass the
model_typeargument and either set it tovit_bor tovit_l(default isvit_h). However, this may lead to worse results.
- - You can save and load the results from the
committed_objects/committed_trackslayer to correct segmentations you obtained from another tool (e.g. CellPose) or to save intermediate annotation results. The results can be saved viaFile -> Save Selected Layer(s) ...in the napari menu (see the tutorial videos for details). They can be loaded again by specifying the corresponding location via thesegmentation_result(2d and 3d segmentation) ortracking_result(tracking) argument.
- - Segment Anything does not work well for very small or fine-grained objects (e.g. filaments). -
- For the automatic segmentation functionality we currently rely on the automatic mask generation provided by SegmentAnything. It is slow and often misses objects in microscopy images. -
- Prompt bounding boxes do not provide the full functionality for tracking yet (they cannot be used for divisions or for starting new tracks). See also this github issue. -
vit_b: Default Segment Anything model with vit-b backbone.vit_t: Segment Anything model with vit-tiny backbone. From the Mobile SAM publication.vit_l_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-l backbone. (zenodo, bioimage.io)
-vit_b_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-b backbone. (zenodo, bioimage.io)
+vit_b_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-b backbone. (zenodo, diplomatic-bug on bioimage.io)vit_t_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-t backbone. (zenodo, bioimage.io)vit_l_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-l backbone. (zenodo, bioimage.io)vit_b_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-b backbone. (zenodo, bioimage.io)
@@ -597,6 +575,216 @@ - Windows:
+
-
+
- Windows 10 Pro, Intel i5 7th Gen, 8GB RAM +
+ - Linux:
+
-
+
- Ubuntu 22.04, Intel i7 12th Gen, 32GB RAM +
+ - Mac:
+
-
+
- macOS Sonoma 14.4.1
+
-
+
- M1 Chip, 8GB RAM +
- M3 Max Chip, 36GB RAM +
+
+ - macOS Sonoma 14.4.1
+
- Segmenting objects in 2d images (using automatic and/or interactive segmentation). +
- Segmenting objects in 3d volumes (using automatic and/or interactive segmentation for the entire object(s)). +
- Tracking objects over time in time-series data. +
- Segmenting objects in a series of 2d / 3d images. +
- (OPTIONAL) You can finetune the Segment Anything /
micro_sammodels on your own microscopy data, in case the provided models do not suffice your needs. One caveat: You need to annotate a few objects before-hand (micro_samhas the potential of improving interactive segmentation with only a few annotated objects) to proceed with the supervised finetuning procedure.
+ - If you are using the
micro_samannotation tools, you can specify the values for thetile_shapeandhalovia thetile_x,tile_y,halo_xandhalo_yparameters in theEmbedding Settingsdrop-down menu.
+ - If you are using the
micro_samlibrary in a python script, you can pass them as tuples, e.g.tile_shape=(1024, 1024), halo=(256, 256). See also the wholeslide annotator example.
+ - If you are using the command line functionality, you can pass them via the options
--tile_shape 1024 1024 --halo 256 256.
+ - You can use the command
micro_sam.precompute_embeddingsfor this (it is installed with the rest of the software). You can specify the location of the precomputed embeddings via theembedding_pathargument.
+ - You can cache the computed embedding in the napari tool (to avoid recomputing the embeddings again) by passing the path to store the embeddings in the
embeddings_save_pathoption in theEmbedding Settingsdrop-down. You can later load the precomputed image embeddings by entering the path to the stored embeddings there as well.
+ - Check out the tutorial notebook on how to fine-tune Segment Anything with our
micro_sam.traininglibrary.
+ - Or check the examples for additional scripts that demonstrate finetuning. +
- If you are not familiar with coding in python at all then you can also use the graphical interface for finetuning. But we recommend using a script for more flexibility and reproducibility. +
- The
micro_samtools for interactive data annotation, built as napari plugin. \n - The
micro_samlibrary to apply Segment Anything to 2d and 3d data or fine-tune it on your data. \n - The
micro_sammodels that are fine-tuned on publicly available microscopy data and that are available on BioImage.IO. \n - Releasing more and better finetuned models. \n
- Integrating parameter efficient training and compressed models for faster fine-tuning. \n
- Improving the 3D segmentation and tracking functionality. \n
Annotation Tools
The annotation tools can be started from the napari plugin menu:
You can find additional information on the annotation tools in the FAQ section.
+Annotator 2D
The 2d annotator can be started by
@@ -459,36 +467,6 @@Finetuning UI
The Configuration option allows you to choose the hardware configuration for training. We try to automatically select the correct setting for your system, but it can also be changed. Please refer to the tooltips for the other parameters.
Tips & Tricks
- --
-
Known limitations
- --
-
Using the Python Library
The python library can be imported via
@@ -534,7 +512,7 @@Training your own model
We also support training an additional decoder for automatic instance segmentation. This yields better results than the automatic mask generation of segment anything and is significantly faster. The notebook explains how to activate training it together with the rest of SAM and how to then use it.
-More advanced examples, including quantitative and qualitative evaluation, of finetuned models can be found in finetuning, which contains the code for training and evaluating our models.
+More advanced examples, including quantitative and qualitative evaluation, of finetuned models can be found in finetuning, which contains the code for training and evaluating our models. You can find further information on model training in the FAQ section.
Finetuned models
@@ -549,7 +527,7 @@Finetuned models
Older Models
We do not recommend to use these models since our new models improve upon them significantly. But we provide the links here in case they are needed to reproduce older segmentation workflows.
+FAQ
+ +Here we provide frequently asked questions and common issues.
+If you encounter a problem or question not addressed here feel free to open an issue or to ask your question on image.sc with the tag micro-sam.
Installation questions
+ +1. How to install micro_sam?
+
+The installation for micro_sam is supported in three ways: from mamba (recommended), from source and from installers. Check out our tutorial video to get started with micro_sam, briefly walking you through the installation process and how to start the tool.
2. I cannot install micro_sam using the installer, I am getting some errors.
+
+The installer should work out-of-the-box on Windows and Linux platforms. Please open an issue to report the error you encounter.
+ +++ +NOTE: The installers enable using
+micro_samwithout mamba or conda. However, we recommend the installation from mamba / from source to use all its features seamlessly. Specifically, the installers currently only support the CPU and won't enable you to use the GPU (if you have one).
3. What is the minimum system requirement for micro_sam?
+
+From our experience, the micro_sam annotation tools work seamlessly on most laptop or workstation CPUs and with > 8GB RAM.
+You might encounter some slowness for $leq$ 8GB RAM. The resources micro_sam's annotation tools have been tested on are:
-
+
Having a GPU will significantly speed up the annotation tools and especially the model finetuning.
+ +4. What is the recommended PyTorch version?
+ +micro_sam has been tested mostly with CUDA 12.1 and PyTorch [2.1.1, 2.2.0]. However, the tool and the library is not constrained to a specific PyTorch or CUDA version. So it should work fine with the standard PyTorch installation for your system.
5. I am missing a few packages (eg. ModuleNotFoundError: No module named 'elf.io). What should I do?
+
+With the latest release 1.0.0, the installation from mamba and source should take care of this and install all the relevant packages for you.
+So please reinstall micro_sam.
6. Can I install micro_sam using pip?
+
+The installation is not supported via pip.
+ +7. I get the following error: importError: cannot import name 'UNETR' from 'torch_em.model'.
+
+It's possible that you have an older version of torch-em installed. Similar errors could often be raised from other libraries, the reasons being: a) Outdated packages installed, or b) Some non-existent module being called. If the source of such error is from micro_sam, then a) is most likely the reason . We recommend installing the latest version following the installation instructions.
Usage questions
+ + + +1. I have some micropscopy images. Can I use the annotator tool for segmenting them?
+ +Yes, you can use the annotator tool for:
+ +-
+
2. Which model should I use for my data?
+ +We currently provide three different kind of models: the default models vit_h, vit_l, vit_b and vit_t; the models for light microscopy vit_l_lm, vit_b_lm and vit_t_lm; the models for electron microscopy vit_l_em_organelles, vit_b_em_organelles and vit_t_em_organelles.
+You should first try the model that best fits the segmentation task your interested in, a lm model for cell or nucleus segmentation in light microscopy or a em_organelles model for segmenting nuclei, mitochondria or other roundish organelles in electron microscopy.
+If your segmentation problem does not meet these descriptions, or if these models don't work well, you should try one of the default models instead.
+The letter after vit denotes the size of the image encoder in SAM, h (huge) being the largest and t (tiny) the smallest. The smaller models are faster but may yield worse results. We recommend to either use a vit_l or vit_b model, they offer the best trade-off between speed and segmentation quality.
+You can find more information on model choice here.
3. I have high-resolution microscopy images, 'micro_sam' does not seem to work.
+ +The Segment Anything model expects inputs of shape 1024 x 1024 pixels. Inputs that do not match this size will be internally resized to match it. Hence, applying Segment Anything to a much larger image will often lead to inferior results, or somethimes not work at all. To address this, micro_sam implements tiling: cutting up the input image into tiles of a fixed size (with a fixed overlap) and running Segment Anything for the individual tiles. You can activate tiling with the tile_shape parameter, which determines the size of the inner tile and halo, which determines the size of the additional overlap.
-
+
++ +NOTE: It's recommended to choose the
+haloso that it is larger than half of the maximal radius of the objects you want to segment.
4. The computation of image embeddings takes very long in napari.
+ +micro_sam pre-computes the image embeddings produced by the vision transformer backbone in Segment Anything, and (optionally) store them on disc. I fyou are using a CPU, this step can take a while for 3d data or time-series (you will see a progress bar in the command-line interface / on the bootom right of napari). If you have access to a GPU without graphical interface (e.g. via a local computer cluster or a cloud provider), you can also pre-compute the embeddings there and then copy them over to your laptop / local machine to speed this up.
-
+
5. Can I use micro_sam on a CPU?
+
+Most other processing steps that are very fast even on a CPU, the automatic segmentation step for the default Segment Anything models (typically called as the "Segment Anything" feature or AMG - Automatic Mask Generation) takes several minutes without a GPU (depending on the image size). For large volumes and time-series, segmenting an object interactively in 3d / tracking across time can take a couple of seconds with a CPU (it is very fast with a GPU).
+ +++ +HINT: All the tutorial videos have been created on CPU resources.
+
6. I generated some segmentations from another tool, can I use it as a starting point in micro_sam?
+
+You can save and load the results from the committed_objects layer to correct segmentations you obtained from another tool (e.g. CellPose) or save intermediate annotation results. The results can be saved via File -> Save Selected Layers (s) ... in the napari menu-bar on top (see the tutorial videos for details). They can be loaded again by specifying the corresponding location via the segmentation_result parameter in the CLI or python script (2d and 3d segmentation).
+If you are using an annotation tool you can load the segmentation you want to edit as segmentation layer and renae it to committed_objects.
7. I am using micro_sam for segmenting objects. I would like to report the steps for reproducability. How can this be done?
+
+The annotation steps and segmentation results can be saved to a zarr file by providing the commit_path in the commit widget. This file will contain all relevant information to reproduce the segmentation.
++ +NOTE: This feature is still under development and we have not implemented rerunning the segmentation from this file yet. See this issue for details.
+
8. I want to segment complex objects. Both the default Segment Anything models and the micro_sam generalist models do not work for my data. What should I do?
+
+micro_sam supports interactive annotation using positive and negative point prompts, box prompts and polygon drawing. You can combine multiple types of prompts to improve the segmentation quality. In case the aforementioned suggestions do not work as desired, micro_sam also supports finetuning a model on your data (see the next section). We recommend the following: a) Check which of the provided models performs relatively good on your data, b) Choose the best model as the starting point to train your own specialist model for the desired segmentation task.
9. I am using the annotation tool and napari outputs the following error: While emmitting signal ... an error ocurred in callback ... This is not a bug in psygnal. See ... above for details.
+
+These messages occur when an internal error happens in micro_sam. In most cases this is due to inconsistent annotations and you can fix them by clearing the annotations.
+We want to remove these errors, so we would be very grateful if you can open an issue and describe the steps you did when encountering it.
10. The objects are not segmented in my 3d data using the interactive annotation tool.
+ +The first thing to check is: a) make sure you are using the latest version of micro_sam (pull the latest commit from master if your installation is from source, or update the installation from conda / mamba using mamba update micro_sam), and b) try out the steps from the 3d annotator tutorial video to verify if this shows the same behaviour (or the same errors) as you faced. For 3d images, it's important to pass the inputs in the python axis convention, ZYX.
+c) try using a different model and change the projection mode for 3d segmentation. This is also explained in the video.
11. I have very small or fine-grained structures in my high-resolution microscopic images. Can I use micro_sam to annotate them?
+
+Segment Anything does not work well for very small or fine-grained objects (e.g. filaments). In these cases, you could try to use tiling to improve results (see Point 3 above for details).
+ +12. napari seems to be very slow for large images.
+ +Editing (drawing / erasing) very large 2d images or 3d volumes is known to be slow at the moment, as the objects in the layers are stored in-memory. See the related issue.
+ +13. While computing the embeddings (and / or automatic segmentation), a window stating: "napari" is not responding. pops up.
+
+This can happen for long running computations. You just need to wait a bit longer and the computation will finish.
+ +Fine-tuning questions
+ +1. I have a microscopy dataset I would like to fine-tune Segment Anything for. Is it possible using 'micro_sam'?
+ +Yes, you can fine-tune Segment Anything on your own dataset. Here's how you can do it:
+ +-
+
2. I would like to fine-tune Segment Anything on open-source cloud services (e.g. Kaggle Notebooks), is it possible?
+ +Yes, you can fine-tune Segment Anything on your custom datasets on Kaggle (and BAND). Check out our tutorial notebook for this.
+ +3. What kind of annotations do I need to finetune Segment Anything?
+ +TODO: explain instance segmentation labels, that you can get them by annotation with micro_sam, and dense vs. sparse annotation (for training without / with decoder)
+ +4. I have finetuned Segment Anything on my microscopy data. How can I use it for annotating new images?
+ +You can load your finetuned model by entering the path to its checkpoint in the custom_weights_path field in the Embedding Settings drop-down menu.
+If you are using the python library or CLI you can specify this path with the checkpoint_path parameter.
5. What is the background of the new AIS (Automatic Instance Segmentation) feature in micro_sam?
+
+micro_sam introduces a new segmentation decoder to the Segment Anything backbone, for enabling faster and accurate automatic instance segmentation, by predicting the distances to the object center and boundary as well as predicting foregrund, and performing seeded watershed-based postprocessing to obtain the instances.
+
6. I have a NVIDIA RTX 4090Ti GPU with 24GB VRAM. Can I finetune Segment Anything?
+ +Finetuning Segment Anything is possible in most consumer-grade GPU and CPU resources (but training being a lot slower on the CPU). For the mentioned resource, it should be possible to finetune a ViT Base (also abbreviated as vit_b) by reducing the number of objects per image to 15.
+This parameter has the biggest impact on the VRAM consumption and quality of the finetuned model.
+You can find an overview of the resources we have tested for finetuning here.
+We also provide a the convenience function micro_sam.training.train_sam_for_configuration that selects the best training settings for these configuration. This function is also used by the finetuning UI.
7. I want to create a dataloader for my data, for finetuning Segment Anything.
+ +Thanks to torch-em, a) Creating PyTorch datasets and dataloaders using the python library is convenient and supported for various data formats and data structures.
+See the tutorial notebook on how to create dataloaders using torch-em and the documentation for details on creating your own datasets and dataloaders; and b) finetuning using the napari tool eases the aforementioned process, by allowing you to add the input parameters (path to the directory for inputs and labels etc.) directly in the tool.
++ +NOTE: If you have images with large input shapes with a sparse density of instance segmentations, we recommend using
+samplerfor choosing the patches with valid segmentation for the finetuning purpose (see the example for PlantSeg (Root) specialist model inmicro_sam).
8. How can I evaluate a model I have finetuned?
+ +TODO: move the content of https://github.com/computational-cell-analytics/micro-sam/blob/master/doc/bioimageio/validation.md here.
+Contribution Guide
-
@@ -869,15 +1057,16 @@
Transfering data to BAND
4.. include:: ../doc/annotation_tools.md 5.. include:: ../doc/python_library.md 6.. include:: ../doc/finetuned_models.md - 7.. include:: ../doc/contributing.md - 8.. include:: ../doc/development.md - 9.. include:: ../doc/band.md -10""" -11import os -12 -13from .__version__ import __version__ -14 -15os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1" + 7.. include:: ../doc/faq.md + 8.. include:: ../doc/contributing.md + 9.. include:: ../doc/development.md +10.. include:: ../doc/band.md +11""" +12import os +13 +14from .__version__ import __version__ +15 +16os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1" diff --git a/micro_sam/__version__.html b/micro_sam/__version__.html index 2462eb9d..25ea674f 100644 --- a/micro_sam/__version__.html +++ b/micro_sam/__version__.html @@ -52,7 +52,7 @@
- 1__version__ = "1.0.0"
+ 1__version__ = "1.0.0post0"
diff --git a/micro_sam/sam_annotator/_widgets.html b/micro_sam/sam_annotator/_widgets.html
index c3c45f6d..fa34fc3f 100644
--- a/micro_sam/sam_annotator/_widgets.html
+++ b/micro_sam/sam_annotator/_widgets.html
@@ -1671,127 +1671,142 @@
1507 settings = _make_collapsible(setting_values, title="Automatic Segmentation Settings")
1508 return settings
1509
-1510 def _run_segmentation_2d(self, kwargs, i=None):
-1511 pbar, pbar_signals = _create_pbar_for_threadworker()
-1512
-1513 @thread_worker
-1514 def seg_impl():
-1515 def pbar_init(total, description):
-1516 pbar_signals.pbar_total.emit(total)
-1517 pbar_signals.pbar_description.emit(description)
-1518
-1519 seg = _instance_segmentation_impl(
-1520 self.with_background, self.min_object_size, i=i,
-1521 pbar_init=pbar_init,
-1522 pbar_update=lambda update: pbar_signals.pbar_update.emit(update),
-1523 **kwargs
-1524 )
-1525 pbar_signals.pbar_stop.emit()
-1526 return seg
-1527
-1528 def update_segmentation(seg):
-1529 if i is None:
-1530 self._viewer.layers["auto_segmentation"].data = seg
-1531 else:
-1532 self._viewer.layers["auto_segmentation"].data[i] = seg
-1533 self._viewer.layers["auto_segmentation"].refresh()
-1534
-1535 worker = seg_impl()
-1536 worker.returned.connect(update_segmentation)
-1537 worker.start()
-1538 return worker
-1539
-1540 # We refuse to run 3D segmentation with the AMG unless we have a GPU or all embeddings
-1541 # are precomputed. Otherwise this would take too long.
-1542 def _allow_segment_3d(self):
-1543 if self.with_decoder:
-1544 return True
-1545 state = AnnotatorState()
-1546 predictor = state.predictor
-1547 if str(predictor.device) == "cpu" or str(predictor.device) == "mps":
-1548 n_slices = self._viewer.layers["auto_segmentation"].data.shape[0]
-1549 embeddings_are_precomputed = (state.amg_state is not None) and (len(state.amg_state) > n_slices)
-1550 if not embeddings_are_precomputed:
-1551 return False
-1552 return True
-1553
-1554 def _run_segmentation_3d(self, kwargs):
-1555 allow_segment_3d = self._allow_segment_3d()
-1556 if not allow_segment_3d:
-1557 val_results = {
-1558 "message_type": "error",
-1559 "message": "Volumetric segmentation with AMG is only supported if you have a GPU."
-1560 }
-1561 return _generate_message(val_results["message_type"], val_results["message"])
-1562
-1563 pbar, pbar_signals = _create_pbar_for_threadworker()
-1564
-1565 @thread_worker
-1566 def seg_impl():
-1567 segmentation = np.zeros_like(self._viewer.layers["auto_segmentation"].data)
-1568 offset = 0
-1569
-1570 def pbar_init(total, description):
-1571 pbar_signals.pbar_total.emit(total)
-1572 pbar_signals.pbar_description.emit(description)
-1573
-1574 pbar_init(segmentation.shape[0], "Segment volume")
-1575
-1576 # Further optimization: parallelize if state is precomputed for all slices
-1577 for i in range(segmentation.shape[0]):
-1578 seg = _instance_segmentation_impl(self.with_background, self.min_object_size, i=i, **kwargs)
-1579 seg_max = seg.max()
-1580 if seg_max == 0:
-1581 continue
-1582 seg[seg != 0] += offset
-1583 offset = seg_max + offset
-1584 segmentation[i] = seg
-1585 pbar_signals.pbar_update.emit(1)
-1586
-1587 pbar_signals.pbar_reset.emit()
-1588 segmentation = merge_instance_segmentation_3d(
-1589 segmentation, beta=0.5, with_background=self.with_background,
-1590 gap_closing=self.gap_closing, min_z_extent=self.min_extent,
-1591 verbose=True, pbar_init=pbar_init,
-1592 pbar_update=lambda update: pbar_signals.pbar_update.emit(1),
-1593 )
-1594 pbar_signals.pbar_stop.emit()
-1595 return segmentation
-1596
-1597 def update_segmentation(segmentation):
-1598 self._viewer.layers["auto_segmentation"].data = segmentation
-1599 self._viewer.layers["auto_segmentation"].refresh()
-1600
-1601 worker = seg_impl()
-1602 worker.returned.connect(update_segmentation)
-1603 worker.start()
-1604 return worker
-1605
-1606 def __call__(self):
-1607 if _validate_embeddings(self._viewer):
-1608 return None
-1609
-1610 if self.with_decoder:
-1611 kwargs = {
-1612 "center_distance_threshold": self.center_distance_thresh,
-1613 "boundary_distance_threshold": self.boundary_distance_thresh,
-1614 "min_size": self.min_object_size,
-1615 }
-1616 else:
-1617 kwargs = {
-1618 "pred_iou_thresh": self.pred_iou_thresh,
-1619 "stability_score_thresh": self.stability_score_thresh,
-1620 "box_nms_thresh": self.box_nms_thresh,
-1621 }
-1622 if self.volumetric and self.apply_to_volume:
-1623 worker = self._run_segmentation_3d(kwargs)
-1624 elif self.volumetric and not self.apply_to_volume:
-1625 i = int(self._viewer.cursor.position[0])
-1626 worker = self._run_segmentation_2d(kwargs, i=i)
-1627 else:
-1628 worker = self._run_segmentation_2d(kwargs)
-1629 _select_layer(self._viewer, "auto_segmentation")
-1630 return worker
+1510 def _empty_segmentation_warning(self):
+1511 msg = "The automatic segmentation result does not contain any objects."
+1512 msg += "Setting a smaller value for 'min_object_size' may help."
+1513 if not self.with_decoder:
+1514 msg += "Setting smaller values for 'pred_iou_thresh' and 'stability_score_thresh' may also help."
+1515 val_results = {"message_type": "error", "message": msg}
+1516 return _generate_message(val_results["message_type"], val_results["message"])
+1517
+1518 def _run_segmentation_2d(self, kwargs, i=None):
+1519 pbar, pbar_signals = _create_pbar_for_threadworker()
+1520
+1521 @thread_worker
+1522 def seg_impl():
+1523 def pbar_init(total, description):
+1524 pbar_signals.pbar_total.emit(total)
+1525 pbar_signals.pbar_description.emit(description)
+1526
+1527 seg = _instance_segmentation_impl(
+1528 self.with_background, self.min_object_size, i=i,
+1529 pbar_init=pbar_init,
+1530 pbar_update=lambda update: pbar_signals.pbar_update.emit(update),
+1531 **kwargs
+1532 )
+1533 pbar_signals.pbar_stop.emit()
+1534 return seg
+1535
+1536 def update_segmentation(seg):
+1537 is_empty = seg.max() == 0
+1538 if is_empty:
+1539 self._empty_segmentation_warning()
+1540
+1541 if i is None:
+1542 self._viewer.layers["auto_segmentation"].data = seg
+1543 else:
+1544 self._viewer.layers["auto_segmentation"].data[i] = seg
+1545 self._viewer.layers["auto_segmentation"].refresh()
+1546
+1547 worker = seg_impl()
+1548 worker.returned.connect(update_segmentation)
+1549 worker.start()
+1550 return worker
+1551
+1552 # We refuse to run 3D segmentation with the AMG unless we have a GPU or all embeddings
+1553 # are precomputed. Otherwise this would take too long.
+1554 def _allow_segment_3d(self):
+1555 if self.with_decoder:
+1556 return True
+1557 state = AnnotatorState()
+1558 predictor = state.predictor
+1559 if str(predictor.device) == "cpu" or str(predictor.device) == "mps":
+1560 n_slices = self._viewer.layers["auto_segmentation"].data.shape[0]
+1561 embeddings_are_precomputed = (state.amg_state is not None) and (len(state.amg_state) > n_slices)
+1562 if not embeddings_are_precomputed:
+1563 return False
+1564 return True
+1565
+1566 def _run_segmentation_3d(self, kwargs):
+1567 allow_segment_3d = self._allow_segment_3d()
+1568 if not allow_segment_3d:
+1569 val_results = {
+1570 "message_type": "error",
+1571 "message": "Volumetric segmentation with AMG is only supported if you have a GPU."
+1572 }
+1573 return _generate_message(val_results["message_type"], val_results["message"])
+1574
+1575 pbar, pbar_signals = _create_pbar_for_threadworker()
+1576
+1577 @thread_worker
+1578 def seg_impl():
+1579 segmentation = np.zeros_like(self._viewer.layers["auto_segmentation"].data)
+1580 offset = 0
+1581
+1582 def pbar_init(total, description):
+1583 pbar_signals.pbar_total.emit(total)
+1584 pbar_signals.pbar_description.emit(description)
+1585
+1586 pbar_init(segmentation.shape[0], "Segment volume")
+1587
+1588 # Further optimization: parallelize if state is precomputed for all slices
+1589 for i in range(segmentation.shape[0]):
+1590 seg = _instance_segmentation_impl(self.with_background, self.min_object_size, i=i, **kwargs)
+1591 seg_max = seg.max()
+1592 if seg_max == 0:
+1593 continue
+1594 seg[seg != 0] += offset
+1595 offset = seg_max + offset
+1596 segmentation[i] = seg
+1597 pbar_signals.pbar_update.emit(1)
+1598
+1599 pbar_signals.pbar_reset.emit()
+1600 segmentation = merge_instance_segmentation_3d(
+1601 segmentation, beta=0.5, with_background=self.with_background,
+1602 gap_closing=self.gap_closing, min_z_extent=self.min_extent,
+1603 verbose=True, pbar_init=pbar_init,
+1604 pbar_update=lambda update: pbar_signals.pbar_update.emit(1),
+1605 )
+1606 pbar_signals.pbar_stop.emit()
+1607 return segmentation
+1608
+1609 def update_segmentation(segmentation):
+1610 is_empty = segmentation.max() == 0
+1611 if is_empty:
+1612 self._empty_segmentation_warning()
+1613 self._viewer.layers["auto_segmentation"].data = segmentation
+1614 self._viewer.layers["auto_segmentation"].refresh()
+1615
+1616 worker = seg_impl()
+1617 worker.returned.connect(update_segmentation)
+1618 worker.start()
+1619 return worker
+1620
+1621 def __call__(self):
+1622 if _validate_embeddings(self._viewer):
+1623 return None
+1624
+1625 if self.with_decoder:
+1626 kwargs = {
+1627 "center_distance_threshold": self.center_distance_thresh,
+1628 "boundary_distance_threshold": self.boundary_distance_thresh,
+1629 "min_size": self.min_object_size,
+1630 }
+1631 else:
+1632 kwargs = {
+1633 "pred_iou_thresh": self.pred_iou_thresh,
+1634 "stability_score_thresh": self.stability_score_thresh,
+1635 "box_nms_thresh": self.box_nms_thresh,
+1636 }
+1637 if self.volumetric and self.apply_to_volume:
+1638 worker = self._run_segmentation_3d(kwargs)
+1639 elif self.volumetric and not self.apply_to_volume:
+1640 i = int(self._viewer.cursor.position[0])
+1641 worker = self._run_segmentation_2d(kwargs, i=i)
+1642 else:
+1643 worker = self._run_segmentation_2d(kwargs)
+1644 _select_layer(self._viewer, "auto_segmentation")
+1645 return worker
@@ -4108,127 +4123,142 @@ Inherited Members
1508 settings = _make_collapsible(setting_values, title="Automatic Segmentation Settings")
1509 return settings
1510
-1511 def _run_segmentation_2d(self, kwargs, i=None):
-1512 pbar, pbar_signals = _create_pbar_for_threadworker()
-1513
-1514 @thread_worker
-1515 def seg_impl():
-1516 def pbar_init(total, description):
-1517 pbar_signals.pbar_total.emit(total)
-1518 pbar_signals.pbar_description.emit(description)
-1519
-1520 seg = _instance_segmentation_impl(
-1521 self.with_background, self.min_object_size, i=i,
-1522 pbar_init=pbar_init,
-1523 pbar_update=lambda update: pbar_signals.pbar_update.emit(update),
-1524 **kwargs
-1525 )
-1526 pbar_signals.pbar_stop.emit()
-1527 return seg
-1528
-1529 def update_segmentation(seg):
-1530 if i is None:
-1531 self._viewer.layers["auto_segmentation"].data = seg
-1532 else:
-1533 self._viewer.layers["auto_segmentation"].data[i] = seg
-1534 self._viewer.layers["auto_segmentation"].refresh()
-1535
-1536 worker = seg_impl()
-1537 worker.returned.connect(update_segmentation)
-1538 worker.start()
-1539 return worker
-1540
-1541 # We refuse to run 3D segmentation with the AMG unless we have a GPU or all embeddings
-1542 # are precomputed. Otherwise this would take too long.
-1543 def _allow_segment_3d(self):
-1544 if self.with_decoder:
-1545 return True
-1546 state = AnnotatorState()
-1547 predictor = state.predictor
-1548 if str(predictor.device) == "cpu" or str(predictor.device) == "mps":
-1549 n_slices = self._viewer.layers["auto_segmentation"].data.shape[0]
-1550 embeddings_are_precomputed = (state.amg_state is not None) and (len(state.amg_state) > n_slices)
-1551 if not embeddings_are_precomputed:
-1552 return False
-1553 return True
-1554
-1555 def _run_segmentation_3d(self, kwargs):
-1556 allow_segment_3d = self._allow_segment_3d()
-1557 if not allow_segment_3d:
-1558 val_results = {
-1559 "message_type": "error",
-1560 "message": "Volumetric segmentation with AMG is only supported if you have a GPU."
-1561 }
-1562 return _generate_message(val_results["message_type"], val_results["message"])
-1563
-1564 pbar, pbar_signals = _create_pbar_for_threadworker()
-1565
-1566 @thread_worker
-1567 def seg_impl():
-1568 segmentation = np.zeros_like(self._viewer.layers["auto_segmentation"].data)
-1569 offset = 0
-1570
-1571 def pbar_init(total, description):
-1572 pbar_signals.pbar_total.emit(total)
-1573 pbar_signals.pbar_description.emit(description)
-1574
-1575 pbar_init(segmentation.shape[0], "Segment volume")
-1576
-1577 # Further optimization: parallelize if state is precomputed for all slices
-1578 for i in range(segmentation.shape[0]):
-1579 seg = _instance_segmentation_impl(self.with_background, self.min_object_size, i=i, **kwargs)
-1580 seg_max = seg.max()
-1581 if seg_max == 0:
-1582 continue
-1583 seg[seg != 0] += offset
-1584 offset = seg_max + offset
-1585 segmentation[i] = seg
-1586 pbar_signals.pbar_update.emit(1)
-1587
-1588 pbar_signals.pbar_reset.emit()
-1589 segmentation = merge_instance_segmentation_3d(
-1590 segmentation, beta=0.5, with_background=self.with_background,
-1591 gap_closing=self.gap_closing, min_z_extent=self.min_extent,
-1592 verbose=True, pbar_init=pbar_init,
-1593 pbar_update=lambda update: pbar_signals.pbar_update.emit(1),
-1594 )
-1595 pbar_signals.pbar_stop.emit()
-1596 return segmentation
-1597
-1598 def update_segmentation(segmentation):
-1599 self._viewer.layers["auto_segmentation"].data = segmentation
-1600 self._viewer.layers["auto_segmentation"].refresh()
-1601
-1602 worker = seg_impl()
-1603 worker.returned.connect(update_segmentation)
-1604 worker.start()
-1605 return worker
-1606
-1607 def __call__(self):
-1608 if _validate_embeddings(self._viewer):
-1609 return None
-1610
-1611 if self.with_decoder:
-1612 kwargs = {
-1613 "center_distance_threshold": self.center_distance_thresh,
-1614 "boundary_distance_threshold": self.boundary_distance_thresh,
-1615 "min_size": self.min_object_size,
-1616 }
-1617 else:
-1618 kwargs = {
-1619 "pred_iou_thresh": self.pred_iou_thresh,
-1620 "stability_score_thresh": self.stability_score_thresh,
-1621 "box_nms_thresh": self.box_nms_thresh,
-1622 }
-1623 if self.volumetric and self.apply_to_volume:
-1624 worker = self._run_segmentation_3d(kwargs)
-1625 elif self.volumetric and not self.apply_to_volume:
-1626 i = int(self._viewer.cursor.position[0])
-1627 worker = self._run_segmentation_2d(kwargs, i=i)
-1628 else:
-1629 worker = self._run_segmentation_2d(kwargs)
-1630 _select_layer(self._viewer, "auto_segmentation")
-1631 return worker
+1511 def _empty_segmentation_warning(self):
+1512 msg = "The automatic segmentation result does not contain any objects."
+1513 msg += "Setting a smaller value for 'min_object_size' may help."
+1514 if not self.with_decoder:
+1515 msg += "Setting smaller values for 'pred_iou_thresh' and 'stability_score_thresh' may also help."
+1516 val_results = {"message_type": "error", "message": msg}
+1517 return _generate_message(val_results["message_type"], val_results["message"])
+1518
+1519 def _run_segmentation_2d(self, kwargs, i=None):
+1520 pbar, pbar_signals = _create_pbar_for_threadworker()
+1521
+1522 @thread_worker
+1523 def seg_impl():
+1524 def pbar_init(total, description):
+1525 pbar_signals.pbar_total.emit(total)
+1526 pbar_signals.pbar_description.emit(description)
+1527
+1528 seg = _instance_segmentation_impl(
+1529 self.with_background, self.min_object_size, i=i,
+1530 pbar_init=pbar_init,
+1531 pbar_update=lambda update: pbar_signals.pbar_update.emit(update),
+1532 **kwargs
+1533 )
+1534 pbar_signals.pbar_stop.emit()
+1535 return seg
+1536
+1537 def update_segmentation(seg):
+1538 is_empty = seg.max() == 0
+1539 if is_empty:
+1540 self._empty_segmentation_warning()
+1541
+1542 if i is None:
+1543 self._viewer.layers["auto_segmentation"].data = seg
+1544 else:
+1545 self._viewer.layers["auto_segmentation"].data[i] = seg
+1546 self._viewer.layers["auto_segmentation"].refresh()
+1547
+1548 worker = seg_impl()
+1549 worker.returned.connect(update_segmentation)
+1550 worker.start()
+1551 return worker
+1552
+1553 # We refuse to run 3D segmentation with the AMG unless we have a GPU or all embeddings
+1554 # are precomputed. Otherwise this would take too long.
+1555 def _allow_segment_3d(self):
+1556 if self.with_decoder:
+1557 return True
+1558 state = AnnotatorState()
+1559 predictor = state.predictor
+1560 if str(predictor.device) == "cpu" or str(predictor.device) == "mps":
+1561 n_slices = self._viewer.layers["auto_segmentation"].data.shape[0]
+1562 embeddings_are_precomputed = (state.amg_state is not None) and (len(state.amg_state) > n_slices)
+1563 if not embeddings_are_precomputed:
+1564 return False
+1565 return True
+1566
+1567 def _run_segmentation_3d(self, kwargs):
+1568 allow_segment_3d = self._allow_segment_3d()
+1569 if not allow_segment_3d:
+1570 val_results = {
+1571 "message_type": "error",
+1572 "message": "Volumetric segmentation with AMG is only supported if you have a GPU."
+1573 }
+1574 return _generate_message(val_results["message_type"], val_results["message"])
+1575
+1576 pbar, pbar_signals = _create_pbar_for_threadworker()
+1577
+1578 @thread_worker
+1579 def seg_impl():
+1580 segmentation = np.zeros_like(self._viewer.layers["auto_segmentation"].data)
+1581 offset = 0
+1582
+1583 def pbar_init(total, description):
+1584 pbar_signals.pbar_total.emit(total)
+1585 pbar_signals.pbar_description.emit(description)
+1586
+1587 pbar_init(segmentation.shape[0], "Segment volume")
+1588
+1589 # Further optimization: parallelize if state is precomputed for all slices
+1590 for i in range(segmentation.shape[0]):
+1591 seg = _instance_segmentation_impl(self.with_background, self.min_object_size, i=i, **kwargs)
+1592 seg_max = seg.max()
+1593 if seg_max == 0:
+1594 continue
+1595 seg[seg != 0] += offset
+1596 offset = seg_max + offset
+1597 segmentation[i] = seg
+1598 pbar_signals.pbar_update.emit(1)
+1599
+1600 pbar_signals.pbar_reset.emit()
+1601 segmentation = merge_instance_segmentation_3d(
+1602 segmentation, beta=0.5, with_background=self.with_background,
+1603 gap_closing=self.gap_closing, min_z_extent=self.min_extent,
+1604 verbose=True, pbar_init=pbar_init,
+1605 pbar_update=lambda update: pbar_signals.pbar_update.emit(1),
+1606 )
+1607 pbar_signals.pbar_stop.emit()
+1608 return segmentation
+1609
+1610 def update_segmentation(segmentation):
+1611 is_empty = segmentation.max() == 0
+1612 if is_empty:
+1613 self._empty_segmentation_warning()
+1614 self._viewer.layers["auto_segmentation"].data = segmentation
+1615 self._viewer.layers["auto_segmentation"].refresh()
+1616
+1617 worker = seg_impl()
+1618 worker.returned.connect(update_segmentation)
+1619 worker.start()
+1620 return worker
+1621
+1622 def __call__(self):
+1623 if _validate_embeddings(self._viewer):
+1624 return None
+1625
+1626 if self.with_decoder:
+1627 kwargs = {
+1628 "center_distance_threshold": self.center_distance_thresh,
+1629 "boundary_distance_threshold": self.boundary_distance_thresh,
+1630 "min_size": self.min_object_size,
+1631 }
+1632 else:
+1633 kwargs = {
+1634 "pred_iou_thresh": self.pred_iou_thresh,
+1635 "stability_score_thresh": self.stability_score_thresh,
+1636 "box_nms_thresh": self.box_nms_thresh,
+1637 }
+1638 if self.volumetric and self.apply_to_volume:
+1639 worker = self._run_segmentation_3d(kwargs)
+1640 elif self.volumetric and not self.apply_to_volume:
+1641 i = int(self._viewer.cursor.position[0])
+1642 worker = self._run_segmentation_2d(kwargs, i=i)
+1643 else:
+1644 worker = self._run_segmentation_2d(kwargs)
+1645 _select_layer(self._viewer, "auto_segmentation")
+1646 return worker
diff --git a/search.js b/search.js
index bc503fd2..5f99fe11 100644
--- a/search.js
+++ b/search.js
@@ -1,6 +1,6 @@
window.pdocSearch = (function(){
/** elasticlunr - http://weixsong.github.io * Copyright (C) 2017 Oliver Nightingale * Copyright (C) 2017 Wei Song * MIT Licensed */!function(){function e(e){if(null===e||"object"!=typeof e)return e;var t=e.constructor();for(var n in e)e.hasOwnProperty(n)&&(t[n]=e[n]);return t}var t=function(e){var n=new t.Index;return n.pipeline.add(t.trimmer,t.stopWordFilter,t.stemmer),e&&e.call(n,n),n};t.version="0.9.5",lunr=t,t.utils={},t.utils.warn=function(e){return function(t){e.console&&console.warn&&console.warn(t)}}(this),t.utils.toString=function(e){return void 0===e||null===e?"":e.toString()},t.EventEmitter=function(){this.events={}},t.EventEmitter.prototype.addListener=function(){var e=Array.prototype.slice.call(arguments),t=e.pop(),n=e;if("function"!=typeof t)throw new TypeError("last argument must be a function");n.forEach(function(e){this.hasHandler(e)||(this.events[e]=[]),this.events[e].push(t)},this)},t.EventEmitter.prototype.removeListener=function(e,t){if(this.hasHandler(e)){var n=this.events[e].indexOf(t);-1!==n&&(this.events[e].splice(n,1),0==this.events[e].length&&delete this.events[e])}},t.EventEmitter.prototype.emit=function(e){if(this.hasHandler(e)){var t=Array.prototype.slice.call(arguments,1);this.events[e].forEach(function(e){e.apply(void 0,t)},this)}},t.EventEmitter.prototype.hasHandler=function(e){return e in this.events},t.tokenizer=function(e){if(!arguments.length||null===e||void 0===e)return[];if(Array.isArray(e)){var n=e.filter(function(e){return null===e||void 0===e?!1:!0});n=n.map(function(e){return t.utils.toString(e).toLowerCase()});var i=[];return n.forEach(function(e){var n=e.split(t.tokenizer.seperator);i=i.concat(n)},this),i}return e.toString().trim().toLowerCase().split(t.tokenizer.seperator)},t.tokenizer.defaultSeperator=/[\s\-]+/,t.tokenizer.seperator=t.tokenizer.defaultSeperator,t.tokenizer.setSeperator=function(e){null!==e&&void 0!==e&&"object"==typeof e&&(t.tokenizer.seperator=e)},t.tokenizer.resetSeperator=function(){t.tokenizer.seperator=t.tokenizer.defaultSeperator},t.tokenizer.getSeperator=function(){return t.tokenizer.seperator},t.Pipeline=function(){this._queue=[]},t.Pipeline.registeredFunctions={},t.Pipeline.registerFunction=function(e,n){n in t.Pipeline.registeredFunctions&&t.utils.warn("Overwriting existing registered function: "+n),e.label=n,t.Pipeline.registeredFunctions[n]=e},t.Pipeline.getRegisteredFunction=function(e){return e in t.Pipeline.registeredFunctions!=!0?null:t.Pipeline.registeredFunctions[e]},t.Pipeline.warnIfFunctionNotRegistered=function(e){var n=e.label&&e.label in this.registeredFunctions;n||t.utils.warn("Function is not registered with pipeline. This may cause problems when serialising the index.\n",e)},t.Pipeline.load=function(e){var n=new t.Pipeline;return e.forEach(function(e){var i=t.Pipeline.getRegisteredFunction(e);if(!i)throw new Error("Cannot load un-registered function: "+e);n.add(i)}),n},t.Pipeline.prototype.add=function(){var e=Array.prototype.slice.call(arguments);e.forEach(function(e){t.Pipeline.warnIfFunctionNotRegistered(e),this._queue.push(e)},this)},t.Pipeline.prototype.after=function(e,n){t.Pipeline.warnIfFunctionNotRegistered(n);var i=this._queue.indexOf(e);if(-1===i)throw new Error("Cannot find existingFn");this._queue.splice(i+1,0,n)},t.Pipeline.prototype.before=function(e,n){t.Pipeline.warnIfFunctionNotRegistered(n);var i=this._queue.indexOf(e);if(-1===i)throw new Error("Cannot find existingFn");this._queue.splice(i,0,n)},t.Pipeline.prototype.remove=function(e){var t=this._queue.indexOf(e);-1!==t&&this._queue.splice(t,1)},t.Pipeline.prototype.run=function(e){for(var t=[],n=e.length,i=this._queue.length,o=0;n>o;o++){for(var r=e[o],s=0;i>s&&(r=this._queue[s](r,o,e),void 0!==r&&null!==r);s++);void 0!==r&&null!==r&&t.push(r)}return t},t.Pipeline.prototype.reset=function(){this._queue=[]},t.Pipeline.prototype.get=function(){return this._queue},t.Pipeline.prototype.toJSON=function(){return this._queue.map(function(e){return t.Pipeline.warnIfFunctionNotRegistered(e),e.label})},t.Index=function(){this._fields=[],this._ref="id",this.pipeline=new t.Pipeline,this.documentStore=new t.DocumentStore,this.index={},this.eventEmitter=new t.EventEmitter,this._idfCache={},this.on("add","remove","update",function(){this._idfCache={}}.bind(this))},t.Index.prototype.on=function(){var e=Array.prototype.slice.call(arguments);return this.eventEmitter.addListener.apply(this.eventEmitter,e)},t.Index.prototype.off=function(e,t){return this.eventEmitter.removeListener(e,t)},t.Index.load=function(e){e.version!==t.version&&t.utils.warn("version mismatch: current "+t.version+" importing "+e.version);var n=new this;n._fields=e.fields,n._ref=e.ref,n.documentStore=t.DocumentStore.load(e.documentStore),n.pipeline=t.Pipeline.load(e.pipeline),n.index={};for(var i in e.index)n.index[i]=t.InvertedIndex.load(e.index[i]);return n},t.Index.prototype.addField=function(e){return this._fields.push(e),this.index[e]=new t.InvertedIndex,this},t.Index.prototype.setRef=function(e){return this._ref=e,this},t.Index.prototype.saveDocument=function(e){return this.documentStore=new t.DocumentStore(e),this},t.Index.prototype.addDoc=function(e,n){if(e){var n=void 0===n?!0:n,i=e[this._ref];this.documentStore.addDoc(i,e),this._fields.forEach(function(n){var o=this.pipeline.run(t.tokenizer(e[n]));this.documentStore.addFieldLength(i,n,o.length);var r={};o.forEach(function(e){e in r?r[e]+=1:r[e]=1},this);for(var s in r){var u=r[s];u=Math.sqrt(u),this.index[n].addToken(s,{ref:i,tf:u})}},this),n&&this.eventEmitter.emit("add",e,this)}},t.Index.prototype.removeDocByRef=function(e){if(e&&this.documentStore.isDocStored()!==!1&&this.documentStore.hasDoc(e)){var t=this.documentStore.getDoc(e);this.removeDoc(t,!1)}},t.Index.prototype.removeDoc=function(e,n){if(e){var n=void 0===n?!0:n,i=e[this._ref];this.documentStore.hasDoc(i)&&(this.documentStore.removeDoc(i),this._fields.forEach(function(n){var o=this.pipeline.run(t.tokenizer(e[n]));o.forEach(function(e){this.index[n].removeToken(e,i)},this)},this),n&&this.eventEmitter.emit("remove",e,this))}},t.Index.prototype.updateDoc=function(e,t){var t=void 0===t?!0:t;this.removeDocByRef(e[this._ref],!1),this.addDoc(e,!1),t&&this.eventEmitter.emit("update",e,this)},t.Index.prototype.idf=function(e,t){var n="@"+t+"/"+e;if(Object.prototype.hasOwnProperty.call(this._idfCache,n))return this._idfCache[n];var i=this.index[t].getDocFreq(e),o=1+Math.log(this.documentStore.length/(i+1));return this._idfCache[n]=o,o},t.Index.prototype.getFields=function(){return this._fields.slice()},t.Index.prototype.search=function(e,n){if(!e)return[];e="string"==typeof e?{any:e}:JSON.parse(JSON.stringify(e));var i=null;null!=n&&(i=JSON.stringify(n));for(var o=new t.Configuration(i,this.getFields()).get(),r={},s=Object.keys(e),u=0;u0&&t.push(e);for(var i in n)"docs"!==i&&"df"!==i&&this.expandToken(e+i,t,n[i]);return t},t.InvertedIndex.prototype.toJSON=function(){return{root:this.root}},t.Configuration=function(e,n){var e=e||"";if(void 0==n||null==n)throw new Error("fields should not be null");this.config={};var i;try{i=JSON.parse(e),this.buildUserConfig(i,n)}catch(o){t.utils.warn("user configuration parse failed, will use default configuration"),this.buildDefaultConfig(n)}},t.Configuration.prototype.buildDefaultConfig=function(e){this.reset(),e.forEach(function(e){this.config[e]={boost:1,bool:"OR",expand:!1}},this)},t.Configuration.prototype.buildUserConfig=function(e,n){var i="OR",o=!1;if(this.reset(),"bool"in e&&(i=e.bool||i),"expand"in e&&(o=e.expand||o),"fields"in e)for(var r in e.fields)if(n.indexOf(r)>-1){var s=e.fields[r],u=o;void 0!=s.expand&&(u=s.expand),this.config[r]={boost:s.boost||0===s.boost?s.boost:1,bool:s.bool||i,expand:u}}else t.utils.warn("field name in user configuration not found in index instance fields");else this.addAllFields2UserConfig(i,o,n)},t.Configuration.prototype.addAllFields2UserConfig=function(e,t,n){n.forEach(function(n){this.config[n]={boost:1,bool:e,expand:t}},this)},t.Configuration.prototype.get=function(){return this.config},t.Configuration.prototype.reset=function(){this.config={}},lunr.SortedSet=function(){this.length=0,this.elements=[]},lunr.SortedSet.load=function(e){var t=new this;return t.elements=e,t.length=e.length,t},lunr.SortedSet.prototype.add=function(){var e,t;for(e=0;e1;){if(r===e)return o;e>r&&(t=o),r>e&&(n=o),i=n-t,o=t+Math.floor(i/2),r=this.elements[o]}return r===e?o:-1},lunr.SortedSet.prototype.locationFor=function(e){for(var t=0,n=this.elements.length,i=n-t,o=t+Math.floor(i/2),r=this.elements[o];i>1;)e>r&&(t=o),r>e&&(n=o),i=n-t,o=t+Math.floor(i/2),r=this.elements[o];return r>e?o:e>r?o+1:void 0},lunr.SortedSet.prototype.intersect=function(e){for(var t=new lunr.SortedSet,n=0,i=0,o=this.length,r=e.length,s=this.elements,u=e.elements;;){if(n>o-1||i>r-1)break;s[n]!==u[i]?s[n]u[i]&&i++:(t.add(s[n]),n++,i++)}return t},lunr.SortedSet.prototype.clone=function(){var e=new lunr.SortedSet;return e.elements=this.toArray(),e.length=e.elements.length,e},lunr.SortedSet.prototype.union=function(e){var t,n,i;this.length>=e.length?(t=this,n=e):(t=e,n=this),i=t.clone();for(var o=0,r=n.toArray();oSegment Anything for Microscopy
\n\n1__version__ = "1.0.0" +diff --git a/micro_sam/sam_annotator/_widgets.html b/micro_sam/sam_annotator/_widgets.html index c3c45f6d..fa34fc3f 100644 --- a/micro_sam/sam_annotator/_widgets.html +++ b/micro_sam/sam_annotator/_widgets.html @@ -1671,127 +1671,142 @@1__version__ = "1.0.0post0"1507 settings = _make_collapsible(setting_values, title="Automatic Segmentation Settings") 1508 return settings 1509 -1510 def _run_segmentation_2d(self, kwargs, i=None): -1511 pbar, pbar_signals = _create_pbar_for_threadworker() -1512 -1513 @thread_worker -1514 def seg_impl(): -1515 def pbar_init(total, description): -1516 pbar_signals.pbar_total.emit(total) -1517 pbar_signals.pbar_description.emit(description) -1518 -1519 seg = _instance_segmentation_impl( -1520 self.with_background, self.min_object_size, i=i, -1521 pbar_init=pbar_init, -1522 pbar_update=lambda update: pbar_signals.pbar_update.emit(update), -1523 **kwargs -1524 ) -1525 pbar_signals.pbar_stop.emit() -1526 return seg -1527 -1528 def update_segmentation(seg): -1529 if i is None: -1530 self._viewer.layers["auto_segmentation"].data = seg -1531 else: -1532 self._viewer.layers["auto_segmentation"].data[i] = seg -1533 self._viewer.layers["auto_segmentation"].refresh() -1534 -1535 worker = seg_impl() -1536 worker.returned.connect(update_segmentation) -1537 worker.start() -1538 return worker -1539 -1540 # We refuse to run 3D segmentation with the AMG unless we have a GPU or all embeddings -1541 # are precomputed. Otherwise this would take too long. -1542 def _allow_segment_3d(self): -1543 if self.with_decoder: -1544 return True -1545 state = AnnotatorState() -1546 predictor = state.predictor -1547 if str(predictor.device) == "cpu" or str(predictor.device) == "mps": -1548 n_slices = self._viewer.layers["auto_segmentation"].data.shape[0] -1549 embeddings_are_precomputed = (state.amg_state is not None) and (len(state.amg_state) > n_slices) -1550 if not embeddings_are_precomputed: -1551 return False -1552 return True -1553 -1554 def _run_segmentation_3d(self, kwargs): -1555 allow_segment_3d = self._allow_segment_3d() -1556 if not allow_segment_3d: -1557 val_results = { -1558 "message_type": "error", -1559 "message": "Volumetric segmentation with AMG is only supported if you have a GPU." -1560 } -1561 return _generate_message(val_results["message_type"], val_results["message"]) -1562 -1563 pbar, pbar_signals = _create_pbar_for_threadworker() -1564 -1565 @thread_worker -1566 def seg_impl(): -1567 segmentation = np.zeros_like(self._viewer.layers["auto_segmentation"].data) -1568 offset = 0 -1569 -1570 def pbar_init(total, description): -1571 pbar_signals.pbar_total.emit(total) -1572 pbar_signals.pbar_description.emit(description) -1573 -1574 pbar_init(segmentation.shape[0], "Segment volume") -1575 -1576 # Further optimization: parallelize if state is precomputed for all slices -1577 for i in range(segmentation.shape[0]): -1578 seg = _instance_segmentation_impl(self.with_background, self.min_object_size, i=i, **kwargs) -1579 seg_max = seg.max() -1580 if seg_max == 0: -1581 continue -1582 seg[seg != 0] += offset -1583 offset = seg_max + offset -1584 segmentation[i] = seg -1585 pbar_signals.pbar_update.emit(1) -1586 -1587 pbar_signals.pbar_reset.emit() -1588 segmentation = merge_instance_segmentation_3d( -1589 segmentation, beta=0.5, with_background=self.with_background, -1590 gap_closing=self.gap_closing, min_z_extent=self.min_extent, -1591 verbose=True, pbar_init=pbar_init, -1592 pbar_update=lambda update: pbar_signals.pbar_update.emit(1), -1593 ) -1594 pbar_signals.pbar_stop.emit() -1595 return segmentation -1596 -1597 def update_segmentation(segmentation): -1598 self._viewer.layers["auto_segmentation"].data = segmentation -1599 self._viewer.layers["auto_segmentation"].refresh() -1600 -1601 worker = seg_impl() -1602 worker.returned.connect(update_segmentation) -1603 worker.start() -1604 return worker -1605 -1606 def __call__(self): -1607 if _validate_embeddings(self._viewer): -1608 return None -1609 -1610 if self.with_decoder: -1611 kwargs = { -1612 "center_distance_threshold": self.center_distance_thresh, -1613 "boundary_distance_threshold": self.boundary_distance_thresh, -1614 "min_size": self.min_object_size, -1615 } -1616 else: -1617 kwargs = { -1618 "pred_iou_thresh": self.pred_iou_thresh, -1619 "stability_score_thresh": self.stability_score_thresh, -1620 "box_nms_thresh": self.box_nms_thresh, -1621 } -1622 if self.volumetric and self.apply_to_volume: -1623 worker = self._run_segmentation_3d(kwargs) -1624 elif self.volumetric and not self.apply_to_volume: -1625 i = int(self._viewer.cursor.position[0]) -1626 worker = self._run_segmentation_2d(kwargs, i=i) -1627 else: -1628 worker = self._run_segmentation_2d(kwargs) -1629 _select_layer(self._viewer, "auto_segmentation") -1630 return worker +1510 def _empty_segmentation_warning(self): +1511 msg = "The automatic segmentation result does not contain any objects." +1512 msg += "Setting a smaller value for 'min_object_size' may help." +1513 if not self.with_decoder: +1514 msg += "Setting smaller values for 'pred_iou_thresh' and 'stability_score_thresh' may also help." +1515 val_results = {"message_type": "error", "message": msg} +1516 return _generate_message(val_results["message_type"], val_results["message"]) +1517 +1518 def _run_segmentation_2d(self, kwargs, i=None): +1519 pbar, pbar_signals = _create_pbar_for_threadworker() +1520 +1521 @thread_worker +1522 def seg_impl(): +1523 def pbar_init(total, description): +1524 pbar_signals.pbar_total.emit(total) +1525 pbar_signals.pbar_description.emit(description) +1526 +1527 seg = _instance_segmentation_impl( +1528 self.with_background, self.min_object_size, i=i, +1529 pbar_init=pbar_init, +1530 pbar_update=lambda update: pbar_signals.pbar_update.emit(update), +1531 **kwargs +1532 ) +1533 pbar_signals.pbar_stop.emit() +1534 return seg +1535 +1536 def update_segmentation(seg): +1537 is_empty = seg.max() == 0 +1538 if is_empty: +1539 self._empty_segmentation_warning() +1540 +1541 if i is None: +1542 self._viewer.layers["auto_segmentation"].data = seg +1543 else: +1544 self._viewer.layers["auto_segmentation"].data[i] = seg +1545 self._viewer.layers["auto_segmentation"].refresh() +1546 +1547 worker = seg_impl() +1548 worker.returned.connect(update_segmentation) +1549 worker.start() +1550 return worker +1551 +1552 # We refuse to run 3D segmentation with the AMG unless we have a GPU or all embeddings +1553 # are precomputed. Otherwise this would take too long. +1554 def _allow_segment_3d(self): +1555 if self.with_decoder: +1556 return True +1557 state = AnnotatorState() +1558 predictor = state.predictor +1559 if str(predictor.device) == "cpu" or str(predictor.device) == "mps": +1560 n_slices = self._viewer.layers["auto_segmentation"].data.shape[0] +1561 embeddings_are_precomputed = (state.amg_state is not None) and (len(state.amg_state) > n_slices) +1562 if not embeddings_are_precomputed: +1563 return False +1564 return True +1565 +1566 def _run_segmentation_3d(self, kwargs): +1567 allow_segment_3d = self._allow_segment_3d() +1568 if not allow_segment_3d: +1569 val_results = { +1570 "message_type": "error", +1571 "message": "Volumetric segmentation with AMG is only supported if you have a GPU." +1572 } +1573 return _generate_message(val_results["message_type"], val_results["message"]) +1574 +1575 pbar, pbar_signals = _create_pbar_for_threadworker() +1576 +1577 @thread_worker +1578 def seg_impl(): +1579 segmentation = np.zeros_like(self._viewer.layers["auto_segmentation"].data) +1580 offset = 0 +1581 +1582 def pbar_init(total, description): +1583 pbar_signals.pbar_total.emit(total) +1584 pbar_signals.pbar_description.emit(description) +1585 +1586 pbar_init(segmentation.shape[0], "Segment volume") +1587 +1588 # Further optimization: parallelize if state is precomputed for all slices +1589 for i in range(segmentation.shape[0]): +1590 seg = _instance_segmentation_impl(self.with_background, self.min_object_size, i=i, **kwargs) +1591 seg_max = seg.max() +1592 if seg_max == 0: +1593 continue +1594 seg[seg != 0] += offset +1595 offset = seg_max + offset +1596 segmentation[i] = seg +1597 pbar_signals.pbar_update.emit(1) +1598 +1599 pbar_signals.pbar_reset.emit() +1600 segmentation = merge_instance_segmentation_3d( +1601 segmentation, beta=0.5, with_background=self.with_background, +1602 gap_closing=self.gap_closing, min_z_extent=self.min_extent, +1603 verbose=True, pbar_init=pbar_init, +1604 pbar_update=lambda update: pbar_signals.pbar_update.emit(1), +1605 ) +1606 pbar_signals.pbar_stop.emit() +1607 return segmentation +1608 +1609 def update_segmentation(segmentation): +1610 is_empty = segmentation.max() == 0 +1611 if is_empty: +1612 self._empty_segmentation_warning() +1613 self._viewer.layers["auto_segmentation"].data = segmentation +1614 self._viewer.layers["auto_segmentation"].refresh() +1615 +1616 worker = seg_impl() +1617 worker.returned.connect(update_segmentation) +1618 worker.start() +1619 return worker +1620 +1621 def __call__(self): +1622 if _validate_embeddings(self._viewer): +1623 return None +1624 +1625 if self.with_decoder: +1626 kwargs = { +1627 "center_distance_threshold": self.center_distance_thresh, +1628 "boundary_distance_threshold": self.boundary_distance_thresh, +1629 "min_size": self.min_object_size, +1630 } +1631 else: +1632 kwargs = { +1633 "pred_iou_thresh": self.pred_iou_thresh, +1634 "stability_score_thresh": self.stability_score_thresh, +1635 "box_nms_thresh": self.box_nms_thresh, +1636 } +1637 if self.volumetric and self.apply_to_volume: +1638 worker = self._run_segmentation_3d(kwargs) +1639 elif self.volumetric and not self.apply_to_volume: +1640 i = int(self._viewer.cursor.position[0]) +1641 worker = self._run_segmentation_2d(kwargs, i=i) +1642 else: +1643 worker = self._run_segmentation_2d(kwargs) +1644 _select_layer(self._viewer, "auto_segmentation") +1645 return worker
Segment Anything for Microscopy implements automatic and interactive annotation for microscopy data. It is built on top of Segment Anything by Meta AI and specializes it for microscopy and other bio-imaging data.\nIts core components are:
\n\n- \n
Based on these components micro_sam enables fast interactive and automatic annotation for microscopy data, like interactive cell segmentation from bounding boxes:
micro_sam is now available as stable version 1.0 and we will not change its user interface significantly in the foreseeable future.\nWe are still working on improving and extending its functionality. The current roadmap includes:
- \n
If you run into any problems or have questions please open an issue or reach out via image.sc using the tag micro-sam.
Quickstart
\n\nYou can install micro_sam via mamba:
$ mamba install -c conda-forge micro_sam\nWe also provide installers for Windows and Linux. For more details on the available installation options check out the installation section.
\n\nAfter installing micro_sam you can start napari and select the annotation tool you want to use from Plugins->Segment Anything for Microscopy. Check out the quickstart tutorial video for a short introduction and the annotation tool section for details.
The micro_sam python library can be imported via
import micro_sam\nIt is explained in more detail here.
\n\nWe provide different finetuned models for microscopy that can be used within our tools or any other tool that supports Segment Anything. See finetuned models for details on the available models.\nYou can also train models on your own data, see here for details.
\n\nCitation
\n\nIf you are using micro_sam in your research please cite
- \n
- Our preprint \n
- and the original Segment Anything publication. \n
- If you use a
vit-tinymodels please also cite Mobile SAM. \n
Installation
\n\nThere are three ways to install micro_sam:
- \n
- From mamba is the recommended way if you want to use all functionality. \n
- From source for setting up a development environment to use the development version and to change and contribute to our software. \n
- From installer to install it without having to use mamba (supported platforms: Windows and Linux, only for CPU users). \n
From mamba
\n\nmamba is a drop-in replacement for conda, but much faster.\nWhile the steps below may also work with conda, we highly recommend using mamba.\nYou can follow the instructions here to install mamba.
IMPORTANT: Make sure to avoid installing anything in the base environment.
\n\nmicro_sam can be installed in an existing environment via:
$ mamba install -c conda-forge micro_sam\nor you can create a new environment (here called micro-sam) with it via:
$ mamba create -c conda-forge -n micro-sam micro_sam\nif you want to use the GPU you need to install PyTorch from the pytorch channel instead of conda-forge. For example:
$ mamba create -c pytorch -c nvidia -c conda-forge micro_sam pytorch pytorch-cuda=12.1\nYou may need to change this command to install the correct CUDA version for your system, see https://pytorch.org/ for details.
\n\nFrom source
\n\nTo install micro_sam from source, we recommend to first set up an environment with the necessary requirements:
- \n
- environment_gpu.yaml: sets up an environment with GPU support. \n
- environment_cpu.yaml: sets up an environment with CPU support. \n
To create one of these environments and install micro_sam into it follow these steps
- \n
- Clone the repository: \n
$ git clone https://github.com/computational-cell-analytics/micro-sam\n- \n
- Enter it: \n
$ cd micro-sam\n- \n
- Create the GPU or CPU environment: \n
$ mamba env create -f <ENV_FILE>.yaml\n- \n
- Activate the environment: \n
$ mamba activate sam\n- \n
- Install
micro_sam: \n
$ pip install -e .\nFrom installer
\n\nWe also provide installers for Linux and Windows:
\n\n\n\nThe installers will not enable you to use a GPU, so if you have one then please consider installing micro_sam via mamba instead. They will also not enable using the python library.
Linux Installer:
\n\nTo use the installer:
\n\n- \n
- Unpack the zip file you have downloaded. \n
- Make the installer executable:
$ chmod +x micro_sam-0.2.0post1-Linux-x86_64.sh\n - Run the installer:
$./micro_sam-0.2.0post1-Linux-x86_64.sh$\n- \n
- You can select where to install
micro_samduring the installation. By default it will be installed in$HOME/micro_sam. \n - The installer will unpack all
micro_samfiles to the installation directory. \n
\n - You can select where to install
- After the installation you can start the annotator with the command
.../micro_sam/bin/micro_sam.annotator.\n- \n
- To make it easier to run the annotation tool you can add
.../micro_sam/binto yourPATHor set a softlink to.../micro_sam/bin/micro_sam.annotator. \n
\n - To make it easier to run the annotation tool you can add
Windows Installer:
\n\n- \n
- Unpack the zip file you have downloaded. \n
- Run the installer by double clicking on it. \n
- Choose installation type:
Just Me(recommended)orAll Users(requires admin privileges). \n - Choose installation path. By default it will be installed in
C:\\Users\\<Username>\\micro_samforJust Meinstallation or inC:\\ProgramData\\micro_samforAll Users.\n- \n
- The installer will unpack all micro_sam files to the installation directory. \n
\n - After the installation you can start the annotator by double clicking on
.\\micro_sam\\Scripts\\micro_sam.annotator.exeor with the command.\\micro_sam\\Scripts\\micro_sam.annotator.exefrom the Command Prompt. \n
Annotation Tools
\n\nmicro_sam provides applications for fast interactive 2d segmentation, 3d segmentation and tracking.\nSee an example for interactive cell segmentation in phase-contrast microscopy (left), interactive segmentation\nof mitochondria in volume EM (middle) and interactive tracking of cells (right).
\n
\n
The annotation tools can be started from the napari plugin menu, the command line or from python scripts.\nThey are built as napari plugin and make use of existing napari functionality wherever possible. If you are not familiar with napari yet, start here.\nThe micro_sam tools mainly use the point layer, shape layer and label layer.
The annotation tools are explained in detail below. We also provide video tutorials.
\n\nThe annotation tools can be started from the napari plugin menu:\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/napari-plugin.png\")
Annotator 2D
\n\nThe 2d annotator can be started by
\n\n- \n
- clicking
Annotator 2din the plugin menu. \n - running
$ micro_sam.annotator_2din the command line. \n - calling
micro_sam.sam_annotator.annotator_2din a python script. Check out examples/annotator_2d.py for details. \n
The user interface of the 2d annotator looks like this:
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/2d-annotator-menu.png\")
It contains the following elements:
\n\n- \n
- The napari layers for the segmentations and prompts:\n
- \n
prompts: shape layer that is used to provide box prompts to SegmentAnything. Annotations can be given as rectangle (box prompt in the image), ellipse or polygon. \npoint_prompts: point layer that is used to provide point prompts to SegmentAnything. Positive prompts (green points) for marking the object you want to segment, negative prompts (red points) for marking the outside of the object. \ncommitted_objects: label layer with the objects that have already been segmented. \nauto_segmentation: label layer with the results from automatic instance segmentation. \ncurrent_object: label layer for the object(s) you're currently segmenting. \n
\n - The embedding menu. For selecting the image to process, the Segment Anything model that is used and computing the image embeddings with the model. The
Embedding Settingscontain advanced settings for loading cached embeddings from file or using tiled embeddings. \n - The prompt menu for changing whether the currently selected point is a positive or a negative prompt. This can also be done by pressing
T. \n - The menu for interactive segmentation. Clicking
Segment Object(or pressingS) will run segmentation for the current prompts. The result is displayed incurrent_object. Activatingbatchedenables segmentation of multiple objects with point prompts. In this case an object will be segmented per positive prompt. \n - The menu for automatic segmentation. Clicking
Automatic Segmentationwill segment all objects n the image. The results will be displayed in theauto_segmentationlayer. We support two different methods for automatic segmentation: automatic mask generation (supported for all models) and instance segmentation with an additional decoder (only supported for our models).\nChanging the parameters underAutomatic Segmentation Settingscontrols the segmentation results, check the tooltips for details. \n - The menu for commiting the segmentation. When clicking
Commit(or pressingC) the result from the selected layer (eithercurrent_objectorauto_segmentation) will be transferred from the respective layer tocommitted_objects.\nWhencommit_pathis given the results will automatically be saved there. \n - The menu for clearing the current annotations. Clicking
Clear Annotations(or pressingShift + C) will clear the current annotations and the current segmentation. \n
Note that point prompts and box prompts can be combined. When you're using point prompts you can only segment one object at a time, unless the batched mode is activated. With box prompts you can segment several objects at once, both in the normal and batched mode.
Check out this video for a tutorial for this tool.
\n\nAnnotator 3D
\n\nThe 3d annotator can be started by
\n\n- \n
- clicking
Annotator 3din the plugin menu. \n - running
$ micro_sam.annotator_3din the command line. \n - calling
micro_sam.sam_annotator.annotator_3din a python script. Check out examples/annotator_3d.py for details. \n
The user interface of the 3d annotator looks like this:
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/3d-annotator-menu.png\")
Most elements are the same as in the 2d annotator:
\n\n- \n
- The napari layers that contain the segmentations and prompts. \n
- The embedding menu. \n
- The prompt menu. \n
- The menu for interactive segmentation. \n
- The menu for interactive 3d segmentation. Clicking
Segment All Slices(orShift + S) will extend the segmentation for the current object across the volume by projecting prompts across slices. The parameters for prompt projection can be set inSegmentation Settings, please refer to the tooltips for details. \n - The menu for automatic segmentation. The overall functionality is the same as for the 2d annotator. To segment the full volume
Apply to Volumeneeds to be checked, otherwise only the current slice will be segmented. Note that 3D segmentation can take quite long without a GPU. \n - The menu for committing the current object. \n
- The menu for clearing the current annotations. If
all slicesis set all annotations will be cleared, otherwise they are only cleared for the current slice. \n
Note that you can only segment one object at a time using the interactive segmentation functionality with this tool.
\n\nCheck out this video for a tutorial for the 3d annotation tool.
\n\nAnnotator Tracking
\n\nThe tracking annotator can be started by
\n\n- \n
- clicking
Annotator Trackingin the plugin menu. \n - running
$ micro_sam.annotator_trackingin the command line. \n - calling
micro_sam.sam_annotator.annotator_trackingin a python script. Check out examples/annotator_tracking.py for details. \n
The user interface of the tracking annotator looks like this:
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/tracking-annotator-menu.png\"){kind=tracking}
Most elements are the same as in the 2d annotator:
\n\n- \n
- The napari layers that contain the segmentations and prompts. Same as for the 2d segmentation app but without the
auto_segmentationlayer. \n - The embedding menu. \n
- The prompt menu. \n
- The menu with tracking settings:
track_stateis used to indicate that the object you are tracking is dividing in the current frame.track_idis used to select which of the tracks after division you are following. \n - The menu for interactive segmentation. \n
- The menu for interactive tracking menu. Click
Track Object(or pressShift + S) to segment the current object across time. \n - The menu for committing the current tracking result. \n
- The menu for clearing the current annotations. \n
Note that the tracking annotator only supports 2d image data, volumetric data is not supported. We also do not support automatic tracking yet.
\n\nCheck out this video for a tutorial for how to use the tracking annotation tool.
\n\nImage Series Annotator
\n\nThe image series annotation tool enables running the 2d annotator or 2d annotator for multiple images that are saved within an folder. This makes it convenient to annotate many images without having to close the tool. It can be started by
\n\n- \n
- clicking
Image Series Annotatorin the plugin menu. \n - running
$ micro_sam.image_series_annotatorin the command line. \n - calling
micro_sam.sam_annotator.image_series_annotatorin a python script. Check out examples/image_series_annotator.py for details. \n
When starting this tool via the plugin menu the following interface opens:
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/series-menu.png\")
You can select the folder where your image data is saved with Input Folder. The annotation results will be saved in Output Folder.\nYou can specify a rule for loading only a subset of images via pattern, for example *.tif to only load tif images. Set is_volumetric if the data you want to annotate is 3d. The rest of the options are settings for the image embedding computation and are the same as for the embedding menu (see above).\nOnce you click Annotate Images the images from the folder you have specified will be loaded and the annotation tool is started for them.
This menu will not open if you start the image series annotator from the command line or via python. In this case the input folder and other settings are passed as parameters instead.
\n\nCheck out this video for a tutorial for how to use the image series annotator.
\n\nFinetuning UI
\n\nWe also provide a graphical tool for finetuning models on your own data. It can be started by clicking Finetuning in the plugin menu.
Note: if you know a bit of python programming we recommend to use a script for model finetuning instead. This will give you more options to configure the training. See these instructions for details.
\n\nWhen starting this tool via the plugin menu the following interface opens:
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/finetuning-menu.png\")
You can select the image data via Path to images. We can either load images from a folder or select a single file for training. By providing Image data key you can either provide a pattern for selecting files from a folder or provide an internal filepath for hdf5, zarr or similar fileformats.
You can select the label data via Path to labels and Label data key, following the same logic as for the image data. We expect label masks stored in the same size as the image data for training. You can for example use annotations created with one of the micro_sam annotation tools for this, they are stored in the correct format!
The Configuration option allows you to choose the hardware configuration for training. We try to automatically select the correct setting for your system, but it can also be changed. Please refer to the tooltips for the other parameters.
Tips & Tricks
\n\n- \n
- Segment Anything was trained with a fixed image size of 1024 x 1024 pixels. Inputs that do not match this size will be internally resized to match it. Hence, applying Segment Anything to a much larger image will often lead to inferior results, because it will be downsampled by a large factor and the objects in the image become too small.\nTo address this image we implement tiling: cutting up the input image into tiles of a fixed size (with a fixed overlap) and running Segment Anything for the individual tiles.\nYou can activate tiling by passing the parameters
tile_shape, which determines the size of the inner tile andhalo, which determines the size of the additional overlap.\n- \n
- If you're using the
micro_samGUI you can specify the values for thehaloandtile_shapevia theTile X,Tile Y,Halo XandHalo Yby clicking onEmbeddings Settings. \n - If you're using a python script you can pass them as tuples, e.g.
tile_shape=(1024, 1024), halo=(128, 128). See also the wholeslide_annotator example. \n - If you're using the command line functions you can pass them via the options
--tile_shape 1024 1024 --halo 128 128\n - Note that prediction with tiling only works when the embeddings are cached to file, so you must specify an
embedding_path(-ein the CLI). \n - You should choose the
halosuch that it is larger than half of the maximal radius of the objects your segmenting. \n
\n - If you're using the
- The applications pre-compute the image embeddings produced by SegmentAnything and (optionally) store them on disc. If you are using a CPU this step can take a while for 3d data or timeseries (you will see a progress bar with a time estimate). If you have access to a GPU without graphical interface (e.g. via a local computer cluster or a cloud provider), you can also pre-compute the embeddings there and then copy them to your laptop / local machine to speed this up. You can use the command
micro_sam.precompute_embeddingsfor this (it is installed with the rest of the applications). You can specify the location of the precomputed embeddings via theembedding_pathargument.\n- \n
- If you use the GUI to save or load embeddings, simply specify an
embeddings save path. Existing embeddings are loaded from the specified path or embeddings are computed and the path is used to save them. \n
\n - If you use the GUI to save or load embeddings, simply specify an
- Most other processing steps are very fast even on a CPU, so interactive annotation is possible. An exception is the automatic segmentation step (2d segmentation), which takes several minutes without a GPU (depending on the image size). For large volumes and timeseries segmenting an object in 3d / tracking across time can take a couple settings with a CPU (it is very fast with a GPU). \n
- You can also try using a smaller version of the SegmentAnything model to speed up the computations. For this you can pass the
model_typeargument and either set it tovit_bor tovit_l(default isvit_h). However, this may lead to worse results. \n - You can save and load the results from the
committed_objects/committed_trackslayer to correct segmentations you obtained from another tool (e.g. CellPose) or to save intermediate annotation results. The results can be saved viaFile -> Save Selected Layer(s) ...in the napari menu (see the tutorial videos for details). They can be loaded again by specifying the corresponding location via thesegmentation_result(2d and 3d segmentation) ortracking_result(tracking) argument. \n
Known limitations
\n\n- \n
- Segment Anything does not work well for very small or fine-grained objects (e.g. filaments). \n
- For the automatic segmentation functionality we currently rely on the automatic mask generation provided by SegmentAnything. It is slow and often misses objects in microscopy images. \n
- Prompt bounding boxes do not provide the full functionality for tracking yet (they cannot be used for divisions or for starting new tracks). See also this github issue. \n
Using the Python Library
\n\nThe python library can be imported via
\n\nimport micro_sam\nThis library extends the Segment Anything library and
\n\n- \n
- implements functions to apply Segment Anything to 2d and 3d data in
micro_sam.prompt_based_segmentation. \n - provides improved automatic instance segmentation functionality in
micro_sam.instance_segmentation. \n - implements training functionality that can be used for finetuning on your own data in
micro_sam.training. \n - provides functionality for quantitative and qualitative evaluation of Segment Anything models in
micro_sam.evaluation. \n
You can import these sub-modules via
\n\nimport micro_sam.prompt_based_segmentation\nimport micro_sam.instance_segmentation\n# etc.\nThis functionality is used to implement the interactive annotation tools in micro_sam.sam_annotator and can be used as a standalone python library.\nWe provide jupyter notebooks that demonstrate how to use it here. You can find the full library documentation by scrolling to the end of this page.
Training your own model
\n\nWe reimplement the training logic described in the Segment Anything publication to enable finetuning on custom data.\nWe use this functionality to provide the finetuned microscopy models and it can also be used to train models on your own data.\nIn fact the best results can be expected when finetuning on your own data, and we found that it does not require much annotated training data to get siginficant improvements in model performance.\nSo a good strategy is to annotate a few images with one of the provided models using our interactive annotation tools and, if the model is not working as good as required for your use-case, finetune on the annotated data.\n
\n\nThe training logic is implemented in micro_sam.training and is based on torch-em. Check out the finetuning notebook to see how to use it.
We also support training an additional decoder for automatic instance segmentation. This yields better results than the automatic mask generation of segment anything and is significantly faster.\nThe notebook explains how to activate training it together with the rest of SAM and how to then use it.
\n\nMore advanced examples, including quantitative and qualitative evaluation, of finetuned models can be found in finetuning, which contains the code for training and evaluating our models.
\n\nFinetuned models
\n\nIn addition to the original Segment Anything models, we provide models that are finetuned on microscopy data.\nThe additional models are available in the bioimage.io modelzoo and are also hosted on zenodo.
\n\nWe currently offer the following models:
\n\n- \n
vit_h: Default Segment Anything model with vit-h backbone. \nvit_l: Default Segment Anything model with vit-l backbone. \nvit_b: Default Segment Anything model with vit-b backbone. \nvit_t: Segment Anything model with vit-tiny backbone. From the Mobile SAM publication. \nvit_l_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-l backbone. (zenodo, bioimage.io) \nvit_b_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-b backbone. (zenodo, bioimage.io) \nvit_t_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-t backbone. (zenodo, bioimage.io) \nvit_l_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-l backbone. (zenodo, bioimage.io) \nvit_b_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-b backbone. (zenodo, bioimage.io) \nvit_t_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-t backbone. (zenodo, bioimage.io) \n
See the two figures below of the improvements through the finetuned model for LM and EM data.
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/lm_comparison.png\")
![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/em_comparison.png\")
You can select which model to use for annotation by selecting the corresponding name in the embedding menu:
\n\n![](\"https://raw.githubusercontent.com/computational-cell-analytics/micro-sam/master/doc/images/model-type-selector.png\")
To use a specific model in the python library you need to pass the corresponding name as value to the model_type parameter exposed by all relevant functions.\nSee for example the 2d annotator example.
Choosing a Model
\n\nAs a rule of thumb:
\n\n- \n
- Use the
vit_l_lmorvit_b_lmmodel for segmenting cells or nuclei in light microscopy. The larger model (vit_l_lm) yields a bit better segmentation quality, especially for automatic segmentation, but needs more computational resources. \n - Use the
vit_l_em_organellesorvit_b_em_organellesmodels for segmenting mitochondria, nuclei or other roundish organelles in electron microscopy. \n - For other use-cases use one of the default models. \n
- The
vit_t_...models run much faster than other models, but yield inferior quality for many applications. It can still make sense to try them for your use-case if your working on a laptop and want to annotate many images or volumetric data. \n
See also the figures above for examples where the finetuned models work better than the default models.\nWe are working on further improving these models and adding new models for other biomedical imaging domains.
\n\nOlder Models
\n\nPrevious versions of our models are available on zenodo:
\n\n- \n
- vit_b_em_boundaries: for segmenting compartments delineated by boundaries such as cells or neurites in EM. \n
- vit_b_em_organelles: for segmenting mitochondria, nuclei or other organelles in EM. \n
- vit_b_lm: for segmenting cells and nuclei in LM. \n
- vit_h_em: for general EM segmentation. \n
- vit_h_lm: for general LM segmentation. \n
We do not recommend to use these models since our new models improve upon them significantly. But we provide the links here in case they are needed to reproduce older segmentation workflows.
\n\nContribution Guide
\n\n- \n
- Discuss your ideas \n
- Clone the repository \n
- Create your development environment \n
- Make your changes \n
- Testing\n \n
- Open a pull request \n
- Optional: Build the documentation \n
- Optional: Benchmark performance\n \n
Discuss your ideas
\n\nWe welcome new contributions!
\n\nFirst, discuss your idea by opening a new issue in micro-sam.
\n\nThis allows you to ask questions, and have the current developers make suggestions about the best way to implement your ideas.
\n\nYou may also find it helpful to look at this developer guide, which explains the organization of the micro-sam code.
\n\nClone the repository
\n\nWe use git for version control.
\n\nClone the repository, and checkout the development branch:
\n\ngit clone https://github.com/computational-cell-analytics/micro-sam.git\ncd micro-sam\ngit checkout dev\nCreate your development environment
\n\nWe use conda to manage our environments. If you don't have this already, install miniconda or mamba to get started.
\n\nNow you can create the environment, install user and develoepr dependencies, and micro-sam as an editable installation:
\n\nconda env create environment-gpu.yml\nconda activate sam\npython -m pip install requirements-dev.txt\npython -m pip install -e .\nMake your changes
\n\nNow it's time to make your code changes.
\n\nTypically, changes are made branching off from the development branch. Checkout dev and then create a new branch to work on your changes.
git checkout dev\ngit checkout -b my-new-feature\nWe use google style python docstrings to create documentation for all new code.
\n\nYou may also find it helpful to look at this developer guide, which explains the organization of the micro-sam code.
\n\nTesting
\n\nRun the tests
\n\nThe tests for micro-sam are run with pytest
\n\nTo run the tests:
\n\npytest\nWriting your own tests
\n\nIf you have written new code, you will need to write tests to go with it.
\n\nUnit tests
\n\nUnit tests are the preferred style of tests for user contributions. Unit tests check small, isolated parts of the code for correctness. If your code is too complicated to write unit tests easily, you may need to consider breaking it up into smaller functions that are easier to test.
\n\nTests involving napari
\n\nIn cases where tests must use the napari viewer, these tips might be helpful (in particular, the make_napari_viewer_proxy fixture).
These kinds of tests should be used only in limited circumstances. Developers are advised to prefer smaller unit tests, and avoid integration tests wherever possible.
\n\nCode coverage
\n\nPytest uses the pytest-cov plugin to automatically determine which lines of code are covered by tests.
\n\nA short summary report is printed to the terminal output whenever you run pytest. The full results are also automatically written to a file named coverage.xml.
The Coverage Gutters VSCode extension is useful for visualizing which parts of the code need better test coverage. PyCharm professional has a similar feature, and you may be able to find similar tools for your preferred editor.
\n\nWe also use codecov.io to display the code coverage results from our Github Actions continuous integration.
\n\nOpen a pull request
\n\nOnce you've made changes to the code and written some tests to go with it, you are ready to open a pull request. You can mark your pull request as a draft if you are still working on it, and still get the benefit of discussing the best approach with maintainers.
\n\nRemember that typically changes to micro-sam are made branching off from the development branch. So, you will need to open your pull request to merge back into the dev branch like this.
Optional: Build the documentation
\n\nWe use pdoc to build the documentation.
\n\nTo build the documentation locally, run this command:
\n\npython build_doc.py\nThis will start a local server and display the HTML documentation. Any changes you make to the documentation will be updated in real time (you may need to refresh your browser to see the changes).
\n\nIf you want to save the HTML files, append --out to the command, like this:
python build_doc.py --out\nThis will save the HTML files into a new directory named tmp.
You can add content to the documentation in two ways:
\n\n- \n
- By adding or updating google style python docstrings in the micro-sam code.\n
- \n
- pdoc will automatically find and include docstrings in the documentation. \n
\n - By adding or editing markdown files in the micro-sam
docdirectory.\n- \n
- If you add a new markdown file to the documentation, you must tell pdoc that it exists by adding a line to the
micro_sam/__init__.pymodule docstring (eg:.. include:: ../doc/my_amazing_new_docs_page.md). Otherwise it will not be included in the final documentation build! \n
\n - If you add a new markdown file to the documentation, you must tell pdoc that it exists by adding a line to the
Optional: Benchmark performance
\n\nThere are a number of options you can use to benchmark performance, and identify problems like slow run times or high memory use in micro-sam.
\n\n\n\nRun the benchmark script
\n\nThere is a performance benchmark script available in the micro-sam repository at development/benchmark.py.
To run the benchmark script:
\n\npython development/benchmark.py --model_type vit_t --device cpu`\nFor more details about the user input arguments for the micro-sam benchmark script, see the help:
\n\npython development/benchmark.py --help\nLine profiling
\n\nFor more detailed line by line performance results, we can use line-profiler.
\n\n\n\n\nline_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either line_profiler or the Python standard library's cProfile or profile modules, depending on what is available.
\n
To do line-by-line profiling:
\n\n- \n
- Ensure you have line profiler installed:
python -m pip install line_profiler\n - Add
@profiledecorator to any function in the call stack \n - Run
kernprof -lv benchmark.py --model_type vit_t --device cpu\n
For more details about how to use line-profiler and kernprof, see the documentation.
\n\nFor more details about the user input arguments for the micro-sam benchmark script, see the help:
\n\npython development/benchmark.py --help\nSnakeviz visualization
\n\nFor more detailed visualizations of profiling results, we use snakeviz.
\n\n\n\n\nSnakeViz is a browser based graphical viewer for the output of Python\u2019s cProfile module
\n
- \n
- Ensure you have snakeviz installed:
python -m pip install snakeviz\n - Generate profile file:
python -m cProfile -o program.prof benchmark.py --model_type vit_h --device cpu\n - Visualize profile file:
snakeviz program.prof\n
For more details about how to use snakeviz, see the documentation.
\n\nMemory profiling with memray
\n\nIf you need to investigate memory use specifically, we use memray.
\n\n\n\n\nMemray is a memory profiler for Python. It can track memory allocations in Python code, in native extension modules, and in the Python interpreter itself. It can generate several different types of reports to help you analyze the captured memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more fine-grained profiling tasks.
\n
For more details about how to use memray, see the documentation.
\n\nUsing micro_sam on BAND
\n\nBAND is a service offered by EMBL Heidelberg that gives access to a virtual desktop for image analysis tasks. It is free to use and micro_sam is installed there.\nIn order to use BAND and start micro_sam on it follow these steps:
Start BAND
\n\n- \n
- Go to https://band.embl.de/ and click Login. If you have not used BAND before you will need to register for BAND. Currently you can only sign up via a google account. \n
- Launch a BAND desktop with sufficient resources. It's particularly important to select a GPU. The settings from the image below are a good choice. \n
- Go to the desktop by clicking GO TO DESKTOP in the Running Desktops menu. See also the screenshot below. \n
Start micro_sam in BAND
\n\n- \n
- Select Applications->Image Analysis->uSAM (see screenshot)\n
- This will open the micro_sam menu, where you can select the tool you want to use (see screenshot). Note: this may take a few minutes.\n
- For testing if the tool works, it's best to use the 2d annotator first.\n
- \n
- You can find an example image to use here:
/scratch/cajal-connectomics/hela-2d-image.png. Select it via Select image. (see screenshot)\n \n
\n - You can find an example image to use here:
- Then press 2d annotator and the tool will start. \n
Transfering data to BAND
\n\nTo copy data to and from BAND you can use any cloud storage, e.g. ownCloud, dropbox or google drive. For this, it's important to note that copy and paste, which you may need for accessing links on BAND, works a bit different in BAND:
\n\n- \n
- To copy text into BAND you first need to copy it on your computer (e.g. via selecting it +
ctrl + c). \n - Then go to the browser window with BAND and press
ctrl + shift + alt. This will open a side window where you can paste your text viactrl + v. \n - Then select the text in this window and copy it via
ctrl + c. \n - Now you can close the side window via
ctrl + shift + altand paste the text in band viactrl + v\n
The video below shows how to copy over a link from owncloud and then download the data on BAND using copy and paste:
\n\n\n"}, {"fullname": "micro_sam.bioimageio", "modulename": "micro_sam.bioimageio", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.bioimageio.model_export", "modulename": "micro_sam.bioimageio.model_export", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.bioimageio.model_export.DEFAULTS", "modulename": "micro_sam.bioimageio.model_export", "qualname": "DEFAULTS", "kind": "variable", "doc": "\n", "default_value": "{'authors': [Author(affiliation='University Goettingen', email=None, orcid=None, name='Anwai Archit', github_user='anwai98'), Author(affiliation='University Goettingen', email=None, orcid=None, name='Constantin Pape', github_user='constantinpape')], 'description': 'Finetuned Segment Anything Model for Microscopy', 'cite': [CiteEntry(text='Archit et al. Segment Anything for Microscopy', doi='10.1101/2023.08.21.554208', url=None)], 'tags': ['segment-anything', 'instance-segmentation']}"}, {"fullname": "micro_sam.bioimageio.model_export.export_sam_model", "modulename": "micro_sam.bioimageio.model_export", "qualname": "export_sam_model", "kind": "function", "doc": "Export SAM model to BioImage.IO model format.
\n\nThe exported model can be uploaded to bioimage.io and\nbe used in tools that support the BioImage.IO model format.
\n\nArguments:
\n\n- \n
- image: The image for generating test data. \n
- label_image: The segmentation correspoding to
image.\nIt is used to derive prompt inputs for the model. \n - model_type: The type of the SAM model. \n
- name: The name of the exported model. \n
- output_path: Where the exported model is saved. \n
- checkpoint_path: Optional checkpoint for loading the SAM model. \n
Wrapper around the SamPredictor.
\n\nThis model supports the same functionality as SamPredictor and can provide mask segmentations\nfrom box, point or mask input prompts.
\n\nArguments:
\n\n- \n
- model_type: The type of the model for the image encoder.\nCan be one of 'vit_b', 'vit_l', 'vit_h' or 'vit_t'.\nFor 'vit_t' support the 'mobile_sam' package has to be installed. \n
Initializes internal Module state, shared by both nn.Module and ScriptModule.
\n", "signature": "(model_type: str)"}, {"fullname": "micro_sam.bioimageio.predictor_adaptor.PredictorAdaptor.sam", "modulename": "micro_sam.bioimageio.predictor_adaptor", "qualname": "PredictorAdaptor.sam", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.bioimageio.predictor_adaptor.PredictorAdaptor.load_state_dict", "modulename": "micro_sam.bioimageio.predictor_adaptor", "qualname": "PredictorAdaptor.load_state_dict", "kind": "function", "doc": "Copies parameters and buffers from state_dict into\nthis module and its descendants. If strict is True, then\nthe keys of state_dict must exactly match the keys returned\nby this module's ~torch.nn.Module.state_dict() function.
If assign is True the optimizer must be created after\nthe call to load_state_dict.
Arguments:
\n\n- \n
- state_dict (dict): a dict containing parameters and\npersistent buffers. \n
- strict (bool, optional): whether to strictly enforce that the keys\nin
state_dictmatch the keys returned by this module's\n~torch.nn.Module.state_dict()function. Default:True\n - assign (bool, optional): whether to assign items in the state\ndictionary to their corresponding keys in the module instead\nof copying them inplace into the module's current parameters and buffers.\nWhen
False, the properties of the tensors in the current\nmodule are preserved while whenTrue, the properties of the\nTensors in the state dict are preserved.\nDefault:False\n
Returns:
\n\n\n\n\n\n
NamedTuplewithmissing_keysandunexpected_keysfields:\n * missing_keys is a list of str containing the missing keys\n * unexpected_keys is a list of str containing the unexpected keys
Note:
\n\n\n\n", "signature": "(self, state):", "funcdef": "def"}, {"fullname": "micro_sam.bioimageio.predictor_adaptor.PredictorAdaptor.forward", "modulename": "micro_sam.bioimageio.predictor_adaptor", "qualname": "PredictorAdaptor.forward", "kind": "function", "doc": "If a parameter or buffer is registered as
\nNoneand its corresponding key\n exists instate_dict,load_state_dict()will raise a\nRuntimeError.
Arguments:
\n\n- \n
- image: torch inputs of dimensions B x C x H x W \n
- box_prompts: box coordinates of dimensions B x OBJECTS x 4 \n
- point_prompts: point coordinates of dimension B x OBJECTS x POINTS x 2 \n
- point_labels: point labels of dimension B x OBJECTS x POINTS \n
- mask_prompts: mask prompts of dimension B x OBJECTS x 256 x 256 \n
- embeddings: precomputed image embeddings B x 256 x 64 x 64 \n
Returns:
\n", "signature": "(\tself,\timage: torch.Tensor,\tbox_prompts: Optional[torch.Tensor] = None,\tpoint_prompts: Optional[torch.Tensor] = None,\tpoint_labels: Optional[torch.Tensor] = None,\tmask_prompts: Optional[torch.Tensor] = None,\tembeddings: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation", "modulename": "micro_sam.evaluation", "kind": "module", "doc": "Functionality for evaluating Segment Anything models on microscopy data.
\n"}, {"fullname": "micro_sam.evaluation.evaluation", "modulename": "micro_sam.evaluation.evaluation", "kind": "module", "doc": "Evaluation functionality for segmentation predictions from micro_sam.evaluation.automatic_mask_generation\nand micro_sam.evaluation.inference.
Run evaluation for instance segmentation predictions.
\n\nArguments:
\n\n- \n
- gt_paths: The list of paths to ground-truth images. \n
- prediction_paths: The list of paths with the instance segmentations to evaluate. \n
- save_path: Optional path for saving the results. \n
- verbose: Whether to print the progress. \n
Returns:
\n\n\n\n", "signature": "(\tgt_paths: List[Union[str, os.PathLike]],\tprediction_paths: List[Union[str, os.PathLike]],\tsave_path: Union[str, os.PathLike, NoneType] = None,\tverbose: bool = True) -> pandas.core.frame.DataFrame:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.evaluation.run_evaluation_for_iterative_prompting", "modulename": "micro_sam.evaluation.evaluation", "qualname": "run_evaluation_for_iterative_prompting", "kind": "function", "doc": "A DataFrame that contains the evaluation results.
\n
Run evaluation for iterative prompt-based segmentation predictions.
\n\nArguments:
\n\n- \n
- gt_paths: The list of paths to ground-truth images. \n
- prediction_root: The folder with the iterative prompt-based instance segmentations to evaluate. \n
- experiment_folder: The folder where all the experiment results are stored. \n
- start_with_box_prompt: Whether to evaluate on experiments with iterative prompting starting with box. \n
Returns:
\n\n\n\n", "signature": "(\tgt_paths: List[Union[str, os.PathLike]],\tprediction_root: Union[os.PathLike, str],\texperiment_folder: Union[os.PathLike, str],\tstart_with_box_prompt: bool = False,\toverwrite_results: bool = False) -> pandas.core.frame.DataFrame:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.experiments", "modulename": "micro_sam.evaluation.experiments", "kind": "module", "doc": "A DataFrame that contains the evaluation results.
\n
Predefined experiment settings for experiments with different prompt strategies.
\n"}, {"fullname": "micro_sam.evaluation.experiments.ExperimentSetting", "modulename": "micro_sam.evaluation.experiments", "qualname": "ExperimentSetting", "kind": "variable", "doc": "\n", "default_value": "typing.Dict"}, {"fullname": "micro_sam.evaluation.experiments.full_experiment_settings", "modulename": "micro_sam.evaluation.experiments", "qualname": "full_experiment_settings", "kind": "function", "doc": "The full experiment settings.
\n\nArguments:
\n\n- \n
- use_boxes: Whether to run the experiments with or without boxes. \n
- positive_range: The different number of positive points that will be used.\nBy defaul the values are set to [1, 2, 4, 8, 16]. \n
- negative_range: The different number of negative points that will be used.\nBy defaul the values are set to [0, 1, 2, 4, 8, 16]. \n
Returns:
\n\n\n\n", "signature": "(\tuse_boxes: bool = False,\tpositive_range: Optional[List[int]] = None,\tnegative_range: Optional[List[int]] = None) -> List[Dict]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.experiments.default_experiment_settings", "modulename": "micro_sam.evaluation.experiments", "qualname": "default_experiment_settings", "kind": "function", "doc": "The list of experiment settings.
\n
The three default experiment settings.
\n\nFor the default experiments we use a single positive prompt,\ntwo positive and four negative prompts and box prompts.
\n\nReturns:
\n\n\n\n", "signature": "() -> List[Dict]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.experiments.get_experiment_setting_name", "modulename": "micro_sam.evaluation.experiments", "qualname": "get_experiment_setting_name", "kind": "function", "doc": "The list of experiment settings.
\n
Get the name for the given experiment setting.
\n\nArguments:
\n\n- \n
- setting: The experiment setting. \n
Returns:
\n\n\n\n", "signature": "(setting: Dict) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.inference", "modulename": "micro_sam.evaluation.inference", "kind": "module", "doc": "The name for this experiment setting.
\n
Inference with Segment Anything models and different prompt strategies.
\n"}, {"fullname": "micro_sam.evaluation.inference.precompute_all_embeddings", "modulename": "micro_sam.evaluation.inference", "qualname": "precompute_all_embeddings", "kind": "function", "doc": "Precompute all image embeddings.
\n\nTo enable running different inference tasks in parallel afterwards.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_paths: The image file paths. \n
- embedding_dir: The directory where the embeddings will be saved. \n
Precompute all point prompts.
\n\nTo enable running different inference tasks in parallel afterwards.
\n\nArguments:
\n\n- \n
- gt_paths: The file paths to the ground-truth segmentations. \n
- prompt_save_dir: The directory where the prompt files will be saved. \n
- prompt_settings: The settings for which the prompts will be computed. \n
Run segment anything inference for multiple images using prompts derived from groundtruth.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_paths: The image file paths. \n
- gt_paths: The ground-truth segmentation file paths. \n
- embedding_dir: The directory where the image embddings will be saved or are already saved. \n
- use_points: Whether to use point prompts. \n
- use_boxes: Whether to use box prompts \n
- n_positives: The number of positive point prompts that will be sampled. \n
- n_negativess: The number of negative point prompts that will be sampled. \n
- dilation: The dilation factor for the radius around the ground-truth object\naround which points will not be sampled. \n
- prompt_save_dir: The directory where point prompts will be saved or are already saved.\nThis enables running multiple experiments in a reproducible manner. \n
- batch_size: The batch size used for batched prediction. \n
Run segment anything inference for multiple images using prompts iteratively\n derived from model outputs and groundtruth
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_paths: The image file paths. \n
- gt_paths: The ground-truth segmentation file paths. \n
- embedding_dir: The directory where the image embeddings will be saved or are already saved. \n
- prediction_dir: The directory where the predictions from SegmentAnything will be saved per iteration. \n
- start_with_box_prompt: Whether to use the first prompt as bounding box or a single point \n
- dilation: The dilation factor for the radius around the ground-truth object\naround which points will not be sampled. \n
- batch_size: The batch size used for batched predictions. \n
- n_iterations: The number of iterations for iterative prompting. \n
- use_masks: Whether to make use of logits from previous prompt-based segmentation \n
Inference and evaluation for the automatic instance segmentation functionality.
\n"}, {"fullname": "micro_sam.evaluation.instance_segmentation.default_grid_search_values_amg", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "default_grid_search_values_amg", "kind": "function", "doc": "Default grid-search parameter for AMG-based instance segmentation.
\n\nReturn grid search values for the two most important parameters:
\n\n- \n
pred_iou_thresh, the threshold for keeping objects according to the IoU predicted by the model. \nstability_score_thresh, the theshold for keepong objects according to their stability. \n
Arguments:
\n\n- \n
- iou_thresh_values: The values for
pred_iou_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n - stability_score_values: The values for
stability_score_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n
Returns:
\n\n\n\n", "signature": "(\tiou_thresh_values: Optional[List[float]] = None,\tstability_score_values: Optional[List[float]] = None) -> Dict[str, List[float]]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.default_grid_search_values_instance_segmentation_with_decoder", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "default_grid_search_values_instance_segmentation_with_decoder", "kind": "function", "doc": "The values for grid search.
\n
Default grid-search parameter for decoder-based instance segmentation.
\n\nArguments:
\n\n- \n
- center_distance_threshold_values: The values for
center_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - boundary_distance_threshold_values: The values for
boundary_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - distance_smoothing_values: The values for
distance_smoothingused in the gridsearch.\nBy default values in the range from 1.0 to 2.0 with a stepsize of 0.1 will be used. \n - min_size_values: The values for
min_sizeused in the gridsearch.\nBy default the values 50, 100 and 200 are used. \n
Returns:
\n\n\n\n", "signature": "(\tcenter_distance_threshold_values: Optional[List[float]] = None,\tboundary_distance_threshold_values: Optional[List[float]] = None,\tdistance_smoothing_values: Optional[List[float]] = None,\tmin_size_values: Optional[List[float]] = None) -> Dict[str, List[float]]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.run_instance_segmentation_grid_search", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "run_instance_segmentation_grid_search", "kind": "function", "doc": "The values for grid search.
\n
Run grid search for automatic mask generation.
\n\nThe parameters and their respective value ranges for the grid search are specified via the\n'grid_search_values' argument. For example, to run a grid search over the parameters 'pred_iou_thresh'\nand 'stability_score_thresh', you can pass the following:
\n\ngrid_search_values = {\n \"pred_iou_thresh\": [0.6, 0.7, 0.8, 0.9],\n \"stability_score_thresh\": [0.6, 0.7, 0.8, 0.9],\n}\nAll combinations of the parameters will be checked.
\n\nYou can use the functions default_grid_search_values_instance_segmentation_with_decoder\nor default_grid_search_values_amg to get the default grid search parameters for the two\nrespective instance segmentation methods.
Arguments:
\n\n- \n
- segmenter: The class implementing the instance segmentation functionality. \n
- grid_search_values: The grid search values for parameters of the
generatefunction. \n - image_paths: The input images for the grid search. \n
- gt_paths: The ground-truth segmentation for the grid search. \n
- result_dir: Folder to cache the evaluation results per image. \n
- embedding_dir: Folder to cache the image embeddings. \n
- fixed_generate_kwargs: Fixed keyword arguments for the
generatemethod of the segmenter. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
- image_key: Key for loading the image data from a more complex file format like HDF5.\nIf not given a simple image format like tif is assumed. \n
- gt_key: Key for loading the ground-truth data from a more complex file format like HDF5.\nIf not given a simple image format like tif is assumed. \n
- rois: Region of interests to resetrict the evaluation to. \n
Run inference for automatic mask generation.
\n\nArguments:
\n\n- \n
- segmenter: The class implementing the instance segmentation functionality. \n
- image_paths: The input images. \n
- embedding_dir: Folder to cache the image embeddings. \n
- prediction_dir: Folder to save the predictions. \n
- generate_kwargs: The keyword arguments for the
generatemethod of the segmenter. \n
Evaluate gridsearch results.
\n\nArguments:
\n\n- \n
- result_dir: The folder with the gridsearch results. \n
- grid_search_parameters: The names for the gridsearch parameters. \n
- criterion: The metric to use for determining the best parameters. \n
Returns:
\n\n\n\n", "signature": "(\tresult_dir: Union[str, os.PathLike],\tgrid_search_parameters: List[str],\tcriterion: str = 'mSA') -> Tuple[Dict[str, Any], float]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.save_grid_search_best_params", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "save_grid_search_best_params", "kind": "function", "doc": "\n", "signature": "(best_kwargs, best_msa, grid_search_result_dir=None):", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.run_instance_segmentation_grid_search_and_inference", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "run_instance_segmentation_grid_search_and_inference", "kind": "function", "doc": "The best parameter setting.\n The evaluation score for the best setting.
\n
Run grid search and inference for automatic mask generation.
\n\nPlease refer to the documentation of run_instance_segmentation_grid_search\nfor details on how to specify the grid search parameters.
Arguments:
\n\n- \n
- segmenter: The class implementing the instance segmentation functionality. \n
- grid_search_values: The grid search values for parameters of the
generatefunction. \n - val_image_paths: The input images for the grid search. \n
- val_gt_paths: The ground-truth segmentation for the grid search. \n
- test_image_paths: The input images for inference. \n
- embedding_dir: Folder to cache the image embeddings. \n
- prediction_dir: Folder to save the predictions. \n
- result_dir: Folder to cache the evaluation results per image. \n
- fixed_generate_kwargs: Fixed keyword arguments for the
generatemethod of the segmenter. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
Inference and evaluation for the LIVECell dataset and\nthe different cell lines contained in it.
\n"}, {"fullname": "micro_sam.evaluation.livecell.CELL_TYPES", "modulename": "micro_sam.evaluation.livecell", "qualname": "CELL_TYPES", "kind": "variable", "doc": "\n", "default_value": "['A172', 'BT474', 'BV2', 'Huh7', 'MCF7', 'SHSY5Y', 'SkBr3', 'SKOV3']"}, {"fullname": "micro_sam.evaluation.livecell.livecell_inference", "modulename": "micro_sam.evaluation.livecell", "qualname": "livecell_inference", "kind": "function", "doc": "Run inference for livecell with a fixed prompt setting.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segment anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- use_points: Whether to use point prompts. \n
- use_boxes: Whether to use box prompts. \n
- n_positives: The number of positive point prompts. \n
- n_negatives: The number of negative point prompts. \n
- prompt_folder: The folder where the prompts should be saved. \n
- predictor: The segment anything predictor. \n
Run precomputation of val and test image embeddings for livecell.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segmenta anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- n_val_per_cell_type: The number of validation images per cell type. \n
Run inference on livecell with iterative prompting setting.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segment anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- start_with_box_prompt: Whether to use the first prompt as bounding box or a single point. \n
- use_masks: Whether to make use of logits from previous prompt-based segmentation. \n
Run automatic mask generation grid-search and inference for livecell.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segmenta anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- iou_thresh_values: The values for
pred_iou_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n - stability_score_values: The values for
stability_score_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
- n_val_per_cell_type: The number of validation images per cell type. \n
Returns:
\n\n\n\n", "signature": "(\tcheckpoint: Union[str, os.PathLike],\tinput_folder: Union[str, os.PathLike],\tmodel_type: str,\texperiment_folder: Union[str, os.PathLike],\tiou_thresh_values: Optional[List[float]] = None,\tstability_score_values: Optional[List[float]] = None,\tverbose_gs: bool = False,\tn_val_per_cell_type: int = 25) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.livecell.run_livecell_instance_segmentation_with_decoder", "modulename": "micro_sam.evaluation.livecell", "qualname": "run_livecell_instance_segmentation_with_decoder", "kind": "function", "doc": "The path where the predicted images are stored.
\n
Run automatic mask generation grid-search and inference for livecell.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segmenta anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- center_distance_threshold_values: The values for
center_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - boundary_distance_threshold_values: The values for
boundary_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - distance_smoothing_values: The values for
distance_smoothingused in the gridsearch.\nBy default values in the range from 1.0 to 2.0 with a stepsize of 0.1 will be used. \n - min_size_values: The values for
min_sizeused in the gridsearch.\nBy default the values 50, 100 and 200 are used. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
- n_val_per_cell_type: The number of validation images per cell type. \n
Returns:
\n\n\n\n", "signature": "(\tcheckpoint: Union[str, os.PathLike],\tinput_folder: Union[str, os.PathLike],\tmodel_type: str,\texperiment_folder: Union[str, os.PathLike],\tcenter_distance_threshold_values: Optional[List[float]] = None,\tboundary_distance_threshold_values: Optional[List[float]] = None,\tdistance_smoothing_values: Optional[List[float]] = None,\tmin_size_values: Optional[List[float]] = None,\tverbose_gs: bool = False,\tn_val_per_cell_type: int = 25) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.livecell.run_livecell_inference", "modulename": "micro_sam.evaluation.livecell", "qualname": "run_livecell_inference", "kind": "function", "doc": "The path where the predicted images are stored.
\n
Run LIVECell inference with command line tool.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.livecell.run_livecell_evaluation", "modulename": "micro_sam.evaluation.livecell", "qualname": "run_livecell_evaluation", "kind": "function", "doc": "Run LiveCELL evaluation with command line tool.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.model_comparison", "modulename": "micro_sam.evaluation.model_comparison", "kind": "module", "doc": "Functionality for qualitative comparison of Segment Anything models on microscopy data.
\n"}, {"fullname": "micro_sam.evaluation.model_comparison.generate_data_for_model_comparison", "modulename": "micro_sam.evaluation.model_comparison", "qualname": "generate_data_for_model_comparison", "kind": "function", "doc": "Generate samples for qualitative model comparison.
\n\nThis precomputes the input for model_comparison and model_comparison_with_napari.
Arguments:
\n\n- \n
- loader: The torch dataloader from which samples are drawn. \n
- output_folder: The folder where the samples will be saved. \n
- model_type1: The first model to use for comparison.\nThe value needs to be a valid model_type for
micro_sam.util.get_sam_model. \n - model_type2: The second model to use for comparison.\nThe value needs to be a valid model_type for
micro_sam.util.get_sam_model. \n - n_samples: The number of samples to draw from the dataloader. \n
- checkpoint1: Optional checkpoint for the first model. \n
- checkpoint2: Optional checkpoint for the second model. \n
Create images for a qualitative model comparision.
\n\nArguments:
\n\n- \n
- output_folder: The folder with the data precomputed by
generate_data_for_model_comparison. \n - n_images_per_sample: The number of images to generate per precomputed sample. \n
- min_size: The min size of ground-truth objects to take into account. \n
- plot_folder: The folder where to save the plots. If not given the plots will be displayed. \n
- point_radius: The radius of the point overlay. \n
- outline_dilation: The dilation factor of the outline overlay. \n
Use napari to display the qualtiative comparison results for two models.
\n\nArguments:
\n\n- \n
- output_folder: The folder with the data precomputed by
generate_data_for_model_comparison. \n - show_points: Whether to show the results for point or for box prompts. \n
Default grid-search parameters for multi-dimensional prompt-based instance segmentation.
\n\nArguments:
\n\n- \n
- iou_threshold_values: The values for
iou_thresholdused in the grid-search.\nBy default values in the range from 0.5 to 0.9 with a stepsize of 0.1 will be used. \n - projection_method_values: The values for
projectionmethod used in the grid-search.\nBy default the valuesmask,bounding_boxandpointsare used. \n - box_extension_values: The values for
box_extensionused in the grid-search.\nBy default values in the range from 0 to 0.25 with a stepsize of 0.025 will be used. \n
Returns:
\n\n\n\n", "signature": "(\tiou_threshold_values: Optional[List[float]] = None,\tprojection_method_values: Union[str, dict, NoneType] = None,\tbox_extension_values: Union[float, int, NoneType] = None) -> Dict[str, List]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.multi_dimensional_segmentation.segment_slices_from_ground_truth", "modulename": "micro_sam.evaluation.multi_dimensional_segmentation", "qualname": "segment_slices_from_ground_truth", "kind": "function", "doc": "The values for grid search.
\n
Segment all objects in a volume by prompt-based segmentation in one slice per object.
\n\nThis function first segments each object in the respective specified slice using interactive\n(prompt-based) segmentation functionality. Then it segments the particular object in the\nremaining slices in the volume.
\n\nArguments:
\n\n- \n
- volume: The input volume. \n
- ground_truth: The label volume with instance segmentations. \n
- model_type: Choice of segment anything model. \n
- checkpoint_path: Path to the model checkpoint. \n
- embedding_path: Path to cache the computed embeddings. \n
- iou_threshold: The criterion to decide whether to link the objects in the consecutive slice's segmentation. \n
- projection: The projection (prompting) method to generate prompts for consecutive slices. \n
- box_extension: Extension factor for increasing the box size after projection. \n
- device: The selected device for computation. \n
- interactive_seg_mode: Method for guiding prompt-based instance segmentation. \n
- verbose: Whether to get the trace for projected segmentations. \n
- return_segmentation: Whether to return the segmented volume. \n
- min_size: The minimal size for evaluating an object in the ground-truth.\nThe size is measured within the central slice. \n
Run grid search for prompt-based multi-dimensional instance segmentation.
\n\nThe parameters and their respective value ranges for the grid search are specified via the\ngrid_search_values argument. For example, to run a grid search over the parameters iou_threshold,\nprojection and box_extension, you can pass the following:
grid_search_values = {\n \"iou_threshold\": [0.5, 0.6, 0.7, 0.8, 0.9],\n \"projection\": [\"mask\", \"bounding_box\", \"points\"],\n \"box_extension\": [0, 0.1, 0.2, 0.3, 0.4, 0,5],\n}\nAll combinations of the parameters will be checked.\nIf passed None, the function default_grid_search_values_multi_dimensional_segmentation is used\nto get the default grid search parameters for the instance segmentation method.
Arguments:
\n\n- \n
- volume: The input volume. \n
- ground_truth: The label volume with instance segmentations. \n
- model_type: Choice of segment anything model. \n
- checkpoint_path: Path to the model checkpoint. \n
- embedding_path: Path to cache the computed embeddings. \n
- result_path: Path to save the grid search results. \n
- interactive_seg_mode: Method for guiding prompt-based instance segmentation. \n
- verbose: Whether to get the trace for projected segmentations. \n
- grid_search_values: The grid search values for parameters of the
segment_slices_from_ground_truthfunction. \n - min_size: The minimal size for evaluating an object in the ground-truth.\nThe size is measured within the central slice. \n
Run batched inference for input prompts.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- image: The input image. \n
- batch_size: The batch size to use for inference. \n
- boxes: The box prompts. Array of shape N_PROMPTS x 4.\nThe bounding boxes are represented by [MIN_X, MIN_Y, MAX_X, MAX_Y]. \n
- points: The point prompt coordinates. Array of shape N_PROMPTS x 1 x 2.\nThe points are represented by their coordinates [X, Y], which are given\nin the last dimension. \n
- point_labels: The point prompt labels. Array of shape N_PROMPTS x 1.\nThe labels are either 0 (negative prompt) or 1 (positive prompt). \n
- multimasking: Whether to predict with 3 or 1 mask. \n
- embedding_path: Cache path for the image embeddings. \n
- return_instance_segmentation: Whether to return a instance segmentation\nor the individual mask data. \n
- segmentation_ids: Fixed segmentation ids to assign to the masks\nderived from the prompts. \n
- reduce_multimasking: Whether to choose the most likely masks with\nhighest ious from multimasking \n
- logits_masks: The logits masks. Array of shape N_PROMPTS x 1 x 256 x 256.\nWhether to use the logits masks from previous segmentation. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\timage: numpy.ndarray,\tbatch_size: int,\tboxes: Optional[numpy.ndarray] = None,\tpoints: Optional[numpy.ndarray] = None,\tpoint_labels: Optional[numpy.ndarray] = None,\tmultimasking: bool = False,\tembedding_path: Union[str, os.PathLike, NoneType] = None,\treturn_instance_segmentation: bool = True,\tsegmentation_ids: Optional[list] = None,\treduce_multimasking: bool = True,\tlogits_masks: Optional[torch.Tensor] = None):", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation", "modulename": "micro_sam.instance_segmentation", "kind": "module", "doc": "The predicted segmentation masks.
\n
Automated instance segmentation functionality.\nThe classes implemented here extend the automatic instance segmentation from Segment Anything:\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html
\n"}, {"fullname": "micro_sam.instance_segmentation.mask_data_to_segmentation", "modulename": "micro_sam.instance_segmentation", "qualname": "mask_data_to_segmentation", "kind": "function", "doc": "Convert the output of the automatic mask generation to an instance segmentation.
\n\nArguments:
\n\n- \n
- masks: The outputs generated by AutomaticMaskGenerator or EmbeddingMaskGenerator.\nOnly supports output_mode=binary_mask. \n
- with_background: Whether the segmentation has background. If yes this function assures that the largest\nobject in the output will be mapped to zero (the background value). \n
- min_object_size: The minimal size of an object in pixels. \n
- max_object_size: The maximal size of an object in pixels. \n
Returns:
\n\n\n\n", "signature": "(\tmasks: List[Dict[str, Any]],\twith_background: bool,\tmin_object_size: int = 0,\tmax_object_size: Optional[int] = None) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.AMGBase", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase", "kind": "class", "doc": "The instance segmentation.
\n
Base class for the automatic mask generators.
\n", "bases": "abc.ABC"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.is_initialized", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.is_initialized", "kind": "variable", "doc": "Whether the mask generator has already been initialized.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.crop_list", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.crop_list", "kind": "variable", "doc": "The list of mask data after initialization.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.crop_boxes", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.crop_boxes", "kind": "variable", "doc": "The list of crop boxes.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.original_size", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.original_size", "kind": "variable", "doc": "The original image size.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.get_state", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.get_state", "kind": "function", "doc": "Get the initialized state of the mask generator.
\n\nReturns:
\n\n\n\n", "signature": "(self) -> Dict[str, Any]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.set_state", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.set_state", "kind": "function", "doc": "State of the mask generator.
\n
Set the state of the mask generator.
\n\nArguments:
\n\n- \n
- state: The state of the mask generator, e.g. from serialized state. \n
Clear the state of the mask generator.
\n", "signature": "(self):", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.AutomaticMaskGenerator", "modulename": "micro_sam.instance_segmentation", "qualname": "AutomaticMaskGenerator", "kind": "class", "doc": "Generates an instance segmentation without prompts, using a point grid.
\n\nThis class implements the same logic as\nhttps://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py\nIt decouples the computationally expensive steps of generating masks from the cheap post-processing operation\nto filter these masks to enable grid search and interactively changing the post-processing.
\n\nUse this class as follows:
\n\namg = AutomaticMaskGenerator(predictor)\namg.initialize(image) # Initialize the masks, this takes care of all expensive computations.\nmasks = amg.generate(pred_iou_thresh=0.8) # Generate the masks. This is fast and enables testing parameters\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- points_per_side: The number of points to be sampled along one side of the image.\nIf None,
point_gridsmust provide explicit point sampling. \n - points_per_batch: The number of points run simultaneously by the model.\nHigher numbers may be faster but use more GPU memory. \n
- crop_n_layers: If >0, the mask prediction will be run again on crops of the image. \n
- crop_overlap_ratio: Sets the degree to which crops overlap. \n
- crop_n_points_downscale_factor: How the number of points is downsampled when predicting with crops. \n
- point_grids: A lisst over explicit grids of points used for sampling masks.\nNormalized to [0, 1] with respect to the image coordinate system. \n
- stability_score_offset: The amount to shift the cutoff when calculating the stability score. \n
Initialize image embeddings and masks for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - verbose: Whether to print computation progress. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Generate instance segmentation for the currently initialized image.
\n\nArguments:
\n\n- \n
- pred_iou_thresh: Filter threshold in [0, 1], using the mask quality predicted by the model. \n
- stability_score_thresh: Filter threshold in [0, 1], using the stability of the mask\nunder changes to the cutoff used to binarize the model prediction. \n
- box_nms_thresh: The IoU threshold used by nonmax suppression to filter duplicate masks. \n
- crop_nms_thresh: The IoU threshold used by nonmax suppression to filter duplicate masks between crops. \n
- min_mask_region_area: Minimal size for the predicted masks. \n
- output_mode: The form masks are returned in. \n
Returns:
\n\n\n\n", "signature": "(\tself,\tpred_iou_thresh: float = 0.88,\tstability_score_thresh: float = 0.95,\tbox_nms_thresh: float = 0.7,\tcrop_nms_thresh: float = 0.7,\tmin_mask_region_area: int = 0,\toutput_mode: str = 'binary_mask') -> List[Dict[str, Any]]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.TiledAutomaticMaskGenerator", "modulename": "micro_sam.instance_segmentation", "qualname": "TiledAutomaticMaskGenerator", "kind": "class", "doc": "The instance segmentation masks.
\n
Generates an instance segmentation without prompts, using a point grid.
\n\nImplements the same functionality as AutomaticMaskGenerator but for tiled embeddings.
Arguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- points_per_side: The number of points to be sampled along one side of the image.\nIf None,
point_gridsmust provide explicit point sampling. \n - points_per_batch: The number of points run simultaneously by the model.\nHigher numbers may be faster but use more GPU memory. \n
- point_grids: A lisst over explicit grids of points used for sampling masks.\nNormalized to [0, 1] with respect to the image coordinate system. \n
- stability_score_offset: The amount to shift the cutoff when calculating the stability score. \n
Initialize image embeddings and masks for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - tile_shape: The tile shape for embedding prediction. \n
- halo: The overlap of between tiles. \n
- verbose: Whether to print computation progress. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Adapter to contain the UNETR decoder in a single module.
\n\nTo apply the decoder on top of pre-computed embeddings for\nthe segmentation functionality.\nSee also: https://github.com/constantinpape/torch-em/blob/main/torch_em/model/unetr.py
\n", "bases": "torch.nn.modules.module.Module"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.__init__", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.__init__", "kind": "function", "doc": "Initializes internal Module state, shared by both nn.Module and ScriptModule.
\n", "signature": "(unetr)"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.base", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.base", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.out_conv", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.out_conv", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv_out", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv_out", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.decoder_head", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.decoder_head", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.final_activation", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.final_activation", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.postprocess_masks", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.postprocess_masks", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.decoder", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.decoder", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv1", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv1", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv2", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv2", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv3", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv3", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv4", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv4", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.forward", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.forward", "kind": "function", "doc": "Defines the computation performed at every call.
\n\nShould be overridden by all subclasses.
\n\nAlthough the recipe for forward pass needs to be defined within\nthis function, one should call the Module instance afterwards\ninstead of this since the former takes care of running the\nregistered hooks while the latter silently ignores them.
Get UNETR model for automatic instance segmentation.
\n\nArguments:
\n\n- \n
- image_encoder: The image encoder of the SAM model.\nThis is used as encoder by the UNETR too. \n
- decoder_state: Optional decoder state to initialize the weights\nof the UNETR decoder. \n
- device: The device. \n
Returns:
\n\n\n\n", "signature": "(\timage_encoder: torch.nn.modules.module.Module,\tdecoder_state: Optional[collections.OrderedDict[str, torch.Tensor]] = None,\tdevice: Union[str, torch.device, NoneType] = None) -> torch.nn.modules.module.Module:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.get_decoder", "modulename": "micro_sam.instance_segmentation", "qualname": "get_decoder", "kind": "function", "doc": "The UNETR model.
\n
Get decoder to predict outputs for automatic instance segmentation
\n\nArguments:
\n\n- \n
- image_encoder: The image encoder of the SAM model. \n
- decoder_state: State to initialize the weights of the UNETR decoder. \n
- device: The device. \n
Returns:
\n\n\n\n", "signature": "(\timage_encoder: torch.nn.modules.module.Module,\tdecoder_state: collections.OrderedDict[str, torch.Tensor],\tdevice: Union[str, torch.device, NoneType] = None) -> micro_sam.instance_segmentation.DecoderAdapter:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.get_predictor_and_decoder", "modulename": "micro_sam.instance_segmentation", "qualname": "get_predictor_and_decoder", "kind": "function", "doc": "The decoder for instance segmentation.
\n
Load the SAM model (predictor) and instance segmentation decoder.
\n\nThis requires a checkpoint that contains the state for both predictor\nand decoder.
\n\nArguments:
\n\n- \n
- model_type: The type of the image encoder used in the SAM model. \n
- checkpoint_path: Path to the checkpoint from which to load the data. \n
- device: The device. \n
Returns:
\n\n\n\n", "signature": "(\tmodel_type: str,\tcheckpoint_path: Union[str, os.PathLike],\tdevice: Union[str, torch.device, NoneType] = None) -> Tuple[segment_anything.predictor.SamPredictor, micro_sam.instance_segmentation.DecoderAdapter]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder", "kind": "class", "doc": "The SAM predictor.\n The decoder for instance segmentation.
\n
Generates an instance segmentation without prompts, using a decoder.
\n\nImplements the same interface as AutomaticMaskGenerator.
Use this class as follows:
\n\nsegmenter = InstanceSegmentationWithDecoder(predictor, decoder)\nsegmenter.initialize(image) # Predict the image embeddings and decoder outputs.\nmasks = segmenter.generate(center_distance_threshold=0.75) # Generate the instance segmentation.\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- decoder: The decoder to predict intermediate representations\nfor instance segmentation. \n
Whether the mask generator has already been initialized.
\n"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder.initialize", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder.initialize", "kind": "function", "doc": "Initialize image embeddings and decoder predictions for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - verbose: Whether to be verbose. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Generate instance segmentation for the currently initialized image.
\n\nArguments:
\n\n- \n
- center_distance_threshold: Center distance predictions below this value will be\nused to find seeds (intersected with thresholded boundary distance predictions). \n
- boundary_distance_threshold: Boundary distance predictions below this value will be\nused to find seeds (intersected with thresholded center distance predictions). \n
- foreground_smoothing: Sigma value for smoothing the foreground predictions, to avoid\ncheckerboard artifacts in the prediction. \n
- foreground_threshold: Foreground predictions above this value will be used as foreground mask. \n
- distance_smoothing: Sigma value for smoothing the distance predictions. \n
- min_size: Minimal object size in the segmentation result. \n
- output_mode: The form masks are returned in. Pass None to directly return the instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\tself,\tcenter_distance_threshold: float = 0.5,\tboundary_distance_threshold: float = 0.5,\tforeground_threshold: float = 0.5,\tforeground_smoothing: float = 1.0,\tdistance_smoothing: float = 1.6,\tmin_size: int = 0,\toutput_mode: Optional[str] = 'binary_mask') -> List[Dict[str, Any]]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder.get_state", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder.get_state", "kind": "function", "doc": "The instance segmentation masks.
\n
Get the initialized state of the instance segmenter.
\n\nReturns:
\n\n\n\n", "signature": "(self) -> Dict[str, Any]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder.set_state", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder.set_state", "kind": "function", "doc": "Instance segmentation state.
\n
Set the state of the instance segmenter.
\n\nArguments:
\n\n- \n
- state: The instance segmentation state \n
Clear the state of the instance segmenter.
\n", "signature": "(self):", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.TiledInstanceSegmentationWithDecoder", "modulename": "micro_sam.instance_segmentation", "qualname": "TiledInstanceSegmentationWithDecoder", "kind": "class", "doc": "Same as InstanceSegmentationWithDecoder but for tiled image embeddings.
Initialize image embeddings and decoder predictions for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - verbose: Dummy input to be compatible with other function signatures. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Get the automatic mask generator class.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- is_tiled: Whether tiled embeddings are used. \n
- decoder: Decoder to predict instacne segmmentation. \n
- kwargs: The keyword arguments for the amg class. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tis_tiled: bool,\tdecoder: Optional[torch.nn.modules.module.Module] = None,\t**kwargs) -> Union[micro_sam.instance_segmentation.AMGBase, micro_sam.instance_segmentation.InstanceSegmentationWithDecoder]:", "funcdef": "def"}, {"fullname": "micro_sam.multi_dimensional_segmentation", "modulename": "micro_sam.multi_dimensional_segmentation", "kind": "module", "doc": "The automatic mask generator.
\n
Multi-dimensional segmentation with segment anything.
\n"}, {"fullname": "micro_sam.multi_dimensional_segmentation.PROJECTION_MODES", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "PROJECTION_MODES", "kind": "variable", "doc": "\n", "default_value": "('box', 'mask', 'points', 'points_and_mask', 'single_point')"}, {"fullname": "micro_sam.multi_dimensional_segmentation.segment_mask_in_volume", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "segment_mask_in_volume", "kind": "function", "doc": "Segment an object mask in in volumetric data.
\n\nArguments:
\n\n- \n
- segmentation: The initial segmentation for the object. \n
- predictor: The segment anything predictor. \n
- image_embeddings: The precomputed image embeddings for the volume. \n
- segmented_slices: List of slices for which this object has already been segmented. \n
- stop_lower: Whether to stop at the lowest segmented slice. \n
- stop_upper: Wheter to stop at the topmost segmented slice. \n
- iou_threshold: The IOU threshold for continuing segmentation across 3d. \n
- projection: The projection method to use. One of 'box', 'mask', 'points', 'points_and_mask' or 'single point'.\nPass a dictionary to choose the excact combination of projection modes. \n
- update_progress: Callback to update an external progress bar. \n
- box_extension: Extension factor for increasing the box size after projection. \n
- verbose: Whether to print details about the segmentation steps. \n
Returns:
\n\n\n\n", "signature": "(\tsegmentation: numpy.ndarray,\tpredictor: segment_anything.predictor.SamPredictor,\timage_embeddings: Dict[str, Any],\tsegmented_slices: numpy.ndarray,\tstop_lower: bool,\tstop_upper: bool,\tiou_threshold: float,\tprojection: Union[str, dict],\tupdate_progress: Optional[<built-in function callable>] = None,\tbox_extension: float = 0.0,\tverbose: bool = False) -> Tuple[numpy.ndarray, Tuple[int, int]]:", "funcdef": "def"}, {"fullname": "micro_sam.multi_dimensional_segmentation.merge_instance_segmentation_3d", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "merge_instance_segmentation_3d", "kind": "function", "doc": "Array with the volumetric segmentation.\n Tuple with the first and last segmented slice.
\n
Merge stacked 2d instance segmentations into a consistent 3d segmentation.
\n\nSolves a multicut problem based on the overlap of objects to merge across z.
\n\nArguments:
\n\n- \n
- slice_segmentation: The stacked segmentation across the slices.\nWe assume that the segmentation is labeled consecutive across z. \n
- beta: The bias term for the multicut. Higher values lead to a larger\ndegree of over-segmentation and vice versa. \n
- with_background: Whether this is a segmentation problem with background.\nIn that case all edges connecting to the background are set to be repulsive. \n
- gap_closing: If given, gaps in the segmentation are closed with a binary closing\noperation. The value is used to determine the number of iterations for the closing. \n
- min_z_extent: Require a minimal extent in z for the segmented objects.\nThis can help to prevent segmentation artifacts. \n
- verbose: Verbosity flag. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Returns:
\n\n\n\n", "signature": "(\tslice_segmentation: numpy.ndarray,\tbeta: float = 0.5,\twith_background: bool = True,\tgap_closing: Optional[int] = None,\tmin_z_extent: Optional[int] = None,\tverbose: bool = True,\tpbar_init: Optional[<built-in function callable>] = None,\tpbar_update: Optional[<built-in function callable>] = None) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.multi_dimensional_segmentation.automatic_3d_segmentation", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "automatic_3d_segmentation", "kind": "function", "doc": "The merged segmentation.
\n
Segment volume in 3d.
\n\nFirst segments slices individually in 2d and then merges them across 3d\nbased on overlap of objects between slices.
\n\nArguments:
\n\n- \n
- volume: The input volume. \n
- predictor: The SAM model. \n
- segmentor: The instance segmentation class. \n
- embedding_path: The path to save pre-computed embeddings. \n
- with_background: Whether the segmentation has background. \n
- gap_closing: If given, gaps in the segmentation are closed with a binary closing\noperation. The value is used to determine the number of iterations for the closing. \n
- min_z_extent: Require a minimal extent in z for the segmented objects.\nThis can help to prevent segmentation artifacts. \n
- verbose: Verbosity flag. \n
- kwargs: Keyword arguments for the 'generate' method of the 'segmentor'. \n
Returns:
\n\n\n\n", "signature": "(\tvolume: numpy.ndarray,\tpredictor: segment_anything.predictor.SamPredictor,\tsegmentor: micro_sam.instance_segmentation.AMGBase,\tembedding_path: Union[os.PathLike, str, NoneType] = None,\twith_background: bool = True,\tgap_closing: Optional[int] = None,\tmin_z_extent: Optional[int] = None,\tverbose: bool = True,\t**kwargs) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.precompute_state", "modulename": "micro_sam.precompute_state", "kind": "module", "doc": "The segmentation.
\n
Precompute image embeddings and automatic mask generator state for image data.
\n"}, {"fullname": "micro_sam.precompute_state.cache_amg_state", "modulename": "micro_sam.precompute_state", "qualname": "cache_amg_state", "kind": "function", "doc": "Compute and cache or load the state for the automatic mask generator.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- raw: The image data. \n
- image_embeddings: The image embeddings. \n
- save_path: The embedding save path. The AMG state will be stored in 'save_path/amg_state.pickle'. \n
- verbose: Whether to run the computation verbose. \n
- i: The index for which to cache the state. \n
- kwargs: The keyword arguments for the amg class. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\traw: numpy.ndarray,\timage_embeddings: Dict[str, Any],\tsave_path: Union[str, os.PathLike],\tverbose: bool = True,\ti: Optional[int] = None,\t**kwargs) -> micro_sam.instance_segmentation.AMGBase:", "funcdef": "def"}, {"fullname": "micro_sam.precompute_state.cache_is_state", "modulename": "micro_sam.precompute_state", "qualname": "cache_is_state", "kind": "function", "doc": "The automatic mask generator class with the cached state.
\n
Compute and cache or load the state for the automatic mask generator.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- decoder: The instance segmentation decoder. \n
- raw: The image data. \n
- image_embeddings: The image embeddings. \n
- save_path: The embedding save path. The AMG state will be stored in 'save_path/amg_state.pickle'. \n
- verbose: Whether to run the computation verbose. \n
- i: The index for which to cache the state. \n
- skip_load: Skip loading the state if it is precomputed. \n
- kwargs: The keyword arguments for the amg class. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tdecoder: torch.nn.modules.module.Module,\traw: numpy.ndarray,\timage_embeddings: Dict[str, Any],\tsave_path: Union[str, os.PathLike],\tverbose: bool = True,\ti: Optional[int] = None,\tskip_load: bool = False,\t**kwargs) -> Optional[micro_sam.instance_segmentation.AMGBase]:", "funcdef": "def"}, {"fullname": "micro_sam.precompute_state.precompute_state", "modulename": "micro_sam.precompute_state", "qualname": "precompute_state", "kind": "function", "doc": "The instance segmentation class with the cached state.
\n
Precompute the image embeddings and other optional state for the input image(s).
\n\nArguments:
\n\n- \n
- input_path: The input image file(s). Can either be a single image file (e.g. tif or png),\na container file (e.g. hdf5 or zarr) or a folder with images files.\nIn case of a container file the argument
keymust be given. In case of a folder\nit can be given to provide a glob pattern to subselect files from the folder. \n - output_path: The output path were the embeddings and other state will be saved. \n
- pattern: Glob pattern to select files in a folder. The embeddings will be computed\nfor each of these files. To select all files in a folder pass \"*\". \n
- model_type: The SegmentAnything model to use. Will use the standard vit_h model by default. \n
- checkpoint_path: Path to a checkpoint for a custom model. \n
- key: The key to the input file. This is needed for contaner files (e.g. hdf5 or zarr)\nor to load several images as 3d volume. Provide a glob pattern, e.g. \"*.tif\", for this case. \n
- ndim: The dimensionality of the data. \n
- tile_shape: Shape of tiles for tiled prediction. By default prediction is run without tiling. \n
- halo: Overlap of the tiles for tiled prediction. \n
- precompute_amg_state: Whether to precompute the state for automatic instance segmentation\nin addition to the image embeddings. \n
Functions for prompt-based segmentation with Segment Anything.
\n"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_points", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_points", "kind": "function", "doc": "Segmentation from point prompts.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- points: The point prompts given in the image coordinate system. \n
- labels: The labels (positive or negative) associated with the points. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
- use_best_multimask: Whether to use multimask output and then choose the best mask.\nBy default this is used for a single positive point and not otherwise. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tpoints: numpy.ndarray,\tlabels: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tmultimask_output: bool = False,\treturn_all: bool = False,\tuse_best_multimask: Optional[bool] = None):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_mask", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_mask", "kind": "function", "doc": "The binary segmentation mask.
\n
Segmentation from a mask prompt.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- mask: The mask used to derive prompts. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- use_box: Whether to derive the bounding box prompt from the mask. \n
- use_mask: Whether to use the mask itself as prompt. \n
- use_points: Whether to derive point prompts from the mask. \n
- original_size: Full image shape. Use this if the mask that is being passed\ndownsampled compared to the original image. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
- box_extension: Relative factor used to enlarge the bounding box prompt. \n
- box: Precomputed bounding box. \n
- points: Precomputed point prompts. \n
- labels: Positive/negative labels corresponding to the point prompts. \n
- use_single_point: Whether to derive just a single point from the mask.\nIn case use_points is true. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tmask: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tuse_box: bool = True,\tuse_mask: bool = True,\tuse_points: bool = False,\toriginal_size: Optional[Tuple[int, ...]] = None,\tmultimask_output: bool = False,\treturn_all: bool = False,\treturn_logits: bool = False,\tbox_extension: float = 0.0,\tbox: Optional[numpy.ndarray] = None,\tpoints: Optional[numpy.ndarray] = None,\tlabels: Optional[numpy.ndarray] = None,\tuse_single_point: bool = False):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_box", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_box", "kind": "function", "doc": "The binary segmentation mask.
\n
Segmentation from a box prompt.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- box: The box prompt. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
- box_extension: Relative factor used to enlarge the bounding box prompt. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tbox: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tmultimask_output: bool = False,\treturn_all: bool = False,\tbox_extension: float = 0.0):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_box_and_points", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_box_and_points", "kind": "function", "doc": "The binary segmentation mask.
\n
Segmentation from a box prompt and point prompts.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- box: The box prompt. \n
- points: The point prompts, given in the image coordinates system. \n
- labels: The point labels, either positive or negative. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tbox: numpy.ndarray,\tpoints: numpy.ndarray,\tlabels: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tmultimask_output: bool = False,\treturn_all: bool = False):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_generators", "modulename": "micro_sam.prompt_generators", "kind": "module", "doc": "The binary segmentation mask.
\n
Classes for generating prompts from ground-truth segmentation masks.\nFor training or evaluation of prompt-based segmentation.
\n"}, {"fullname": "micro_sam.prompt_generators.PromptGeneratorBase", "modulename": "micro_sam.prompt_generators", "qualname": "PromptGeneratorBase", "kind": "class", "doc": "PromptGeneratorBase is an interface to implement specific prompt generators.
\n"}, {"fullname": "micro_sam.prompt_generators.PointAndBoxPromptGenerator", "modulename": "micro_sam.prompt_generators", "qualname": "PointAndBoxPromptGenerator", "kind": "class", "doc": "Generate point and/or box prompts from an instance segmentation.
\n\nYou can use this class to derive prompts from an instance segmentation, either for\nevaluation purposes or for training Segment Anything on custom data.\nIn order to use this generator you need to precompute the bounding boxes and center\ncoordiantes of the instance segmentation, using e.g. util.get_centers_and_bounding_boxes.
Here's an example for how to use this class:
\n\n# Initialize generator for 1 positive and 4 negative point prompts.\nprompt_generator = PointAndBoxPromptGenerator(1, 4, dilation_strength=8)\n\n# Precompute the bounding boxes for the given segmentation\nbounding_boxes, _ = util.get_centers_and_bounding_boxes(segmentation)\n\n# generate point prompts for the objects with ids 1, 2 and 3\nseg_ids = (1, 2, 3)\nobject_mask = np.stack([segmentation == seg_id for seg_id in seg_ids])[:, None]\nthis_bounding_boxes = [bounding_boxes[seg_id] for seg_id in seg_ids]\npoint_coords, point_labels, _, _ = prompt_generator(object_mask, this_bounding_boxes)\nArguments:
\n\n- \n
- n_positive_points: The number of positive point prompts to generate per mask. \n
- n_negative_points: The number of negative point prompts to generate per mask. \n
- dilation_strength: The factor by which the mask is dilated before generating prompts. \n
- get_point_prompts: Whether to generate point prompts. \n
- get_box_prompts: Whether to generate box prompts. \n
Generate point prompts from an instance segmentation iteratively.
\n", "bases": "PromptGeneratorBase"}, {"fullname": "micro_sam.sam_annotator", "modulename": "micro_sam.sam_annotator", "kind": "module", "doc": "The interactive annotation tools.
\n"}, {"fullname": "micro_sam.sam_annotator.annotator_2d", "modulename": "micro_sam.sam_annotator.annotator_2d", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.annotator_2d.Annotator2d", "modulename": "micro_sam.sam_annotator.annotator_2d", "qualname": "Annotator2d", "kind": "class", "doc": "Base class for micro_sam annotation plugins.
\n\nImplements the logic for the 2d, 3d and tracking annotator.\nThe annotators differ in their data dimensionality and the widgets.
\n", "bases": "micro_sam.sam_annotator._annotator._AnnotatorBase"}, {"fullname": "micro_sam.sam_annotator.annotator_2d.Annotator2d.__init__", "modulename": "micro_sam.sam_annotator.annotator_2d", "qualname": "Annotator2d.__init__", "kind": "function", "doc": "Create the annotator GUI.
\n\nArguments:
\n\n- \n
- viewer: The napari viewer. \n
- ndim: The number of spatial dimension of the image data (2 or 3). \n
Start the 2d annotation tool for a given image.
\n\nArguments:
\n\n- \n
- image: The image data. \n
- embedding_path: Filepath where to save the embeddings. \n
- segmentation_result: An initial segmentation to load.\nThis can be used to correct segmentations with Segment Anything or to save and load progress.\nThe segmentation will be loaded as the 'committed_objects' layer. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- precompute_amg_state: Whether to precompute the state for automatic mask generation.\nThis will take more time when precomputing embeddings, but will then make\nautomatic mask generation much faster. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- device: The computational device to use for the SAM model. \n
- prefer_decoder: Whether to use decoder based instance segmentation if\nthe model used has an additional decoder for instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\timage: numpy.ndarray,\tembedding_path: Optional[str] = None,\tsegmentation_result: Optional[numpy.ndarray] = None,\tmodel_type: str = 'vit_l',\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\treturn_viewer: bool = False,\tviewer: Optional[napari.viewer.Viewer] = None,\tprecompute_amg_state: bool = False,\tcheckpoint_path: Optional[str] = None,\tdevice: Union[str, torch.device, NoneType] = None,\tprefer_decoder: bool = True) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.annotator_3d", "modulename": "micro_sam.sam_annotator.annotator_3d", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.annotator_3d.Annotator3d", "modulename": "micro_sam.sam_annotator.annotator_3d", "qualname": "Annotator3d", "kind": "class", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Base class for micro_sam annotation plugins.
\n\nImplements the logic for the 2d, 3d and tracking annotator.\nThe annotators differ in their data dimensionality and the widgets.
\n", "bases": "micro_sam.sam_annotator._annotator._AnnotatorBase"}, {"fullname": "micro_sam.sam_annotator.annotator_3d.Annotator3d.__init__", "modulename": "micro_sam.sam_annotator.annotator_3d", "qualname": "Annotator3d.__init__", "kind": "function", "doc": "Create the annotator GUI.
\n\nArguments:
\n\n- \n
- viewer: The napari viewer. \n
- ndim: The number of spatial dimension of the image data (2 or 3). \n
Start the 3d annotation tool for a given image volume.
\n\nArguments:
\n\n- \n
- image: The volumetric image data. \n
- embedding_path: Filepath for saving the precomputed embeddings. \n
- segmentation_result: An initial segmentation to load.\nThis can be used to correct segmentations with Segment Anything or to save and load progress.\nThe segmentation will be loaded as the 'committed_objects' layer. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- precompute_amg_state: Whether to precompute the state for automatic mask generation.\nThis will take more time when precomputing embeddings, but will then make\nautomatic mask generation much faster. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- device: The computational device to use for the SAM model. \n
- prefer_decoder: Whether to use decoder based instance segmentation if\nthe model used has an additional decoder for instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\timage: numpy.ndarray,\tembedding_path: Optional[str] = None,\tsegmentation_result: Optional[numpy.ndarray] = None,\tmodel_type: str = 'vit_l',\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\treturn_viewer: bool = False,\tviewer: Optional[napari.viewer.Viewer] = None,\tprecompute_amg_state: bool = False,\tcheckpoint_path: Optional[str] = None,\tdevice: Union[str, torch.device, NoneType] = None,\tprefer_decoder: bool = True) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.annotator_tracking", "modulename": "micro_sam.sam_annotator.annotator_tracking", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.annotator_tracking.AnnotatorTracking", "modulename": "micro_sam.sam_annotator.annotator_tracking", "qualname": "AnnotatorTracking", "kind": "class", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Base class for micro_sam annotation plugins.
\n\nImplements the logic for the 2d, 3d and tracking annotator.\nThe annotators differ in their data dimensionality and the widgets.
\n", "bases": "micro_sam.sam_annotator._annotator._AnnotatorBase"}, {"fullname": "micro_sam.sam_annotator.annotator_tracking.AnnotatorTracking.__init__", "modulename": "micro_sam.sam_annotator.annotator_tracking", "qualname": "AnnotatorTracking.__init__", "kind": "function", "doc": "Create the annotator GUI.
\n\nArguments:
\n\n- \n
- viewer: The napari viewer. \n
- ndim: The number of spatial dimension of the image data (2 or 3). \n
Start the tracking annotation tool fora given timeseries.
\n\nArguments:
\n\n- \n
- raw: The image data. \n
- embedding_path: Filepath for saving the precomputed embeddings. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- device: The computational device to use for the SAM model. \n
Returns:
\n\n\n\n", "signature": "(\timage: numpy.ndarray,\tembedding_path: Optional[str] = None,\tmodel_type: str = 'vit_l',\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\treturn_viewer: bool = False,\tviewer: Optional[napari.viewer.Viewer] = None,\tcheckpoint_path: Optional[str] = None,\tdevice: Union[str, torch.device, NoneType] = None) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.image_series_annotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "image_series_annotator", "kind": "function", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Run the annotation tool for a series of images (supported for both 2d and 3d images).
\n\nArguments:
\n\n- \n
- images: List of the file paths or list of (set of) slices for the images to be annotated. \n
- output_folder: The folder where the segmentation results are saved. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- embedding_path: Filepath where to save the embeddings. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- precompute_amg_state: Whether to precompute the state for automatic mask generation.\nThis will take more time when precomputing embeddings, but will then make\nautomatic mask generation much faster. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- is_volumetric: Whether to use the 3d annotator. \n
- prefer_decoder: Whether to use decoder based instance segmentation if\nthe model used has an additional decoder for instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\timages: Union[List[Union[os.PathLike, str]], List[numpy.ndarray]],\toutput_folder: str,\tmodel_type: str = 'vit_l',\tembedding_path: Optional[str] = None,\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\tviewer: Optional[napari.viewer.Viewer] = None,\treturn_viewer: bool = False,\tprecompute_amg_state: bool = False,\tcheckpoint_path: Optional[str] = None,\tis_volumetric: bool = False,\tdevice: Union[str, torch.device, NoneType] = None,\tprefer_decoder: bool = True) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.image_folder_annotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "image_folder_annotator", "kind": "function", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Run the 2d annotation tool for a series of images in a folder.
\n\nArguments:
\n\n- \n
- input_folder: The folder with the images to be annotated. \n
- output_folder: The folder where the segmentation results are saved. \n
- pattern: The glob patter for loading files from
input_folder.\nBy default all files will be loaded. \n - viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- kwargs: The keyword arguments for
micro_sam.sam_annotator.image_series_annotator. \n
Returns:
\n\n\n\n", "signature": "(\tinput_folder: str,\toutput_folder: str,\tpattern: str = '*',\tviewer: Optional[napari.viewer.Viewer] = None,\treturn_viewer: bool = False,\t**kwargs) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.ImageSeriesAnnotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "ImageSeriesAnnotator", "kind": "class", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
QWidget(parent: typing.Optional[QWidget] = None, flags: Union[Qt.WindowFlags, Qt.WindowType] = Qt.WindowFlags())
\n", "bases": "micro_sam.sam_annotator._widgets._WidgetBase"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.ImageSeriesAnnotator.__init__", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "ImageSeriesAnnotator.__init__", "kind": "function", "doc": "\n", "signature": "(viewer: napari.viewer.Viewer, parent=None)"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.ImageSeriesAnnotator.run_button", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "ImageSeriesAnnotator.run_button", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.training_ui", "modulename": "micro_sam.sam_annotator.training_ui", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.training_ui.TrainingWidget", "modulename": "micro_sam.sam_annotator.training_ui", "qualname": "TrainingWidget", "kind": "class", "doc": "QWidget(parent: typing.Optional[QWidget] = None, flags: Union[Qt.WindowFlags, Qt.WindowType] = Qt.WindowFlags())
\n", "bases": "micro_sam.sam_annotator._widgets._WidgetBase"}, {"fullname": "micro_sam.sam_annotator.training_ui.TrainingWidget.__init__", "modulename": "micro_sam.sam_annotator.training_ui", "qualname": "TrainingWidget.__init__", "kind": "function", "doc": "\n", "signature": "(parent=None)"}, {"fullname": "micro_sam.sam_annotator.training_ui.TrainingWidget.run_button", "modulename": "micro_sam.sam_annotator.training_ui", "qualname": "TrainingWidget.run_button", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.util", "modulename": "micro_sam.sam_annotator.util", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.util.point_layer_to_prompts", "modulename": "micro_sam.sam_annotator.util", "qualname": "point_layer_to_prompts", "kind": "function", "doc": "Extract point prompts for SAM from a napari point layer.
\n\nArguments:
\n\n- \n
- layer: The point layer from which to extract the prompts. \n
- i: Index for the data (required for 3d or timeseries data). \n
- track_id: Id of the current track (required for tracking data). \n
- with_stop_annotation: Whether a single negative point will be interpreted\nas stop annotation or just returned as normal prompt. \n
Returns:
\n\n\n\n", "signature": "(\tlayer: napari.layers.points.points.Points,\ti=None,\ttrack_id=None,\twith_stop_annotation=True) -> Optional[Tuple[numpy.ndarray, numpy.ndarray]]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.util.shape_layer_to_prompts", "modulename": "micro_sam.sam_annotator.util", "qualname": "shape_layer_to_prompts", "kind": "function", "doc": "The point coordinates for the prompts.\n The labels (positive or negative / 1 or 0) for the prompts.
\n
Extract prompts for SAM from a napari shape layer.
\n\nExtracts the bounding box for 'rectangle' shapes and the bounding box and corresponding mask\nfor 'ellipse' and 'polygon' shapes.
\n\nArguments:
\n\n- \n
- prompt_layer: The napari shape layer. \n
- shape: The image shape. \n
- i: Index for the data (required for 3d or timeseries data). \n
- track_id: Id of the current track (required for tracking data). \n
Returns:
\n\n\n\n", "signature": "(\tlayer: napari.layers.shapes.shapes.Shapes,\tshape: Tuple[int, int],\ti=None,\ttrack_id=None) -> Tuple[List[numpy.ndarray], List[Optional[numpy.ndarray]]]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.util.prompt_layer_to_state", "modulename": "micro_sam.sam_annotator.util", "qualname": "prompt_layer_to_state", "kind": "function", "doc": "The box prompts.\n The mask prompts.
\n
Get the state of the track from a point layer for a given timeframe.
\n\nOnly relevant for annotator_tracking.
\n\nArguments:
\n\n- \n
- prompt_layer: The napari layer. \n
- i: Timeframe of the data. \n
Returns:
\n\n\n\n", "signature": "(prompt_layer: napari.layers.points.points.Points, i: int) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.util.prompt_layers_to_state", "modulename": "micro_sam.sam_annotator.util", "qualname": "prompt_layers_to_state", "kind": "function", "doc": "The state of this frame (either \"division\" or \"track\").
\n
Get the state of the track from a point layer and shape layer for a given timeframe.
\n\nOnly relevant for annotator_tracking.
\n\nArguments:
\n\n- \n
- point_layer: The napari point layer. \n
- box_layer: The napari box layer. \n
- i: Timeframe of the data. \n
Returns:
\n\n\n\n", "signature": "(\tpoint_layer: napari.layers.points.points.Points,\tbox_layer: napari.layers.shapes.shapes.Shapes,\ti: int) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data", "modulename": "micro_sam.sample_data", "kind": "module", "doc": "The state of this frame (either \"division\" or \"track\").
\n
Sample microscopy data.
\n\nYou can change the download location for sample data and model weights\nby setting the environment variable: MICROSAM_CACHEDIR
\n\nBy default sample data is downloaded to a folder named 'micro_sam/sample_data'\ninside your default cache directory, eg:\n * Mac: ~/Library/Caches/
Download the sample images for the image series annotator.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_image_series", "modulename": "micro_sam.sample_data", "qualname": "sample_data_image_series", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides image series example image to napari.
\n\nOpens as three separate image layers in napari (one per image in series).\nThe third image in the series has a different size and modality.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_wholeslide_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_wholeslide_example_data", "kind": "function", "doc": "Download the sample data for the 2d annotator.
\n\nThis downloads part of a whole-slide image from the NeurIPS Cell Segmentation Challenge.\nSee https://neurips22-cellseg.grand-challenge.org/ for details on the data.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_wholeslide", "modulename": "micro_sam.sample_data", "qualname": "sample_data_wholeslide", "kind": "function", "doc": "The path of the downloaded image.
\n
Provides wholeslide 2d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_livecell_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_livecell_example_data", "kind": "function", "doc": "Download the sample data for the 2d annotator.
\n\nThis downloads a single image from the LiveCELL dataset.\nSee https://doi.org/10.1038/s41592-021-01249-6 for details on the data.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_livecell", "modulename": "micro_sam.sample_data", "qualname": "sample_data_livecell", "kind": "function", "doc": "The path of the downloaded image.
\n
Provides livecell 2d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_hela_2d_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_hela_2d_example_data", "kind": "function", "doc": "Download the sample data for the 2d annotator.
\n\nThis downloads a single image from the HeLa CTC dataset.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> Union[str, os.PathLike]:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_hela_2d", "modulename": "micro_sam.sample_data", "qualname": "sample_data_hela_2d", "kind": "function", "doc": "The path of the downloaded image.
\n
Provides HeLa 2d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_3d_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_3d_example_data", "kind": "function", "doc": "Download the sample data for the 3d annotator.
\n\nThis downloads the Lucchi++ datasets from https://casser.io/connectomics/.\nIt is a dataset for mitochondria segmentation in EM.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_3d", "modulename": "micro_sam.sample_data", "qualname": "sample_data_3d", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides Lucchi++ 3d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_tracking_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_tracking_example_data", "kind": "function", "doc": "Download the sample data for the tracking annotator.
\n\nThis data is the cell tracking challenge dataset DIC-C2DH-HeLa.\nCell tracking challenge webpage: http://data.celltrackingchallenge.net\nHeLa cells on a flat glass\nDr. G. van Cappellen. Erasmus Medical Center, Rotterdam, The Netherlands\nTraining dataset: http://data.celltrackingchallenge.net/training-datasets/DIC-C2DH-HeLa.zip (37 MB)\nChallenge dataset: http://data.celltrackingchallenge.net/challenge-datasets/DIC-C2DH-HeLa.zip (41 MB)
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_tracking", "modulename": "micro_sam.sample_data", "qualname": "sample_data_tracking", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides tracking example dataset to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_tracking_segmentation_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_tracking_segmentation_data", "kind": "function", "doc": "Download groundtruth segmentation for the tracking example data.
\n\nThis downloads the groundtruth segmentation for the image data from fetch_tracking_example_data.
Arguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_segmentation", "modulename": "micro_sam.sample_data", "qualname": "sample_data_segmentation", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides segmentation example dataset to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.synthetic_data", "modulename": "micro_sam.sample_data", "qualname": "synthetic_data", "kind": "function", "doc": "Create synthetic image data and segmentation for training.
\n", "signature": "(shape, seed=None):", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_nucleus_3d_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_nucleus_3d_example_data", "kind": "function", "doc": "Download the sample data for 3d segmentation of nuclei.
\n\nThis data contains a small crop from a volume from the publication\n\"Efficient automatic 3D segmentation of cell nuclei for high-content screening\"\nhttps://doi.org/10.1186/s12859-022-04737-4
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.training", "modulename": "micro_sam.training", "kind": "module", "doc": "The path of the downloaded image.
\n
Functionality for training Segment Anything.
\n"}, {"fullname": "micro_sam.training.joint_sam_trainer", "modulename": "micro_sam.training.joint_sam_trainer", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.training.joint_sam_trainer.JointSamTrainer", "modulename": "micro_sam.training.joint_sam_trainer", "qualname": "JointSamTrainer", "kind": "class", "doc": "Trainer class for training the Segment Anything model.
\n\nThis class is derived from torch_em.trainer.DefaultTrainer.\nCheck out https://github.com/constantinpape/torch-em/blob/main/torch_em/trainer/default_trainer.py\nfor details on its usage and implementation.
Arguments:
\n\n- \n
- convert_inputs: The class that converts outputs of the dataloader to the expected input format of SAM.\nThe class
micro_sam.training.util.ConvertToSamInputscan be used here. \n - n_sub_iteration: The number of iteration steps for which the masks predicted for one object are updated.\nIn each sub-iteration new point prompts are sampled where the model was wrong. \n
- n_objects_per_batch: If not given, we compute the loss for all objects in a sample.\nOtherwise the loss computation is limited to n_objects_per_batch, and the objects are randomly sampled. \n
- mse_loss: The regression loss to compare the IoU predicted by the model with the true IoU. \n
- prompt_generator: The iterative prompt generator which takes care of the iterative prompting logic for training \n
- mask_prob: The probability of using the mask inputs in the iterative prompting (per
n_sub_iteration) \n - **kwargs: The keyword arguments of the DefaultTrainer super class. \n
Trainer class for training the Segment Anything model.
\n\nThis class is derived from torch_em.trainer.DefaultTrainer.\nCheck out https://github.com/constantinpape/torch-em/blob/main/torch_em/trainer/default_trainer.py\nfor details on its usage and implementation.
Arguments:
\n\n- \n
- convert_inputs: The class that converts outputs of the dataloader to the expected input format of SAM.\nThe class
micro_sam.training.util.ConvertToSamInputscan be used here. \n - n_sub_iteration: The number of iteration steps for which the masks predicted for one object are updated.\nIn each sub-iteration new point prompts are sampled where the model was wrong. \n
- n_objects_per_batch: If not given, we compute the loss for all objects in a sample.\nOtherwise the loss computation is limited to n_objects_per_batch, and the objects are randomly sampled. \n
- mse_loss: The regression loss to compare the IoU predicted by the model with the true IoU. \n
- prompt_generator: The iterative prompt generator which takes care of the iterative prompting logic for training \n
- mask_prob: The probability of using the mask inputs in the iterative prompting (per
n_sub_iteration) \n - **kwargs: The keyword arguments of the DefaultTrainer super class. \n
Wrapper to make the SegmentAnything model trainable.
\n\nArguments:
\n\n- \n
- sam: The SegmentAnything Model. \n
- device: The device for training. \n
Initializes internal Module state, shared by both nn.Module and ScriptModule.
\n", "signature": "(\tsam: segment_anything.modeling.sam.Sam,\tdevice: Union[str, torch.device])"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.sam", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.sam", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.device", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.device", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.transform", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.transform", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.preprocess", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.preprocess", "kind": "function", "doc": "Resize, normalize pixel values and pad to a square input.
\n\nArguments:
\n\n- \n
- x: The input tensor. \n
Returns:
\n\n\n\n", "signature": "(self, x: torch.Tensor) -> Tuple[torch.Tensor, Tuple[int, int]]:", "funcdef": "def"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.image_embeddings_oft", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.image_embeddings_oft", "kind": "function", "doc": "\n", "signature": "(self, batched_inputs):", "funcdef": "def"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.forward", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.forward", "kind": "function", "doc": "The resized, normalized and padded tensor.\n The shape of the image after resizing.
\n
Forward pass.
\n\nArguments:
\n\n- \n
- batched_inputs: The batched input images and prompts. \n
- image_embeddings: The precompute image embeddings. If not passed then they will be computed. \n
- multimask_output: Whether to predict mutiple or just a single mask. \n
Returns:
\n\n\n\n", "signature": "(\tself,\tbatched_inputs: List[Dict[str, Any]],\timage_embeddings: torch.Tensor,\tmultimask_output: bool = False) -> List[Dict[str, Any]]:", "funcdef": "def"}, {"fullname": "micro_sam.training.training", "modulename": "micro_sam.training.training", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.training.training.FilePath", "modulename": "micro_sam.training.training", "qualname": "FilePath", "kind": "variable", "doc": "\n", "default_value": "typing.Union[str, os.PathLike]"}, {"fullname": "micro_sam.training.training.train_sam", "modulename": "micro_sam.training.training", "qualname": "train_sam", "kind": "function", "doc": "The predicted segmentation masks and iou values.
\n
Run training for a SAM model.
\n\nArguments:
\n\n- \n
- name: The name of the model to be trained.\nThe checkpoint and logs wil have this name. \n
- model_type: The type of the SAM model. \n
- train_loader: The dataloader for training. \n
- val_loader: The dataloader for validation. \n
- n_epochs: The number of epochs to train for. \n
- early_stopping: Enable early stopping after this number of epochs\nwithout improvement. \n
- n_objects_per_batch: The number of objects per batch used to compute\nthe loss for interative segmentation. If None all objects will be used,\nif given objects will be randomly sub-sampled. \n
- checkpoint_path: Path to checkpoint for initializing the SAM model. \n
- with_segmentation_decoder: Whether to train additional UNETR decoder\nfor automatic instance segmentation. \n
- freeze: Specify parts of the model that should be frozen, namely:\nimage_encoder, prompt_encoder and mask_decoder\nBy default nothing is frozen and the full model is updated. \n
- device: The device to use for training. \n
- lr: The learning rate. \n
- n_sub_iteration: The number of iterative prompts per training iteration. \n
- save_root: Optional root directory for saving the checkpoints and logs.\nIf not given the current working directory is used. \n
- mask_prob: The probability for using a mask as input in a given training sub-iteration. \n
- n_iterations: The number of iterations to use for training. This will over-ride n_epochs if given. \n
- scheduler_class: The learning rate scheduler to update the learning rate.\nBy default, ReduceLROnPlateau is used. \n
- scheduler_kwargs: The learning rate scheduler parameters.\nIf passed None, the chosen default parameters are used in ReduceLROnPlateau. \n
- save_every_kth_epoch: Save checkpoints after every kth epoch separately. \n
- pbar_signals: Controls for napari progress bar. \n
Create a PyTorch Dataset for training a SAM model.
\n\nArguments:
\n\n- \n
- raw_paths: The path(s) to the image data used for training.\nCan either be multiple 2D images or volumetric data. \n
- raw_key: The key for accessing the image data. Internal filepath for hdf5-like input\nor a glob pattern for selecting multiple files. \n
- label_paths: The path(s) to the label data used for training.\nCan either be multiple 2D images or volumetric data. \n
- label_key: The key for accessing the label data. Internal filepath for hdf5-like input\nor a glob pattern for selecting multiple files. \n
- patch_shape: The shape for training patches. \n
- with_segmentation_decoder: Whether to train with additional segmentation decoder. \n
- with_channels: Whether the image data has RGB channels. \n
- sampler: A sampler to reject batches according to a given criterion. \n
- n_samples: The number of samples for this dataset. \n
- is_train: Whether this dataset is used for training or validation. \n
Returns:
\n\n\n\n", "signature": "(\traw_paths: Union[List[Union[os.PathLike, str]], str, os.PathLike],\traw_key: Optional[str],\tlabel_paths: Union[List[Union[os.PathLike, str]], str, os.PathLike],\tlabel_key: Optional[str],\tpatch_shape: Tuple[int],\twith_segmentation_decoder: bool,\twith_channels: bool = False,\tsampler=None,\tn_samples: Optional[int] = None,\tis_train: bool = True,\t**kwargs) -> torch.utils.data.dataset.Dataset:", "funcdef": "def"}, {"fullname": "micro_sam.training.training.default_sam_loader", "modulename": "micro_sam.training.training", "qualname": "default_sam_loader", "kind": "function", "doc": "\n", "signature": "(**kwargs) -> torch.utils.data.dataloader.DataLoader:", "funcdef": "def"}, {"fullname": "micro_sam.training.training.CONFIGURATIONS", "modulename": "micro_sam.training.training", "qualname": "CONFIGURATIONS", "kind": "variable", "doc": "The dataset.
\n
Best training configurations for given hardware resources.
\n", "default_value": "{'Minimal': {'model_type': 'vit_t', 'n_objects_per_batch': 4, 'n_sub_iteration': 4}, 'CPU': {'model_type': 'vit_b', 'n_objects_per_batch': 10}, 'gtx1080': {'model_type': 'vit_t', 'n_objects_per_batch': 5}, 'rtx5000': {'model_type': 'vit_b', 'n_objects_per_batch': 10}, 'V100': {'model_type': 'vit_b'}, 'A100': {'model_type': 'vit_h'}}"}, {"fullname": "micro_sam.training.training.train_sam_for_configuration", "modulename": "micro_sam.training.training", "qualname": "train_sam_for_configuration", "kind": "function", "doc": "Run training for a SAM model with the configuration for a given hardware resource.
\n\nSelects the best training settings for the given configuration.\nThe available configurations are listed in CONFIGURATIONS.
Arguments:
\n\n- \n
- name: The name of the model to be trained.\nThe checkpoint and logs wil have this name. \n
- configuration: The configuration (= name of hardware resource). \n
- train_loader: The dataloader for training. \n
- val_loader: The dataloader for validation. \n
- checkpoint_path: Path to checkpoint for initializing the SAM model. \n
- with_segmentation_decoder: Whether to train additional UNETR decoder\nfor automatic instance segmentation. \n
- kwargs: Additional keyword parameterts that will be passed to
train_sam. \n
Identity transformation.
\n\nThis is a helper function to skip data normalization when finetuning SAM.\nData normalization is performed within the model and should thus be skipped as\na preprocessing step in training.
\n", "signature": "(x):", "funcdef": "def"}, {"fullname": "micro_sam.training.util.require_8bit", "modulename": "micro_sam.training.util", "qualname": "require_8bit", "kind": "function", "doc": "Transformation to require 8bit input data range (0-255).
\n", "signature": "(x):", "funcdef": "def"}, {"fullname": "micro_sam.training.util.get_trainable_sam_model", "modulename": "micro_sam.training.util", "qualname": "get_trainable_sam_model", "kind": "function", "doc": "Get the trainable sam model.
\n\nArguments:
\n\n- \n
- model_type: The segment anything model that should be finetuned.\nThe weights of this model will be used for initialization, unless a\ncustom weight file is passed via
checkpoint_path. \n - device: The device to use for training. \n
- checkpoint_path: Path to a custom checkpoint from which to load the model weights. \n
- freeze: Specify parts of the model that should be frozen, namely: image_encoder, prompt_encoder and mask_decoder\nBy default nothing is frozen and the full model is updated. \n
- return_state: Whether to return the full checkpoint state. \n
Returns:
\n\n\n\n", "signature": "(\tmodel_type: str = 'vit_l',\tdevice: Union[str, torch.device, NoneType] = None,\tcheckpoint_path: Union[os.PathLike, str, NoneType] = None,\tfreeze: Optional[List[str]] = None,\treturn_state: bool = False) -> micro_sam.training.trainable_sam.TrainableSAM:", "funcdef": "def"}, {"fullname": "micro_sam.training.util.ConvertToSamInputs", "modulename": "micro_sam.training.util", "qualname": "ConvertToSamInputs", "kind": "class", "doc": "The trainable segment anything model.
\n
Convert outputs of data loader to the expected batched inputs of the SegmentAnything model.
\n\nArguments:
\n\n- \n
- transform: The transformation to resize the prompts. Should be the same transform used in the\nmodel to resize the inputs. If
Nonethe prompts will not be resized. \n - dilation_strength: The dilation factor.\nIt determines a \"safety\" border from which prompts are not sampled to avoid ambiguous prompts\ndue to imprecise groundtruth masks. \n
- box_distortion_factor: Factor for distorting the box annotations derived from the groundtruth masks. \n
Helper functions for downloading Segment Anything models and predicting image embeddings.
\n"}, {"fullname": "micro_sam.util.get_cache_directory", "modulename": "micro_sam.util", "qualname": "get_cache_directory", "kind": "function", "doc": "Get micro-sam cache directory location.
\n\nUsers can set the MICROSAM_CACHEDIR environment variable for a custom cache directory.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.util.microsam_cachedir", "modulename": "micro_sam.util", "qualname": "microsam_cachedir", "kind": "function", "doc": "Return the micro-sam cache directory.
\n\nReturns the top level cache directory for micro-sam models and sample data.
\n\nEvery time this function is called, we check for any user updates made to\nthe MICROSAM_CACHEDIR os environment variable since the last time.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.util.models", "modulename": "micro_sam.util", "qualname": "models", "kind": "function", "doc": "Return the segmentation models registry.
\n\nWe recreate the model registry every time this function is called,\nso any user changes to the default micro-sam cache directory location\nare respected.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.util.get_device", "modulename": "micro_sam.util", "qualname": "get_device", "kind": "function", "doc": "Get the torch device.
\n\nIf no device is passed the default device for your system is used.\nElse it will be checked if the device you have passed is supported.
\n\nArguments:
\n\n- \n
- device: The input device. \n
Returns:
\n\n\n\n", "signature": "(\tdevice: Union[str, torch.device, NoneType] = None) -> Union[str, torch.device]:", "funcdef": "def"}, {"fullname": "micro_sam.util.get_sam_model", "modulename": "micro_sam.util", "qualname": "get_sam_model", "kind": "function", "doc": "The device.
\n
Get the SegmentAnything Predictor.
\n\nThis function will download the required model or load it from the cached weight file.\nThis location of the cache can be changed by setting the environment variable: MICROSAM_CACHEDIR.\nThe name of the requested model can be set via model_type.\nSee https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models\nfor an overview of the available models
Alternatively this function can also load a model from weights stored in a local filepath.\nThe corresponding file path is given via checkpoint_path. In this case model_type\nmust be given as the matching encoder architecture, e.g. \"vit_b\" if the weights are for\na SAM model with vit_b encoder.
By default the models are downloaded to a folder named 'micro_sam/models'\ninside your default cache directory, eg:
\n\n- \n
- Mac: ~/Library/Caches/
- Unix: ~/.cache/
or the value of the XDG_CACHE_HOME environment variable, if defined. \n - Windows: C:\\Users<user>\\AppData\\Local<AppAuthor><AppName>\\Cache\nSee the pooch.os_cache() documentation for more details:\nhttps://www.fatiando.org/pooch/latest/api/generated/pooch.os_cache.html \n
Arguments:
\n\n- \n
- model_type: The SegmentAnything model to use. Will use the standard vit_h model by default.\nTo get a list of all available model names you can call
get_model_names. \n - device: The device for the model. If none is given will use GPU if available. \n
- checkpoint_path: The path to a file with weights that should be used instead of using the\nweights corresponding to
model_type. If given,model_typemust match the architecture\ncorresponding to the weight file. E.g. if you use weights for SAM with vit_b encoder\nthenmodel_typemust be given as \"vit_b\". \n - return_sam: Return the sam model object as well as the predictor. \n
- return_state: Return the unpickled checkpoint state. \n
Returns:
\n\n\n\n", "signature": "(\tmodel_type: str = 'vit_l',\tdevice: Union[str, torch.device, NoneType] = None,\tcheckpoint_path: Union[str, os.PathLike, NoneType] = None,\treturn_sam: bool = False,\treturn_state: bool = False) -> mobile_sam.predictor.SamPredictor:", "funcdef": "def"}, {"fullname": "micro_sam.util.export_custom_sam_model", "modulename": "micro_sam.util", "qualname": "export_custom_sam_model", "kind": "function", "doc": "The segment anything predictor.
\n
Export a finetuned segment anything model to the standard model format.
\n\nThe exported model can be used by the interactive annotation tools in micro_sam.annotator.
Arguments:
\n\n- \n
- checkpoint_path: The path to the corresponding checkpoint if not in the default model folder. \n
- model_type: The SegmentAnything model type corresponding to the checkpoint (vit_h, vit_b, vit_l or vit_t). \n
- save_path: Where to save the exported model. \n
Compute the image embeddings (output of the encoder) for the input.
\n\nIf 'save_path' is given the embeddings will be loaded/saved in a zarr container.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- input_: The input data. Can be 2 or 3 dimensional, corresponding to an image, volume or timeseries. \n
- save_path: Path to save the embeddings in a zarr container. \n
- lazy_loading: Whether to load all embeddings into memory or return an\nobject to load them on demand when required. This only has an effect if 'save_path' is given\nand if the input is 3 dimensional. \n
- ndim: The dimensionality of the data. If not given will be deduced from the input data. \n
- tile_shape: Shape of tiles for tiled prediction. By default prediction is run without tiling. \n
- halo: Overlap of the tiles for tiled prediction. \n
- verbose: Whether to be verbose in the computation. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: mobile_sam.predictor.SamPredictor,\tinput_: numpy.ndarray,\tsave_path: Union[str, os.PathLike, NoneType] = None,\tlazy_loading: bool = False,\tndim: Optional[int] = None,\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\tverbose: bool = True,\tpbar_init: Optional[<built-in function callable>] = None,\tpbar_update: Optional[<built-in function callable>] = None) -> Dict[str, Any]:", "funcdef": "def"}, {"fullname": "micro_sam.util.set_precomputed", "modulename": "micro_sam.util", "qualname": "set_precomputed", "kind": "function", "doc": "The image embeddings.
\n
Set the precomputed image embeddings for a predictor.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_embeddings: The precomputed image embeddings computed by
precompute_image_embeddings. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - tile_id: Index for the tile. This is required if the embeddings are tiled. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: mobile_sam.predictor.SamPredictor,\timage_embeddings: Dict[str, Any],\ti: Optional[int] = None,\ttile_id: Optional[int] = None) -> mobile_sam.predictor.SamPredictor:", "funcdef": "def"}, {"fullname": "micro_sam.util.compute_iou", "modulename": "micro_sam.util", "qualname": "compute_iou", "kind": "function", "doc": "The predictor with set features.
\n
Compute the intersection over union of two masks.
\n\nArguments:
\n\n- \n
- mask1: The first mask. \n
- mask2: The second mask. \n
Returns:
\n\n\n\n", "signature": "(mask1: numpy.ndarray, mask2: numpy.ndarray) -> float:", "funcdef": "def"}, {"fullname": "micro_sam.util.get_centers_and_bounding_boxes", "modulename": "micro_sam.util", "qualname": "get_centers_and_bounding_boxes", "kind": "function", "doc": "The intersection over union of the two masks.
\n
Returns the center coordinates of the foreground instances in the ground-truth.
\n\nArguments:
\n\n- \n
- segmentation: The segmentation. \n
- mode: Determines the functionality used for computing the centers. \n
- If 'v', the object's eccentricity centers computed by vigra are used. \n
- If 'p' the object's centroids computed by skimage are used. \n
Returns:
\n\n\n\n", "signature": "(\tsegmentation: numpy.ndarray,\tmode: str = 'v') -> Tuple[Dict[int, numpy.ndarray], Dict[int, tuple]]:", "funcdef": "def"}, {"fullname": "micro_sam.util.load_image_data", "modulename": "micro_sam.util", "qualname": "load_image_data", "kind": "function", "doc": "A dictionary that maps object ids to the corresponding centroid.\n A dictionary that maps object_ids to the corresponding bounding box.
\n
Helper function to load image data from file.
\n\nArguments:
\n\n- \n
- path: The filepath to the image data. \n
- key: The internal filepath for complex data formats like hdf5. \n
- lazy_loading: Whether to lazyly load data. Only supported for n5 and zarr data. \n
Returns:
\n\n\n\n", "signature": "(\tpath: str,\tkey: Optional[str] = None,\tlazy_loading: bool = False) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.util.segmentation_to_one_hot", "modulename": "micro_sam.util", "qualname": "segmentation_to_one_hot", "kind": "function", "doc": "The image data.
\n
Convert the segmentation to one-hot encoded masks.
\n\nArguments:
\n\n- \n
- segmentation: The segmentation. \n
- segmentation_ids: Optional subset of ids that will be used to subsample the masks. \n
Returns:
\n\n\n\n", "signature": "(\tsegmentation: numpy.ndarray,\tsegmentation_ids: Optional[numpy.ndarray] = None) -> torch.Tensor:", "funcdef": "def"}, {"fullname": "micro_sam.util.get_block_shape", "modulename": "micro_sam.util", "qualname": "get_block_shape", "kind": "function", "doc": "The one-hot encoded masks.
\n
Get a suitable block shape for chunking a given shape.
\n\nThe primary use for this is determining chunk sizes for\nzarr arrays or block shapes for parallelization.
\n\nArguments:
\n\n- \n
- shape: The image or volume shape. \n
Returns:
\n\n\n\n", "signature": "(shape: Tuple[int]) -> Tuple[int]:", "funcdef": "def"}, {"fullname": "micro_sam.visualization", "modulename": "micro_sam.visualization", "kind": "module", "doc": "The block shape.
\n
Functionality for visualizing image embeddings.
\n"}, {"fullname": "micro_sam.visualization.compute_pca", "modulename": "micro_sam.visualization", "qualname": "compute_pca", "kind": "function", "doc": "Compute the pca projection of the embeddings to visualize them as RGB image.
\n\nArguments:
\n\n- \n
- embeddings: The embeddings. For example predicted by the SAM image encoder. \n
Returns:
\n\n\n\n", "signature": "(embeddings: numpy.ndarray) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.visualization.project_embeddings_for_visualization", "modulename": "micro_sam.visualization", "qualname": "project_embeddings_for_visualization", "kind": "function", "doc": "PCA of the embeddings, mapped to the pixels.
\n
Project image embeddings to pixel-wise PCA.
\n\nArguments:
\n\n- \n
- image_embeddings: The image embeddings. \n
Returns:
\n\n\n\n", "signature": "(\timage_embeddings: Dict[str, Any]) -> Tuple[numpy.ndarray, Tuple[float, ...]]:", "funcdef": "def"}]; + /** pdoc search index */const docs = [{"fullname": "micro_sam", "modulename": "micro_sam", "kind": "module", "doc": "The PCA of the embeddings.\n The scale factor for resizing to the original image size.
\n
Segment Anything for Microscopy
\n\nSegment Anything for Microscopy implements automatic and interactive annotation for microscopy data. It is built on top of Segment Anything by Meta AI and specializes it for microscopy and other bio-imaging data.\nIts core components are:
\n\n- \n
- The
micro_samtools for interactive data annotation, built as napari plugin. \n - The
micro_samlibrary to apply Segment Anything to 2d and 3d data or fine-tune it on your data. \n - The
micro_sammodels that are fine-tuned on publicly available microscopy data and that are available on BioImage.IO. \n
Based on these components micro_sam enables fast interactive and automatic annotation for microscopy data, like interactive cell segmentation from bounding boxes:
micro_sam is now available as stable version 1.0 and we will not change its user interface significantly in the foreseeable future.\nWe are still working on improving and extending its functionality. The current roadmap includes:
- \n
- Releasing more and better finetuned models. \n
- Integrating parameter efficient training and compressed models for faster fine-tuning. \n
- Improving the 3D segmentation and tracking functionality. \n
If you run into any problems or have questions please open an issue or reach out via image.sc using the tag micro-sam.
Quickstart
\n\nYou can install micro_sam via mamba:
$ mamba install -c conda-forge micro_sam\nWe also provide installers for Windows and Linux. For more details on the available installation options check out the installation section.
\n\nAfter installing micro_sam you can start napari and select the annotation tool you want to use from Plugins->Segment Anything for Microscopy. Check out the quickstart tutorial video for a short introduction and the annotation tool section for details.
The micro_sam python library can be imported via
import micro_sam\nIt is explained in more detail here.
\n\nWe provide different finetuned models for microscopy that can be used within our tools or any other tool that supports Segment Anything. See finetuned models for details on the available models.\nYou can also train models on your own data, see here for details.
\n\nCitation
\n\nIf you are using micro_sam in your research please cite
- \n
- Our preprint \n
- and the original Segment Anything publication. \n
- If you use a
vit-tinymodels please also cite Mobile SAM. \n
Installation
\n\nThere are three ways to install micro_sam:
- \n
- From mamba is the recommended way if you want to use all functionality. \n
- From source for setting up a development environment to use the development version and to change and contribute to our software. \n
- From installer to install it without having to use mamba (supported platforms: Windows and Linux, only for CPU users). \n
You can find more information on the installation and how to troubleshoot it in the FAQ section.
\n\nFrom mamba
\n\nmamba is a drop-in replacement for conda, but much faster.\nWhile the steps below may also work with conda, we highly recommend using mamba.\nYou can follow the instructions here to install mamba.
IMPORTANT: Make sure to avoid installing anything in the base environment.
\n\nmicro_sam can be installed in an existing environment via:
$ mamba install -c conda-forge micro_sam\nor you can create a new environment (here called micro-sam) with it via:
$ mamba create -c conda-forge -n micro-sam micro_sam\nif you want to use the GPU you need to install PyTorch from the pytorch channel instead of conda-forge. For example:
$ mamba create -c pytorch -c nvidia -c conda-forge micro_sam pytorch pytorch-cuda=12.1\nYou may need to change this command to install the correct CUDA version for your system, see https://pytorch.org/ for details.
\n\nFrom source
\n\nTo install micro_sam from source, we recommend to first set up an environment with the necessary requirements:
- \n
- environment_gpu.yaml: sets up an environment with GPU support. \n
- environment_cpu.yaml: sets up an environment with CPU support. \n
To create one of these environments and install micro_sam into it follow these steps
- \n
- Clone the repository: \n
$ git clone https://github.com/computational-cell-analytics/micro-sam\n- \n
- Enter it: \n
$ cd micro-sam\n- \n
- Create the GPU or CPU environment: \n
$ mamba env create -f <ENV_FILE>.yaml\n- \n
- Activate the environment: \n
$ mamba activate sam\n- \n
- Install
micro_sam: \n
$ pip install -e .\nFrom installer
\n\nWe also provide installers for Linux and Windows:
\n\n\n\nThe installers will not enable you to use a GPU, so if you have one then please consider installing micro_sam via mamba instead. They will also not enable using the python library.
Linux Installer:
\n\nTo use the installer:
\n\n- \n
- Unpack the zip file you have downloaded. \n
- Make the installer executable:
$ chmod +x micro_sam-0.2.0post1-Linux-x86_64.sh\n - Run the installer:
$./micro_sam-0.2.0post1-Linux-x86_64.sh$\n- \n
- You can select where to install
micro_samduring the installation. By default it will be installed in$HOME/micro_sam. \n - The installer will unpack all
micro_samfiles to the installation directory. \n
\n - You can select where to install
- After the installation you can start the annotator with the command
.../micro_sam/bin/micro_sam.annotator.\n- \n
- To make it easier to run the annotation tool you can add
.../micro_sam/binto yourPATHor set a softlink to.../micro_sam/bin/micro_sam.annotator. \n
\n - To make it easier to run the annotation tool you can add
Windows Installer:
\n\n- \n
- Unpack the zip file you have downloaded. \n
- Run the installer by double clicking on it. \n
- Choose installation type:
Just Me(recommended)orAll Users(requires admin privileges). \n - Choose installation path. By default it will be installed in
C:\\Users\\<Username>\\micro_samforJust Meinstallation or inC:\\ProgramData\\micro_samforAll Users.\n- \n
- The installer will unpack all micro_sam files to the installation directory. \n
\n - After the installation you can start the annotator by double clicking on
.\\micro_sam\\Scripts\\micro_sam.annotator.exeor with the command.\\micro_sam\\Scripts\\micro_sam.annotator.exefrom the Command Prompt. \n
Annotation Tools
\n\nmicro_sam provides applications for fast interactive 2d segmentation, 3d segmentation and tracking.\nSee an example for interactive cell segmentation in phase-contrast microscopy (left), interactive segmentation\nof mitochondria in volume EM (middle) and interactive tracking of cells (right).
\n
\n
The annotation tools can be started from the napari plugin menu, the command line or from python scripts.\nThey are built as napari plugin and make use of existing napari functionality wherever possible. If you are not familiar with napari yet, start here.\nThe micro_sam tools mainly use the point layer, shape layer and label layer.
The annotation tools are explained in detail below. We also provide video tutorials.
\n\nThe annotation tools can be started from the napari plugin menu:\n
You can find additional information on the annotation tools in the FAQ section.
\n\nAnnotator 2D
\n\nThe 2d annotator can be started by
\n\n- \n
- clicking
Annotator 2din the plugin menu. \n - running
$ micro_sam.annotator_2din the command line. \n - calling
micro_sam.sam_annotator.annotator_2din a python script. Check out examples/annotator_2d.py for details. \n
The user interface of the 2d annotator looks like this:
\n\n
It contains the following elements:
\n\n- \n
- The napari layers for the segmentations and prompts:\n
- \n
prompts: shape layer that is used to provide box prompts to SegmentAnything. Annotations can be given as rectangle (box prompt in the image), ellipse or polygon. \npoint_prompts: point layer that is used to provide point prompts to SegmentAnything. Positive prompts (green points) for marking the object you want to segment, negative prompts (red points) for marking the outside of the object. \ncommitted_objects: label layer with the objects that have already been segmented. \nauto_segmentation: label layer with the results from automatic instance segmentation. \ncurrent_object: label layer for the object(s) you're currently segmenting. \n
\n - The embedding menu. For selecting the image to process, the Segment Anything model that is used and computing the image embeddings with the model. The
Embedding Settingscontain advanced settings for loading cached embeddings from file or using tiled embeddings. \n - The prompt menu for changing whether the currently selected point is a positive or a negative prompt. This can also be done by pressing
T. \n - The menu for interactive segmentation. Clicking
Segment Object(or pressingS) will run segmentation for the current prompts. The result is displayed incurrent_object. Activatingbatchedenables segmentation of multiple objects with point prompts. In this case an object will be segmented per positive prompt. \n - The menu for automatic segmentation. Clicking
Automatic Segmentationwill segment all objects n the image. The results will be displayed in theauto_segmentationlayer. We support two different methods for automatic segmentation: automatic mask generation (supported for all models) and instance segmentation with an additional decoder (only supported for our models).\nChanging the parameters underAutomatic Segmentation Settingscontrols the segmentation results, check the tooltips for details. \n - The menu for commiting the segmentation. When clicking
Commit(or pressingC) the result from the selected layer (eithercurrent_objectorauto_segmentation) will be transferred from the respective layer tocommitted_objects.\nWhencommit_pathis given the results will automatically be saved there. \n - The menu for clearing the current annotations. Clicking
Clear Annotations(or pressingShift + C) will clear the current annotations and the current segmentation. \n
Note that point prompts and box prompts can be combined. When you're using point prompts you can only segment one object at a time, unless the batched mode is activated. With box prompts you can segment several objects at once, both in the normal and batched mode.
Check out this video for a tutorial for this tool.
\n\nAnnotator 3D
\n\nThe 3d annotator can be started by
\n\n- \n
- clicking
Annotator 3din the plugin menu. \n - running
$ micro_sam.annotator_3din the command line. \n - calling
micro_sam.sam_annotator.annotator_3din a python script. Check out examples/annotator_3d.py for details. \n
The user interface of the 3d annotator looks like this:
\n\n
Most elements are the same as in the 2d annotator:
\n\n- \n
- The napari layers that contain the segmentations and prompts. \n
- The embedding menu. \n
- The prompt menu. \n
- The menu for interactive segmentation. \n
- The menu for interactive 3d segmentation. Clicking
Segment All Slices(orShift + S) will extend the segmentation for the current object across the volume by projecting prompts across slices. The parameters for prompt projection can be set inSegmentation Settings, please refer to the tooltips for details. \n - The menu for automatic segmentation. The overall functionality is the same as for the 2d annotator. To segment the full volume
Apply to Volumeneeds to be checked, otherwise only the current slice will be segmented. Note that 3D segmentation can take quite long without a GPU. \n - The menu for committing the current object. \n
- The menu for clearing the current annotations. If
all slicesis set all annotations will be cleared, otherwise they are only cleared for the current slice. \n
Note that you can only segment one object at a time using the interactive segmentation functionality with this tool.
\n\nCheck out this video for a tutorial for the 3d annotation tool.
\n\nAnnotator Tracking
\n\nThe tracking annotator can be started by
\n\n- \n
- clicking
Annotator Trackingin the plugin menu. \n - running
$ micro_sam.annotator_trackingin the command line. \n - calling
micro_sam.sam_annotator.annotator_trackingin a python script. Check out examples/annotator_tracking.py for details. \n
The user interface of the tracking annotator looks like this:
\n\n
Most elements are the same as in the 2d annotator:
\n\n- \n
- The napari layers that contain the segmentations and prompts. Same as for the 2d segmentation app but without the
auto_segmentationlayer. \n - The embedding menu. \n
- The prompt menu. \n
- The menu with tracking settings:
track_stateis used to indicate that the object you are tracking is dividing in the current frame.track_idis used to select which of the tracks after division you are following. \n - The menu for interactive segmentation. \n
- The menu for interactive tracking menu. Click
Track Object(or pressShift + S) to segment the current object across time. \n - The menu for committing the current tracking result. \n
- The menu for clearing the current annotations. \n
Note that the tracking annotator only supports 2d image data, volumetric data is not supported. We also do not support automatic tracking yet.
\n\nCheck out this video for a tutorial for how to use the tracking annotation tool.
\n\nImage Series Annotator
\n\nThe image series annotation tool enables running the 2d annotator or 2d annotator for multiple images that are saved within an folder. This makes it convenient to annotate many images without having to close the tool. It can be started by
\n\n- \n
- clicking
Image Series Annotatorin the plugin menu. \n - running
$ micro_sam.image_series_annotatorin the command line. \n - calling
micro_sam.sam_annotator.image_series_annotatorin a python script. Check out examples/image_series_annotator.py for details. \n
When starting this tool via the plugin menu the following interface opens:
\n\n
You can select the folder where your image data is saved with Input Folder. The annotation results will be saved in Output Folder.\nYou can specify a rule for loading only a subset of images via pattern, for example *.tif to only load tif images. Set is_volumetric if the data you want to annotate is 3d. The rest of the options are settings for the image embedding computation and are the same as for the embedding menu (see above).\nOnce you click Annotate Images the images from the folder you have specified will be loaded and the annotation tool is started for them.
This menu will not open if you start the image series annotator from the command line or via python. In this case the input folder and other settings are passed as parameters instead.
\n\nCheck out this video for a tutorial for how to use the image series annotator.
\n\nFinetuning UI
\n\nWe also provide a graphical tool for finetuning models on your own data. It can be started by clicking Finetuning in the plugin menu.
Note: if you know a bit of python programming we recommend to use a script for model finetuning instead. This will give you more options to configure the training. See these instructions for details.
\n\nWhen starting this tool via the plugin menu the following interface opens:
\n\n
You can select the image data via Path to images. We can either load images from a folder or select a single file for training. By providing Image data key you can either provide a pattern for selecting files from a folder or provide an internal filepath for hdf5, zarr or similar fileformats.
You can select the label data via Path to labels and Label data key, following the same logic as for the image data. We expect label masks stored in the same size as the image data for training. You can for example use annotations created with one of the micro_sam annotation tools for this, they are stored in the correct format!
The Configuration option allows you to choose the hardware configuration for training. We try to automatically select the correct setting for your system, but it can also be changed. Please refer to the tooltips for the other parameters.
Using the Python Library
\n\nThe python library can be imported via
\n\nimport micro_sam\nThis library extends the Segment Anything library and
\n\n- \n
- implements functions to apply Segment Anything to 2d and 3d data in
micro_sam.prompt_based_segmentation. \n - provides improved automatic instance segmentation functionality in
micro_sam.instance_segmentation. \n - implements training functionality that can be used for finetuning on your own data in
micro_sam.training. \n - provides functionality for quantitative and qualitative evaluation of Segment Anything models in
micro_sam.evaluation. \n
You can import these sub-modules via
\n\nimport micro_sam.prompt_based_segmentation\nimport micro_sam.instance_segmentation\n# etc.\nThis functionality is used to implement the interactive annotation tools in micro_sam.sam_annotator and can be used as a standalone python library.\nWe provide jupyter notebooks that demonstrate how to use it here. You can find the full library documentation by scrolling to the end of this page.
Training your own model
\n\nWe reimplement the training logic described in the Segment Anything publication to enable finetuning on custom data.\nWe use this functionality to provide the finetuned microscopy models and it can also be used to train models on your own data.\nIn fact the best results can be expected when finetuning on your own data, and we found that it does not require much annotated training data to get siginficant improvements in model performance.\nSo a good strategy is to annotate a few images with one of the provided models using our interactive annotation tools and, if the model is not working as good as required for your use-case, finetune on the annotated data.\n
\n\nThe training logic is implemented in micro_sam.training and is based on torch-em. Check out the finetuning notebook to see how to use it.
We also support training an additional decoder for automatic instance segmentation. This yields better results than the automatic mask generation of segment anything and is significantly faster.\nThe notebook explains how to activate training it together with the rest of SAM and how to then use it.
\n\nMore advanced examples, including quantitative and qualitative evaluation, of finetuned models can be found in finetuning, which contains the code for training and evaluating our models. You can find further information on model training in the FAQ section.
\n\nFinetuned models
\n\nIn addition to the original Segment Anything models, we provide models that are finetuned on microscopy data.\nThe additional models are available in the bioimage.io modelzoo and are also hosted on zenodo.
\n\nWe currently offer the following models:
\n\n- \n
vit_h: Default Segment Anything model with vit-h backbone. \nvit_l: Default Segment Anything model with vit-l backbone. \nvit_b: Default Segment Anything model with vit-b backbone. \nvit_t: Segment Anything model with vit-tiny backbone. From the Mobile SAM publication. \nvit_l_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-l backbone. (zenodo, bioimage.io) \nvit_b_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-b backbone. (zenodo, diplomatic-bug on bioimage.io) \nvit_t_lm: Finetuned Segment Anything model for cells and nuclei in light microscopy data with vit-t backbone. (zenodo, bioimage.io) \nvit_l_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-l backbone. (zenodo, bioimage.io) \nvit_b_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-b backbone. (zenodo, bioimage.io) \nvit_t_em_organelles: Finetuned Segment Anything model for mitochodria and nuclei in electron microscopy data with vit-t backbone. (zenodo, bioimage.io) \n
See the two figures below of the improvements through the finetuned model for LM and EM data.
\n\n
You can select which model to use for annotation by selecting the corresponding name in the embedding menu:
\n\n
To use a specific model in the python library you need to pass the corresponding name as value to the model_type parameter exposed by all relevant functions.\nSee for example the 2d annotator example.
Choosing a Model
\n\nAs a rule of thumb:
\n\n- \n
- Use the
vit_l_lmorvit_b_lmmodel for segmenting cells or nuclei in light microscopy. The larger model (vit_l_lm) yields a bit better segmentation quality, especially for automatic segmentation, but needs more computational resources. \n - Use the
vit_l_em_organellesorvit_b_em_organellesmodels for segmenting mitochondria, nuclei or other roundish organelles in electron microscopy. \n - For other use-cases use one of the default models. \n
- The
vit_t_...models run much faster than other models, but yield inferior quality for many applications. It can still make sense to try them for your use-case if your working on a laptop and want to annotate many images or volumetric data. \n
See also the figures above for examples where the finetuned models work better than the default models.\nWe are working on further improving these models and adding new models for other biomedical imaging domains.
\n\nOlder Models
\n\nPrevious versions of our models are available on zenodo:
\n\n- \n
- vit_b_em_boundaries: for segmenting compartments delineated by boundaries such as cells or neurites in EM. \n
- vit_b_em_organelles: for segmenting mitochondria, nuclei or other organelles in EM. \n
- vit_b_lm: for segmenting cells and nuclei in LM. \n
- vit_h_em: for general EM segmentation. \n
- vit_h_lm: for general LM segmentation. \n
We do not recommend to use these models since our new models improve upon them significantly. But we provide the links here in case they are needed to reproduce older segmentation workflows.
\n\nFAQ
\n\nHere we provide frequently asked questions and common issues.\nIf you encounter a problem or question not addressed here feel free to open an issue or to ask your question on image.sc with the tag micro-sam.
Installation questions
\n\n1. How to install micro_sam?
\n\nThe installation for micro_sam is supported in three ways: from mamba (recommended), from source and from installers. Check out our tutorial video to get started with micro_sam, briefly walking you through the installation process and how to start the tool.
2. I cannot install micro_sam using the installer, I am getting some errors.
\n\nThe installer should work out-of-the-box on Windows and Linux platforms. Please open an issue to report the error you encounter.
\n\n\n\n\nNOTE: The installers enable using
\nmicro_samwithout mamba or conda. However, we recommend the installation from mamba / from source to use all its features seamlessly. Specifically, the installers currently only support the CPU and won't enable you to use the GPU (if you have one).
3. What is the minimum system requirement for micro_sam?
\n\nFrom our experience, the micro_sam annotation tools work seamlessly on most laptop or workstation CPUs and with > 8GB RAM.\nYou might encounter some slowness for $leq$ 8GB RAM. The resources micro_sam's annotation tools have been tested on are:
- \n
- Windows:\n
- \n
- Windows 10 Pro, Intel i5 7th Gen, 8GB RAM \n
\n - Linux:\n
- \n
- Ubuntu 22.04, Intel i7 12th Gen, 32GB RAM \n
\n - Mac:\n
- \n
- macOS Sonoma 14.4.1\n
- \n
- M1 Chip, 8GB RAM \n
- M3 Max Chip, 36GB RAM \n
\n
\n - macOS Sonoma 14.4.1\n
Having a GPU will significantly speed up the annotation tools and especially the model finetuning.
\n\n4. What is the recommended PyTorch version?
\n\nmicro_sam has been tested mostly with CUDA 12.1 and PyTorch [2.1.1, 2.2.0]. However, the tool and the library is not constrained to a specific PyTorch or CUDA version. So it should work fine with the standard PyTorch installation for your system.
5. I am missing a few packages (eg. ModuleNotFoundError: No module named 'elf.io). What should I do?
\n\nWith the latest release 1.0.0, the installation from mamba and source should take care of this and install all the relevant packages for you.\nSo please reinstall micro_sam.
6. Can I install micro_sam using pip?
\n\nThe installation is not supported via pip.
\n\n7. I get the following error: importError: cannot import name 'UNETR' from 'torch_em.model'.
\n\nIt's possible that you have an older version of torch-em installed. Similar errors could often be raised from other libraries, the reasons being: a) Outdated packages installed, or b) Some non-existent module being called. If the source of such error is from micro_sam, then a) is most likely the reason . We recommend installing the latest version following the installation instructions.
Usage questions
\n\n\n\n1. I have some micropscopy images. Can I use the annotator tool for segmenting them?
\n\nYes, you can use the annotator tool for:
\n\n- \n
- Segmenting objects in 2d images (using automatic and/or interactive segmentation). \n
- Segmenting objects in 3d volumes (using automatic and/or interactive segmentation for the entire object(s)). \n
- Tracking objects over time in time-series data. \n
- Segmenting objects in a series of 2d / 3d images. \n
- (OPTIONAL) You can finetune the Segment Anything /
micro_sammodels on your own microscopy data, in case the provided models do not suffice your needs. One caveat: You need to annotate a few objects before-hand (micro_samhas the potential of improving interactive segmentation with only a few annotated objects) to proceed with the supervised finetuning procedure. \n
2. Which model should I use for my data?
\n\nWe currently provide three different kind of models: the default models vit_h, vit_l, vit_b and vit_t; the models for light microscopy vit_l_lm, vit_b_lm and vit_t_lm; the models for electron microscopy vit_l_em_organelles, vit_b_em_organelles and vit_t_em_organelles.\nYou should first try the model that best fits the segmentation task your interested in, a lm model for cell or nucleus segmentation in light microscopy or a em_organelles model for segmenting nuclei, mitochondria or other roundish organelles in electron microscopy.\nIf your segmentation problem does not meet these descriptions, or if these models don't work well, you should try one of the default models instead.\nThe letter after vit denotes the size of the image encoder in SAM, h (huge) being the largest and t (tiny) the smallest. The smaller models are faster but may yield worse results. We recommend to either use a vit_l or vit_b model, they offer the best trade-off between speed and segmentation quality.\nYou can find more information on model choice here.
3. I have high-resolution microscopy images, 'micro_sam' does not seem to work.
\n\nThe Segment Anything model expects inputs of shape 1024 x 1024 pixels. Inputs that do not match this size will be internally resized to match it. Hence, applying Segment Anything to a much larger image will often lead to inferior results, or somethimes not work at all. To address this, micro_sam implements tiling: cutting up the input image into tiles of a fixed size (with a fixed overlap) and running Segment Anything for the individual tiles. You can activate tiling with the tile_shape parameter, which determines the size of the inner tile and halo, which determines the size of the additional overlap.
- \n
- If you are using the
micro_samannotation tools, you can specify the values for thetile_shapeandhalovia thetile_x,tile_y,halo_xandhalo_yparameters in theEmbedding Settingsdrop-down menu. \n - If you are using the
micro_samlibrary in a python script, you can pass them as tuples, e.g.tile_shape=(1024, 1024), halo=(256, 256). See also the wholeslide annotator example. \n - If you are using the command line functionality, you can pass them via the options
--tile_shape 1024 1024 --halo 256 256. \n
\n\n\nNOTE: It's recommended to choose the
\nhaloso that it is larger than half of the maximal radius of the objects you want to segment.
4. The computation of image embeddings takes very long in napari.
\n\nmicro_sam pre-computes the image embeddings produced by the vision transformer backbone in Segment Anything, and (optionally) store them on disc. I fyou are using a CPU, this step can take a while for 3d data or time-series (you will see a progress bar in the command-line interface / on the bootom right of napari). If you have access to a GPU without graphical interface (e.g. via a local computer cluster or a cloud provider), you can also pre-compute the embeddings there and then copy them over to your laptop / local machine to speed this up.
- \n
- You can use the command
micro_sam.precompute_embeddingsfor this (it is installed with the rest of the software). You can specify the location of the precomputed embeddings via theembedding_pathargument. \n - You can cache the computed embedding in the napari tool (to avoid recomputing the embeddings again) by passing the path to store the embeddings in the
embeddings_save_pathoption in theEmbedding Settingsdrop-down. You can later load the precomputed image embeddings by entering the path to the stored embeddings there as well. \n
5. Can I use micro_sam on a CPU?
\n\nMost other processing steps that are very fast even on a CPU, the automatic segmentation step for the default Segment Anything models (typically called as the \"Segment Anything\" feature or AMG - Automatic Mask Generation) takes several minutes without a GPU (depending on the image size). For large volumes and time-series, segmenting an object interactively in 3d / tracking across time can take a couple of seconds with a CPU (it is very fast with a GPU).
\n\n\n\n\nHINT: All the tutorial videos have been created on CPU resources.
\n
6. I generated some segmentations from another tool, can I use it as a starting point in micro_sam?
\n\nYou can save and load the results from the committed_objects layer to correct segmentations you obtained from another tool (e.g. CellPose) or save intermediate annotation results. The results can be saved via File -> Save Selected Layers (s) ... in the napari menu-bar on top (see the tutorial videos for details). They can be loaded again by specifying the corresponding location via the segmentation_result parameter in the CLI or python script (2d and 3d segmentation).\nIf you are using an annotation tool you can load the segmentation you want to edit as segmentation layer and renae it to committed_objects.
7. I am using micro_sam for segmenting objects. I would like to report the steps for reproducability. How can this be done?
\n\nThe annotation steps and segmentation results can be saved to a zarr file by providing the commit_path in the commit widget. This file will contain all relevant information to reproduce the segmentation.
\n\n\nNOTE: This feature is still under development and we have not implemented rerunning the segmentation from this file yet. See this issue for details.
\n
8. I want to segment complex objects. Both the default Segment Anything models and the micro_sam generalist models do not work for my data. What should I do?
\n\nmicro_sam supports interactive annotation using positive and negative point prompts, box prompts and polygon drawing. You can combine multiple types of prompts to improve the segmentation quality. In case the aforementioned suggestions do not work as desired, micro_sam also supports finetuning a model on your data (see the next section). We recommend the following: a) Check which of the provided models performs relatively good on your data, b) Choose the best model as the starting point to train your own specialist model for the desired segmentation task.
9. I am using the annotation tool and napari outputs the following error: While emmitting signal ... an error ocurred in callback ... This is not a bug in psygnal. See ... above for details.
\n\nThese messages occur when an internal error happens in micro_sam. In most cases this is due to inconsistent annotations and you can fix them by clearing the annotations.\nWe want to remove these errors, so we would be very grateful if you can open an issue and describe the steps you did when encountering it.
10. The objects are not segmented in my 3d data using the interactive annotation tool.
\n\nThe first thing to check is: a) make sure you are using the latest version of micro_sam (pull the latest commit from master if your installation is from source, or update the installation from conda / mamba using mamba update micro_sam), and b) try out the steps from the 3d annotator tutorial video to verify if this shows the same behaviour (or the same errors) as you faced. For 3d images, it's important to pass the inputs in the python axis convention, ZYX.\nc) try using a different model and change the projection mode for 3d segmentation. This is also explained in the video.
11. I have very small or fine-grained structures in my high-resolution microscopic images. Can I use micro_sam to annotate them?
\n\nSegment Anything does not work well for very small or fine-grained objects (e.g. filaments). In these cases, you could try to use tiling to improve results (see Point 3 above for details).
\n\n12. napari seems to be very slow for large images.
\n\nEditing (drawing / erasing) very large 2d images or 3d volumes is known to be slow at the moment, as the objects in the layers are stored in-memory. See the related issue.
\n\n13. While computing the embeddings (and / or automatic segmentation), a window stating: \"napari\" is not responding. pops up.
\n\nThis can happen for long running computations. You just need to wait a bit longer and the computation will finish.
\n\nFine-tuning questions
\n\n1. I have a microscopy dataset I would like to fine-tune Segment Anything for. Is it possible using 'micro_sam'?
\n\nYes, you can fine-tune Segment Anything on your own dataset. Here's how you can do it:
\n\n- \n
- Check out the tutorial notebook on how to fine-tune Segment Anything with our
micro_sam.traininglibrary. \n - Or check the examples for additional scripts that demonstrate finetuning. \n
- If you are not familiar with coding in python at all then you can also use the graphical interface for finetuning. But we recommend using a script for more flexibility and reproducibility. \n
2. I would like to fine-tune Segment Anything on open-source cloud services (e.g. Kaggle Notebooks), is it possible?
\n\nYes, you can fine-tune Segment Anything on your custom datasets on Kaggle (and BAND). Check out our tutorial notebook for this.
\n\n3. What kind of annotations do I need to finetune Segment Anything?
\n\nTODO: explain instance segmentation labels, that you can get them by annotation with micro_sam, and dense vs. sparse annotation (for training without / with decoder)
\n\n4. I have finetuned Segment Anything on my microscopy data. How can I use it for annotating new images?
\n\nYou can load your finetuned model by entering the path to its checkpoint in the custom_weights_path field in the Embedding Settings drop-down menu.\nIf you are using the python library or CLI you can specify this path with the checkpoint_path parameter.
5. What is the background of the new AIS (Automatic Instance Segmentation) feature in micro_sam?
\n\nmicro_sam introduces a new segmentation decoder to the Segment Anything backbone, for enabling faster and accurate automatic instance segmentation, by predicting the distances to the object center and boundary as well as predicting foregrund, and performing seeded watershed-based postprocessing to obtain the instances.\n
6. I have a NVIDIA RTX 4090Ti GPU with 24GB VRAM. Can I finetune Segment Anything?
\n\nFinetuning Segment Anything is possible in most consumer-grade GPU and CPU resources (but training being a lot slower on the CPU). For the mentioned resource, it should be possible to finetune a ViT Base (also abbreviated as vit_b) by reducing the number of objects per image to 15.\nThis parameter has the biggest impact on the VRAM consumption and quality of the finetuned model.\nYou can find an overview of the resources we have tested for finetuning here.\nWe also provide a the convenience function micro_sam.training.train_sam_for_configuration that selects the best training settings for these configuration. This function is also used by the finetuning UI.
7. I want to create a dataloader for my data, for finetuning Segment Anything.
\n\nThanks to torch-em, a) Creating PyTorch datasets and dataloaders using the python library is convenient and supported for various data formats and data structures.\nSee the tutorial notebook on how to create dataloaders using torch-em and the documentation for details on creating your own datasets and dataloaders; and b) finetuning using the napari tool eases the aforementioned process, by allowing you to add the input parameters (path to the directory for inputs and labels etc.) directly in the tool.
\n\n\nNOTE: If you have images with large input shapes with a sparse density of instance segmentations, we recommend using
\nsamplerfor choosing the patches with valid segmentation for the finetuning purpose (see the example for PlantSeg (Root) specialist model inmicro_sam).
8. How can I evaluate a model I have finetuned?
\n\nTODO: move the content of https://github.com/computational-cell-analytics/micro-sam/blob/master/doc/bioimageio/validation.md here.
\n\nContribution Guide
\n\n- \n
- Discuss your ideas \n
- Clone the repository \n
- Create your development environment \n
- Make your changes \n
- Testing\n \n
- Open a pull request \n
- Optional: Build the documentation \n
- Optional: Benchmark performance\n \n
Discuss your ideas
\n\nWe welcome new contributions!
\n\nFirst, discuss your idea by opening a new issue in micro-sam.
\n\nThis allows you to ask questions, and have the current developers make suggestions about the best way to implement your ideas.
\n\nYou may also find it helpful to look at this developer guide, which explains the organization of the micro-sam code.
\n\nClone the repository
\n\nWe use git for version control.
\n\nClone the repository, and checkout the development branch:
\n\ngit clone https://github.com/computational-cell-analytics/micro-sam.git\ncd micro-sam\ngit checkout dev\nCreate your development environment
\n\nWe use conda to manage our environments. If you don't have this already, install miniconda or mamba to get started.
\n\nNow you can create the environment, install user and develoepr dependencies, and micro-sam as an editable installation:
\n\nconda env create environment-gpu.yml\nconda activate sam\npython -m pip install requirements-dev.txt\npython -m pip install -e .\nMake your changes
\n\nNow it's time to make your code changes.
\n\nTypically, changes are made branching off from the development branch. Checkout dev and then create a new branch to work on your changes.
git checkout dev\ngit checkout -b my-new-feature\nWe use google style python docstrings to create documentation for all new code.
\n\nYou may also find it helpful to look at this developer guide, which explains the organization of the micro-sam code.
\n\nTesting
\n\nRun the tests
\n\nThe tests for micro-sam are run with pytest
\n\nTo run the tests:
\n\npytest\nWriting your own tests
\n\nIf you have written new code, you will need to write tests to go with it.
\n\nUnit tests
\n\nUnit tests are the preferred style of tests for user contributions. Unit tests check small, isolated parts of the code for correctness. If your code is too complicated to write unit tests easily, you may need to consider breaking it up into smaller functions that are easier to test.
\n\nTests involving napari
\n\nIn cases where tests must use the napari viewer, these tips might be helpful (in particular, the make_napari_viewer_proxy fixture).
These kinds of tests should be used only in limited circumstances. Developers are advised to prefer smaller unit tests, and avoid integration tests wherever possible.
\n\nCode coverage
\n\nPytest uses the pytest-cov plugin to automatically determine which lines of code are covered by tests.
\n\nA short summary report is printed to the terminal output whenever you run pytest. The full results are also automatically written to a file named coverage.xml.
The Coverage Gutters VSCode extension is useful for visualizing which parts of the code need better test coverage. PyCharm professional has a similar feature, and you may be able to find similar tools for your preferred editor.
\n\nWe also use codecov.io to display the code coverage results from our Github Actions continuous integration.
\n\nOpen a pull request
\n\nOnce you've made changes to the code and written some tests to go with it, you are ready to open a pull request. You can mark your pull request as a draft if you are still working on it, and still get the benefit of discussing the best approach with maintainers.
\n\nRemember that typically changes to micro-sam are made branching off from the development branch. So, you will need to open your pull request to merge back into the dev branch like this.
Optional: Build the documentation
\n\nWe use pdoc to build the documentation.
\n\nTo build the documentation locally, run this command:
\n\npython build_doc.py\nThis will start a local server and display the HTML documentation. Any changes you make to the documentation will be updated in real time (you may need to refresh your browser to see the changes).
\n\nIf you want to save the HTML files, append --out to the command, like this:
python build_doc.py --out\nThis will save the HTML files into a new directory named tmp.
You can add content to the documentation in two ways:
\n\n- \n
- By adding or updating google style python docstrings in the micro-sam code.\n
- \n
- pdoc will automatically find and include docstrings in the documentation. \n
\n - By adding or editing markdown files in the micro-sam
docdirectory.\n- \n
- If you add a new markdown file to the documentation, you must tell pdoc that it exists by adding a line to the
micro_sam/__init__.pymodule docstring (eg:.. include:: ../doc/my_amazing_new_docs_page.md). Otherwise it will not be included in the final documentation build! \n
\n - If you add a new markdown file to the documentation, you must tell pdoc that it exists by adding a line to the
Optional: Benchmark performance
\n\nThere are a number of options you can use to benchmark performance, and identify problems like slow run times or high memory use in micro-sam.
\n\n\n\nRun the benchmark script
\n\nThere is a performance benchmark script available in the micro-sam repository at development/benchmark.py.
To run the benchmark script:
\n\npython development/benchmark.py --model_type vit_t --device cpu`\nFor more details about the user input arguments for the micro-sam benchmark script, see the help:
\n\npython development/benchmark.py --help\nLine profiling
\n\nFor more detailed line by line performance results, we can use line-profiler.
\n\n\n\n\nline_profiler is a module for doing line-by-line profiling of functions. kernprof is a convenient script for running either line_profiler or the Python standard library's cProfile or profile modules, depending on what is available.
\n
To do line-by-line profiling:
\n\n- \n
- Ensure you have line profiler installed:
python -m pip install line_profiler\n - Add
@profiledecorator to any function in the call stack \n - Run
kernprof -lv benchmark.py --model_type vit_t --device cpu\n
For more details about how to use line-profiler and kernprof, see the documentation.
\n\nFor more details about the user input arguments for the micro-sam benchmark script, see the help:
\n\npython development/benchmark.py --help\nSnakeviz visualization
\n\nFor more detailed visualizations of profiling results, we use snakeviz.
\n\n\n\n\nSnakeViz is a browser based graphical viewer for the output of Python\u2019s cProfile module
\n
- \n
- Ensure you have snakeviz installed:
python -m pip install snakeviz\n - Generate profile file:
python -m cProfile -o program.prof benchmark.py --model_type vit_h --device cpu\n - Visualize profile file:
snakeviz program.prof\n
For more details about how to use snakeviz, see the documentation.
\n\nMemory profiling with memray
\n\nIf you need to investigate memory use specifically, we use memray.
\n\n\n\n\nMemray is a memory profiler for Python. It can track memory allocations in Python code, in native extension modules, and in the Python interpreter itself. It can generate several different types of reports to help you analyze the captured memory usage data. While commonly used as a CLI tool, it can also be used as a library to perform more fine-grained profiling tasks.
\n
For more details about how to use memray, see the documentation.
\n\nUsing micro_sam on BAND
\n\nBAND is a service offered by EMBL Heidelberg that gives access to a virtual desktop for image analysis tasks. It is free to use and micro_sam is installed there.\nIn order to use BAND and start micro_sam on it follow these steps:
Start BAND
\n\n- \n
- Go to https://band.embl.de/ and click Login. If you have not used BAND before you will need to register for BAND. Currently you can only sign up via a google account. \n
- Launch a BAND desktop with sufficient resources. It's particularly important to select a GPU. The settings from the image below are a good choice. \n
- Go to the desktop by clicking GO TO DESKTOP in the Running Desktops menu. See also the screenshot below. \n
Start micro_sam in BAND
\n\n- \n
- Select Applications->Image Analysis->uSAM (see screenshot)\n
- This will open the micro_sam menu, where you can select the tool you want to use (see screenshot). Note: this may take a few minutes.\n
- For testing if the tool works, it's best to use the 2d annotator first.\n
- \n
- You can find an example image to use here:
/scratch/cajal-connectomics/hela-2d-image.png. Select it via Select image. (see screenshot)\n \n
\n - You can find an example image to use here:
- Then press 2d annotator and the tool will start. \n
Transfering data to BAND
\n\nTo copy data to and from BAND you can use any cloud storage, e.g. ownCloud, dropbox or google drive. For this, it's important to note that copy and paste, which you may need for accessing links on BAND, works a bit different in BAND:
\n\n- \n
- To copy text into BAND you first need to copy it on your computer (e.g. via selecting it +
ctrl + c). \n - Then go to the browser window with BAND and press
ctrl + shift + alt. This will open a side window where you can paste your text viactrl + v. \n - Then select the text in this window and copy it via
ctrl + c. \n - Now you can close the side window via
ctrl + shift + altand paste the text in band viactrl + v\n
The video below shows how to copy over a link from owncloud and then download the data on BAND using copy and paste:
\n\n\n"}, {"fullname": "micro_sam.bioimageio", "modulename": "micro_sam.bioimageio", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.bioimageio.model_export", "modulename": "micro_sam.bioimageio.model_export", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.bioimageio.model_export.DEFAULTS", "modulename": "micro_sam.bioimageio.model_export", "qualname": "DEFAULTS", "kind": "variable", "doc": "\n", "default_value": "{'authors': [Author(affiliation='University Goettingen', email=None, orcid=None, name='Anwai Archit', github_user='anwai98'), Author(affiliation='University Goettingen', email=None, orcid=None, name='Constantin Pape', github_user='constantinpape')], 'description': 'Finetuned Segment Anything Model for Microscopy', 'cite': [CiteEntry(text='Archit et al. Segment Anything for Microscopy', doi='10.1101/2023.08.21.554208', url=None)], 'tags': ['segment-anything', 'instance-segmentation']}"}, {"fullname": "micro_sam.bioimageio.model_export.export_sam_model", "modulename": "micro_sam.bioimageio.model_export", "qualname": "export_sam_model", "kind": "function", "doc": "Export SAM model to BioImage.IO model format.
\n\nThe exported model can be uploaded to bioimage.io and\nbe used in tools that support the BioImage.IO model format.
\n\nArguments:
\n\n- \n
- image: The image for generating test data. \n
- label_image: The segmentation correspoding to
image.\nIt is used to derive prompt inputs for the model. \n - model_type: The type of the SAM model. \n
- name: The name of the exported model. \n
- output_path: Where the exported model is saved. \n
- checkpoint_path: Optional checkpoint for loading the SAM model. \n
Wrapper around the SamPredictor.
\n\nThis model supports the same functionality as SamPredictor and can provide mask segmentations\nfrom box, point or mask input prompts.
\n\nArguments:
\n\n- \n
- model_type: The type of the model for the image encoder.\nCan be one of 'vit_b', 'vit_l', 'vit_h' or 'vit_t'.\nFor 'vit_t' support the 'mobile_sam' package has to be installed. \n
Initializes internal Module state, shared by both nn.Module and ScriptModule.
\n", "signature": "(model_type: str)"}, {"fullname": "micro_sam.bioimageio.predictor_adaptor.PredictorAdaptor.sam", "modulename": "micro_sam.bioimageio.predictor_adaptor", "qualname": "PredictorAdaptor.sam", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.bioimageio.predictor_adaptor.PredictorAdaptor.load_state_dict", "modulename": "micro_sam.bioimageio.predictor_adaptor", "qualname": "PredictorAdaptor.load_state_dict", "kind": "function", "doc": "Copies parameters and buffers from state_dict into\nthis module and its descendants. If strict is True, then\nthe keys of state_dict must exactly match the keys returned\nby this module's ~torch.nn.Module.state_dict() function.
If assign is True the optimizer must be created after\nthe call to load_state_dict.
Arguments:
\n\n- \n
- state_dict (dict): a dict containing parameters and\npersistent buffers. \n
- strict (bool, optional): whether to strictly enforce that the keys\nin
state_dictmatch the keys returned by this module's\n~torch.nn.Module.state_dict()function. Default:True\n - assign (bool, optional): whether to assign items in the state\ndictionary to their corresponding keys in the module instead\nof copying them inplace into the module's current parameters and buffers.\nWhen
False, the properties of the tensors in the current\nmodule are preserved while whenTrue, the properties of the\nTensors in the state dict are preserved.\nDefault:False\n
Returns:
\n\n\n\n\n\n
NamedTuplewithmissing_keysandunexpected_keysfields:\n * missing_keys is a list of str containing the missing keys\n * unexpected_keys is a list of str containing the unexpected keys
Note:
\n\n\n\n", "signature": "(self, state):", "funcdef": "def"}, {"fullname": "micro_sam.bioimageio.predictor_adaptor.PredictorAdaptor.forward", "modulename": "micro_sam.bioimageio.predictor_adaptor", "qualname": "PredictorAdaptor.forward", "kind": "function", "doc": "If a parameter or buffer is registered as
\nNoneand its corresponding key\n exists instate_dict,load_state_dict()will raise a\nRuntimeError.
Arguments:
\n\n- \n
- image: torch inputs of dimensions B x C x H x W \n
- box_prompts: box coordinates of dimensions B x OBJECTS x 4 \n
- point_prompts: point coordinates of dimension B x OBJECTS x POINTS x 2 \n
- point_labels: point labels of dimension B x OBJECTS x POINTS \n
- mask_prompts: mask prompts of dimension B x OBJECTS x 256 x 256 \n
- embeddings: precomputed image embeddings B x 256 x 64 x 64 \n
Returns:
\n", "signature": "(\tself,\timage: torch.Tensor,\tbox_prompts: Optional[torch.Tensor] = None,\tpoint_prompts: Optional[torch.Tensor] = None,\tpoint_labels: Optional[torch.Tensor] = None,\tmask_prompts: Optional[torch.Tensor] = None,\tembeddings: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation", "modulename": "micro_sam.evaluation", "kind": "module", "doc": "Functionality for evaluating Segment Anything models on microscopy data.
\n"}, {"fullname": "micro_sam.evaluation.evaluation", "modulename": "micro_sam.evaluation.evaluation", "kind": "module", "doc": "Evaluation functionality for segmentation predictions from micro_sam.evaluation.automatic_mask_generation\nand micro_sam.evaluation.inference.
Run evaluation for instance segmentation predictions.
\n\nArguments:
\n\n- \n
- gt_paths: The list of paths to ground-truth images. \n
- prediction_paths: The list of paths with the instance segmentations to evaluate. \n
- save_path: Optional path for saving the results. \n
- verbose: Whether to print the progress. \n
Returns:
\n\n\n\n", "signature": "(\tgt_paths: List[Union[str, os.PathLike]],\tprediction_paths: List[Union[str, os.PathLike]],\tsave_path: Union[str, os.PathLike, NoneType] = None,\tverbose: bool = True) -> pandas.core.frame.DataFrame:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.evaluation.run_evaluation_for_iterative_prompting", "modulename": "micro_sam.evaluation.evaluation", "qualname": "run_evaluation_for_iterative_prompting", "kind": "function", "doc": "A DataFrame that contains the evaluation results.
\n
Run evaluation for iterative prompt-based segmentation predictions.
\n\nArguments:
\n\n- \n
- gt_paths: The list of paths to ground-truth images. \n
- prediction_root: The folder with the iterative prompt-based instance segmentations to evaluate. \n
- experiment_folder: The folder where all the experiment results are stored. \n
- start_with_box_prompt: Whether to evaluate on experiments with iterative prompting starting with box. \n
Returns:
\n\n\n\n", "signature": "(\tgt_paths: List[Union[str, os.PathLike]],\tprediction_root: Union[os.PathLike, str],\texperiment_folder: Union[os.PathLike, str],\tstart_with_box_prompt: bool = False,\toverwrite_results: bool = False) -> pandas.core.frame.DataFrame:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.experiments", "modulename": "micro_sam.evaluation.experiments", "kind": "module", "doc": "A DataFrame that contains the evaluation results.
\n
Predefined experiment settings for experiments with different prompt strategies.
\n"}, {"fullname": "micro_sam.evaluation.experiments.ExperimentSetting", "modulename": "micro_sam.evaluation.experiments", "qualname": "ExperimentSetting", "kind": "variable", "doc": "\n", "default_value": "typing.Dict"}, {"fullname": "micro_sam.evaluation.experiments.full_experiment_settings", "modulename": "micro_sam.evaluation.experiments", "qualname": "full_experiment_settings", "kind": "function", "doc": "The full experiment settings.
\n\nArguments:
\n\n- \n
- use_boxes: Whether to run the experiments with or without boxes. \n
- positive_range: The different number of positive points that will be used.\nBy defaul the values are set to [1, 2, 4, 8, 16]. \n
- negative_range: The different number of negative points that will be used.\nBy defaul the values are set to [0, 1, 2, 4, 8, 16]. \n
Returns:
\n\n\n\n", "signature": "(\tuse_boxes: bool = False,\tpositive_range: Optional[List[int]] = None,\tnegative_range: Optional[List[int]] = None) -> List[Dict]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.experiments.default_experiment_settings", "modulename": "micro_sam.evaluation.experiments", "qualname": "default_experiment_settings", "kind": "function", "doc": "The list of experiment settings.
\n
The three default experiment settings.
\n\nFor the default experiments we use a single positive prompt,\ntwo positive and four negative prompts and box prompts.
\n\nReturns:
\n\n\n\n", "signature": "() -> List[Dict]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.experiments.get_experiment_setting_name", "modulename": "micro_sam.evaluation.experiments", "qualname": "get_experiment_setting_name", "kind": "function", "doc": "The list of experiment settings.
\n
Get the name for the given experiment setting.
\n\nArguments:
\n\n- \n
- setting: The experiment setting. \n
Returns:
\n\n\n\n", "signature": "(setting: Dict) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.inference", "modulename": "micro_sam.evaluation.inference", "kind": "module", "doc": "The name for this experiment setting.
\n
Inference with Segment Anything models and different prompt strategies.
\n"}, {"fullname": "micro_sam.evaluation.inference.precompute_all_embeddings", "modulename": "micro_sam.evaluation.inference", "qualname": "precompute_all_embeddings", "kind": "function", "doc": "Precompute all image embeddings.
\n\nTo enable running different inference tasks in parallel afterwards.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_paths: The image file paths. \n
- embedding_dir: The directory where the embeddings will be saved. \n
Precompute all point prompts.
\n\nTo enable running different inference tasks in parallel afterwards.
\n\nArguments:
\n\n- \n
- gt_paths: The file paths to the ground-truth segmentations. \n
- prompt_save_dir: The directory where the prompt files will be saved. \n
- prompt_settings: The settings for which the prompts will be computed. \n
Run segment anything inference for multiple images using prompts derived from groundtruth.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_paths: The image file paths. \n
- gt_paths: The ground-truth segmentation file paths. \n
- embedding_dir: The directory where the image embddings will be saved or are already saved. \n
- use_points: Whether to use point prompts. \n
- use_boxes: Whether to use box prompts \n
- n_positives: The number of positive point prompts that will be sampled. \n
- n_negativess: The number of negative point prompts that will be sampled. \n
- dilation: The dilation factor for the radius around the ground-truth object\naround which points will not be sampled. \n
- prompt_save_dir: The directory where point prompts will be saved or are already saved.\nThis enables running multiple experiments in a reproducible manner. \n
- batch_size: The batch size used for batched prediction. \n
Run segment anything inference for multiple images using prompts iteratively\n derived from model outputs and groundtruth
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_paths: The image file paths. \n
- gt_paths: The ground-truth segmentation file paths. \n
- embedding_dir: The directory where the image embeddings will be saved or are already saved. \n
- prediction_dir: The directory where the predictions from SegmentAnything will be saved per iteration. \n
- start_with_box_prompt: Whether to use the first prompt as bounding box or a single point \n
- dilation: The dilation factor for the radius around the ground-truth object\naround which points will not be sampled. \n
- batch_size: The batch size used for batched predictions. \n
- n_iterations: The number of iterations for iterative prompting. \n
- use_masks: Whether to make use of logits from previous prompt-based segmentation \n
Inference and evaluation for the automatic instance segmentation functionality.
\n"}, {"fullname": "micro_sam.evaluation.instance_segmentation.default_grid_search_values_amg", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "default_grid_search_values_amg", "kind": "function", "doc": "Default grid-search parameter for AMG-based instance segmentation.
\n\nReturn grid search values for the two most important parameters:
\n\n- \n
pred_iou_thresh, the threshold for keeping objects according to the IoU predicted by the model. \nstability_score_thresh, the theshold for keepong objects according to their stability. \n
Arguments:
\n\n- \n
- iou_thresh_values: The values for
pred_iou_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n - stability_score_values: The values for
stability_score_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n
Returns:
\n\n\n\n", "signature": "(\tiou_thresh_values: Optional[List[float]] = None,\tstability_score_values: Optional[List[float]] = None) -> Dict[str, List[float]]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.default_grid_search_values_instance_segmentation_with_decoder", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "default_grid_search_values_instance_segmentation_with_decoder", "kind": "function", "doc": "The values for grid search.
\n
Default grid-search parameter for decoder-based instance segmentation.
\n\nArguments:
\n\n- \n
- center_distance_threshold_values: The values for
center_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - boundary_distance_threshold_values: The values for
boundary_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - distance_smoothing_values: The values for
distance_smoothingused in the gridsearch.\nBy default values in the range from 1.0 to 2.0 with a stepsize of 0.1 will be used. \n - min_size_values: The values for
min_sizeused in the gridsearch.\nBy default the values 50, 100 and 200 are used. \n
Returns:
\n\n\n\n", "signature": "(\tcenter_distance_threshold_values: Optional[List[float]] = None,\tboundary_distance_threshold_values: Optional[List[float]] = None,\tdistance_smoothing_values: Optional[List[float]] = None,\tmin_size_values: Optional[List[float]] = None) -> Dict[str, List[float]]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.run_instance_segmentation_grid_search", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "run_instance_segmentation_grid_search", "kind": "function", "doc": "The values for grid search.
\n
Run grid search for automatic mask generation.
\n\nThe parameters and their respective value ranges for the grid search are specified via the\n'grid_search_values' argument. For example, to run a grid search over the parameters 'pred_iou_thresh'\nand 'stability_score_thresh', you can pass the following:
\n\ngrid_search_values = {\n \"pred_iou_thresh\": [0.6, 0.7, 0.8, 0.9],\n \"stability_score_thresh\": [0.6, 0.7, 0.8, 0.9],\n}\nAll combinations of the parameters will be checked.
\n\nYou can use the functions default_grid_search_values_instance_segmentation_with_decoder\nor default_grid_search_values_amg to get the default grid search parameters for the two\nrespective instance segmentation methods.
Arguments:
\n\n- \n
- segmenter: The class implementing the instance segmentation functionality. \n
- grid_search_values: The grid search values for parameters of the
generatefunction. \n - image_paths: The input images for the grid search. \n
- gt_paths: The ground-truth segmentation for the grid search. \n
- result_dir: Folder to cache the evaluation results per image. \n
- embedding_dir: Folder to cache the image embeddings. \n
- fixed_generate_kwargs: Fixed keyword arguments for the
generatemethod of the segmenter. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
- image_key: Key for loading the image data from a more complex file format like HDF5.\nIf not given a simple image format like tif is assumed. \n
- gt_key: Key for loading the ground-truth data from a more complex file format like HDF5.\nIf not given a simple image format like tif is assumed. \n
- rois: Region of interests to resetrict the evaluation to. \n
Run inference for automatic mask generation.
\n\nArguments:
\n\n- \n
- segmenter: The class implementing the instance segmentation functionality. \n
- image_paths: The input images. \n
- embedding_dir: Folder to cache the image embeddings. \n
- prediction_dir: Folder to save the predictions. \n
- generate_kwargs: The keyword arguments for the
generatemethod of the segmenter. \n
Evaluate gridsearch results.
\n\nArguments:
\n\n- \n
- result_dir: The folder with the gridsearch results. \n
- grid_search_parameters: The names for the gridsearch parameters. \n
- criterion: The metric to use for determining the best parameters. \n
Returns:
\n\n\n\n", "signature": "(\tresult_dir: Union[str, os.PathLike],\tgrid_search_parameters: List[str],\tcriterion: str = 'mSA') -> Tuple[Dict[str, Any], float]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.save_grid_search_best_params", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "save_grid_search_best_params", "kind": "function", "doc": "\n", "signature": "(best_kwargs, best_msa, grid_search_result_dir=None):", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.instance_segmentation.run_instance_segmentation_grid_search_and_inference", "modulename": "micro_sam.evaluation.instance_segmentation", "qualname": "run_instance_segmentation_grid_search_and_inference", "kind": "function", "doc": "The best parameter setting.\n The evaluation score for the best setting.
\n
Run grid search and inference for automatic mask generation.
\n\nPlease refer to the documentation of run_instance_segmentation_grid_search\nfor details on how to specify the grid search parameters.
Arguments:
\n\n- \n
- segmenter: The class implementing the instance segmentation functionality. \n
- grid_search_values: The grid search values for parameters of the
generatefunction. \n - val_image_paths: The input images for the grid search. \n
- val_gt_paths: The ground-truth segmentation for the grid search. \n
- test_image_paths: The input images for inference. \n
- embedding_dir: Folder to cache the image embeddings. \n
- prediction_dir: Folder to save the predictions. \n
- result_dir: Folder to cache the evaluation results per image. \n
- fixed_generate_kwargs: Fixed keyword arguments for the
generatemethod of the segmenter. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
Inference and evaluation for the LIVECell dataset and\nthe different cell lines contained in it.
\n"}, {"fullname": "micro_sam.evaluation.livecell.CELL_TYPES", "modulename": "micro_sam.evaluation.livecell", "qualname": "CELL_TYPES", "kind": "variable", "doc": "\n", "default_value": "['A172', 'BT474', 'BV2', 'Huh7', 'MCF7', 'SHSY5Y', 'SkBr3', 'SKOV3']"}, {"fullname": "micro_sam.evaluation.livecell.livecell_inference", "modulename": "micro_sam.evaluation.livecell", "qualname": "livecell_inference", "kind": "function", "doc": "Run inference for livecell with a fixed prompt setting.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segment anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- use_points: Whether to use point prompts. \n
- use_boxes: Whether to use box prompts. \n
- n_positives: The number of positive point prompts. \n
- n_negatives: The number of negative point prompts. \n
- prompt_folder: The folder where the prompts should be saved. \n
- predictor: The segment anything predictor. \n
Run precomputation of val and test image embeddings for livecell.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segmenta anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- n_val_per_cell_type: The number of validation images per cell type. \n
Run inference on livecell with iterative prompting setting.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segment anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- start_with_box_prompt: Whether to use the first prompt as bounding box or a single point. \n
- use_masks: Whether to make use of logits from previous prompt-based segmentation. \n
Run automatic mask generation grid-search and inference for livecell.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segmenta anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- iou_thresh_values: The values for
pred_iou_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n - stability_score_values: The values for
stability_score_threshused in the gridsearch.\nBy default values in the range from 0.6 to 0.9 with a stepsize of 0.025 will be used. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
- n_val_per_cell_type: The number of validation images per cell type. \n
Returns:
\n\n\n\n", "signature": "(\tcheckpoint: Union[str, os.PathLike],\tinput_folder: Union[str, os.PathLike],\tmodel_type: str,\texperiment_folder: Union[str, os.PathLike],\tiou_thresh_values: Optional[List[float]] = None,\tstability_score_values: Optional[List[float]] = None,\tverbose_gs: bool = False,\tn_val_per_cell_type: int = 25) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.livecell.run_livecell_instance_segmentation_with_decoder", "modulename": "micro_sam.evaluation.livecell", "qualname": "run_livecell_instance_segmentation_with_decoder", "kind": "function", "doc": "The path where the predicted images are stored.
\n
Run automatic mask generation grid-search and inference for livecell.
\n\nArguments:
\n\n- \n
- checkpoint: The segment anything model checkpoint. \n
- input_folder: The folder with the livecell data. \n
- model_type: The type of the segmenta anything model. \n
- experiment_folder: The folder where to save all data associated with the experiment. \n
- center_distance_threshold_values: The values for
center_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - boundary_distance_threshold_values: The values for
boundary_distance_thresholdused in the gridsearch.\nBy default values in the range from 0.3 to 0.7 with a stepsize of 0.1 will be used. \n - distance_smoothing_values: The values for
distance_smoothingused in the gridsearch.\nBy default values in the range from 1.0 to 2.0 with a stepsize of 0.1 will be used. \n - min_size_values: The values for
min_sizeused in the gridsearch.\nBy default the values 50, 100 and 200 are used. \n - verbose_gs: Whether to run the gridsearch for individual images in a verbose mode. \n
- n_val_per_cell_type: The number of validation images per cell type. \n
Returns:
\n\n\n\n", "signature": "(\tcheckpoint: Union[str, os.PathLike],\tinput_folder: Union[str, os.PathLike],\tmodel_type: str,\texperiment_folder: Union[str, os.PathLike],\tcenter_distance_threshold_values: Optional[List[float]] = None,\tboundary_distance_threshold_values: Optional[List[float]] = None,\tdistance_smoothing_values: Optional[List[float]] = None,\tmin_size_values: Optional[List[float]] = None,\tverbose_gs: bool = False,\tn_val_per_cell_type: int = 25) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.livecell.run_livecell_inference", "modulename": "micro_sam.evaluation.livecell", "qualname": "run_livecell_inference", "kind": "function", "doc": "The path where the predicted images are stored.
\n
Run LIVECell inference with command line tool.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.livecell.run_livecell_evaluation", "modulename": "micro_sam.evaluation.livecell", "qualname": "run_livecell_evaluation", "kind": "function", "doc": "Run LiveCELL evaluation with command line tool.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.model_comparison", "modulename": "micro_sam.evaluation.model_comparison", "kind": "module", "doc": "Functionality for qualitative comparison of Segment Anything models on microscopy data.
\n"}, {"fullname": "micro_sam.evaluation.model_comparison.generate_data_for_model_comparison", "modulename": "micro_sam.evaluation.model_comparison", "qualname": "generate_data_for_model_comparison", "kind": "function", "doc": "Generate samples for qualitative model comparison.
\n\nThis precomputes the input for model_comparison and model_comparison_with_napari.
Arguments:
\n\n- \n
- loader: The torch dataloader from which samples are drawn. \n
- output_folder: The folder where the samples will be saved. \n
- model_type1: The first model to use for comparison.\nThe value needs to be a valid model_type for
micro_sam.util.get_sam_model. \n - model_type2: The second model to use for comparison.\nThe value needs to be a valid model_type for
micro_sam.util.get_sam_model. \n - n_samples: The number of samples to draw from the dataloader. \n
- checkpoint1: Optional checkpoint for the first model. \n
- checkpoint2: Optional checkpoint for the second model. \n
Create images for a qualitative model comparision.
\n\nArguments:
\n\n- \n
- output_folder: The folder with the data precomputed by
generate_data_for_model_comparison. \n - n_images_per_sample: The number of images to generate per precomputed sample. \n
- min_size: The min size of ground-truth objects to take into account. \n
- plot_folder: The folder where to save the plots. If not given the plots will be displayed. \n
- point_radius: The radius of the point overlay. \n
- outline_dilation: The dilation factor of the outline overlay. \n
Use napari to display the qualtiative comparison results for two models.
\n\nArguments:
\n\n- \n
- output_folder: The folder with the data precomputed by
generate_data_for_model_comparison. \n - show_points: Whether to show the results for point or for box prompts. \n
Default grid-search parameters for multi-dimensional prompt-based instance segmentation.
\n\nArguments:
\n\n- \n
- iou_threshold_values: The values for
iou_thresholdused in the grid-search.\nBy default values in the range from 0.5 to 0.9 with a stepsize of 0.1 will be used. \n - projection_method_values: The values for
projectionmethod used in the grid-search.\nBy default the valuesmask,bounding_boxandpointsare used. \n - box_extension_values: The values for
box_extensionused in the grid-search.\nBy default values in the range from 0 to 0.25 with a stepsize of 0.025 will be used. \n
Returns:
\n\n\n\n", "signature": "(\tiou_threshold_values: Optional[List[float]] = None,\tprojection_method_values: Union[str, dict, NoneType] = None,\tbox_extension_values: Union[float, int, NoneType] = None) -> Dict[str, List]:", "funcdef": "def"}, {"fullname": "micro_sam.evaluation.multi_dimensional_segmentation.segment_slices_from_ground_truth", "modulename": "micro_sam.evaluation.multi_dimensional_segmentation", "qualname": "segment_slices_from_ground_truth", "kind": "function", "doc": "The values for grid search.
\n
Segment all objects in a volume by prompt-based segmentation in one slice per object.
\n\nThis function first segments each object in the respective specified slice using interactive\n(prompt-based) segmentation functionality. Then it segments the particular object in the\nremaining slices in the volume.
\n\nArguments:
\n\n- \n
- volume: The input volume. \n
- ground_truth: The label volume with instance segmentations. \n
- model_type: Choice of segment anything model. \n
- checkpoint_path: Path to the model checkpoint. \n
- embedding_path: Path to cache the computed embeddings. \n
- iou_threshold: The criterion to decide whether to link the objects in the consecutive slice's segmentation. \n
- projection: The projection (prompting) method to generate prompts for consecutive slices. \n
- box_extension: Extension factor for increasing the box size after projection. \n
- device: The selected device for computation. \n
- interactive_seg_mode: Method for guiding prompt-based instance segmentation. \n
- verbose: Whether to get the trace for projected segmentations. \n
- return_segmentation: Whether to return the segmented volume. \n
- min_size: The minimal size for evaluating an object in the ground-truth.\nThe size is measured within the central slice. \n
Run grid search for prompt-based multi-dimensional instance segmentation.
\n\nThe parameters and their respective value ranges for the grid search are specified via the\ngrid_search_values argument. For example, to run a grid search over the parameters iou_threshold,\nprojection and box_extension, you can pass the following:
grid_search_values = {\n \"iou_threshold\": [0.5, 0.6, 0.7, 0.8, 0.9],\n \"projection\": [\"mask\", \"bounding_box\", \"points\"],\n \"box_extension\": [0, 0.1, 0.2, 0.3, 0.4, 0,5],\n}\nAll combinations of the parameters will be checked.\nIf passed None, the function default_grid_search_values_multi_dimensional_segmentation is used\nto get the default grid search parameters for the instance segmentation method.
Arguments:
\n\n- \n
- volume: The input volume. \n
- ground_truth: The label volume with instance segmentations. \n
- model_type: Choice of segment anything model. \n
- checkpoint_path: Path to the model checkpoint. \n
- embedding_path: Path to cache the computed embeddings. \n
- result_path: Path to save the grid search results. \n
- interactive_seg_mode: Method for guiding prompt-based instance segmentation. \n
- verbose: Whether to get the trace for projected segmentations. \n
- grid_search_values: The grid search values for parameters of the
segment_slices_from_ground_truthfunction. \n - min_size: The minimal size for evaluating an object in the ground-truth.\nThe size is measured within the central slice. \n
Run batched inference for input prompts.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- image: The input image. \n
- batch_size: The batch size to use for inference. \n
- boxes: The box prompts. Array of shape N_PROMPTS x 4.\nThe bounding boxes are represented by [MIN_X, MIN_Y, MAX_X, MAX_Y]. \n
- points: The point prompt coordinates. Array of shape N_PROMPTS x 1 x 2.\nThe points are represented by their coordinates [X, Y], which are given\nin the last dimension. \n
- point_labels: The point prompt labels. Array of shape N_PROMPTS x 1.\nThe labels are either 0 (negative prompt) or 1 (positive prompt). \n
- multimasking: Whether to predict with 3 or 1 mask. \n
- embedding_path: Cache path for the image embeddings. \n
- return_instance_segmentation: Whether to return a instance segmentation\nor the individual mask data. \n
- segmentation_ids: Fixed segmentation ids to assign to the masks\nderived from the prompts. \n
- reduce_multimasking: Whether to choose the most likely masks with\nhighest ious from multimasking \n
- logits_masks: The logits masks. Array of shape N_PROMPTS x 1 x 256 x 256.\nWhether to use the logits masks from previous segmentation. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\timage: numpy.ndarray,\tbatch_size: int,\tboxes: Optional[numpy.ndarray] = None,\tpoints: Optional[numpy.ndarray] = None,\tpoint_labels: Optional[numpy.ndarray] = None,\tmultimasking: bool = False,\tembedding_path: Union[str, os.PathLike, NoneType] = None,\treturn_instance_segmentation: bool = True,\tsegmentation_ids: Optional[list] = None,\treduce_multimasking: bool = True,\tlogits_masks: Optional[torch.Tensor] = None):", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation", "modulename": "micro_sam.instance_segmentation", "kind": "module", "doc": "The predicted segmentation masks.
\n
Automated instance segmentation functionality.\nThe classes implemented here extend the automatic instance segmentation from Segment Anything:\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html
\n"}, {"fullname": "micro_sam.instance_segmentation.mask_data_to_segmentation", "modulename": "micro_sam.instance_segmentation", "qualname": "mask_data_to_segmentation", "kind": "function", "doc": "Convert the output of the automatic mask generation to an instance segmentation.
\n\nArguments:
\n\n- \n
- masks: The outputs generated by AutomaticMaskGenerator or EmbeddingMaskGenerator.\nOnly supports output_mode=binary_mask. \n
- with_background: Whether the segmentation has background. If yes this function assures that the largest\nobject in the output will be mapped to zero (the background value). \n
- min_object_size: The minimal size of an object in pixels. \n
- max_object_size: The maximal size of an object in pixels. \n
Returns:
\n\n\n\n", "signature": "(\tmasks: List[Dict[str, Any]],\twith_background: bool,\tmin_object_size: int = 0,\tmax_object_size: Optional[int] = None) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.AMGBase", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase", "kind": "class", "doc": "The instance segmentation.
\n
Base class for the automatic mask generators.
\n", "bases": "abc.ABC"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.is_initialized", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.is_initialized", "kind": "variable", "doc": "Whether the mask generator has already been initialized.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.crop_list", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.crop_list", "kind": "variable", "doc": "The list of mask data after initialization.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.crop_boxes", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.crop_boxes", "kind": "variable", "doc": "The list of crop boxes.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.original_size", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.original_size", "kind": "variable", "doc": "The original image size.
\n"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.get_state", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.get_state", "kind": "function", "doc": "Get the initialized state of the mask generator.
\n\nReturns:
\n\n\n\n", "signature": "(self) -> Dict[str, Any]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.AMGBase.set_state", "modulename": "micro_sam.instance_segmentation", "qualname": "AMGBase.set_state", "kind": "function", "doc": "State of the mask generator.
\n
Set the state of the mask generator.
\n\nArguments:
\n\n- \n
- state: The state of the mask generator, e.g. from serialized state. \n
Clear the state of the mask generator.
\n", "signature": "(self):", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.AutomaticMaskGenerator", "modulename": "micro_sam.instance_segmentation", "qualname": "AutomaticMaskGenerator", "kind": "class", "doc": "Generates an instance segmentation without prompts, using a point grid.
\n\nThis class implements the same logic as\nhttps://github.com/facebookresearch/segment-anything/blob/main/segment_anything/automatic_mask_generator.py\nIt decouples the computationally expensive steps of generating masks from the cheap post-processing operation\nto filter these masks to enable grid search and interactively changing the post-processing.
\n\nUse this class as follows:
\n\namg = AutomaticMaskGenerator(predictor)\namg.initialize(image) # Initialize the masks, this takes care of all expensive computations.\nmasks = amg.generate(pred_iou_thresh=0.8) # Generate the masks. This is fast and enables testing parameters\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- points_per_side: The number of points to be sampled along one side of the image.\nIf None,
point_gridsmust provide explicit point sampling. \n - points_per_batch: The number of points run simultaneously by the model.\nHigher numbers may be faster but use more GPU memory. \n
- crop_n_layers: If >0, the mask prediction will be run again on crops of the image. \n
- crop_overlap_ratio: Sets the degree to which crops overlap. \n
- crop_n_points_downscale_factor: How the number of points is downsampled when predicting with crops. \n
- point_grids: A lisst over explicit grids of points used for sampling masks.\nNormalized to [0, 1] with respect to the image coordinate system. \n
- stability_score_offset: The amount to shift the cutoff when calculating the stability score. \n
Initialize image embeddings and masks for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - verbose: Whether to print computation progress. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Generate instance segmentation for the currently initialized image.
\n\nArguments:
\n\n- \n
- pred_iou_thresh: Filter threshold in [0, 1], using the mask quality predicted by the model. \n
- stability_score_thresh: Filter threshold in [0, 1], using the stability of the mask\nunder changes to the cutoff used to binarize the model prediction. \n
- box_nms_thresh: The IoU threshold used by nonmax suppression to filter duplicate masks. \n
- crop_nms_thresh: The IoU threshold used by nonmax suppression to filter duplicate masks between crops. \n
- min_mask_region_area: Minimal size for the predicted masks. \n
- output_mode: The form masks are returned in. \n
Returns:
\n\n\n\n", "signature": "(\tself,\tpred_iou_thresh: float = 0.88,\tstability_score_thresh: float = 0.95,\tbox_nms_thresh: float = 0.7,\tcrop_nms_thresh: float = 0.7,\tmin_mask_region_area: int = 0,\toutput_mode: str = 'binary_mask') -> List[Dict[str, Any]]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.TiledAutomaticMaskGenerator", "modulename": "micro_sam.instance_segmentation", "qualname": "TiledAutomaticMaskGenerator", "kind": "class", "doc": "The instance segmentation masks.
\n
Generates an instance segmentation without prompts, using a point grid.
\n\nImplements the same functionality as AutomaticMaskGenerator but for tiled embeddings.
Arguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- points_per_side: The number of points to be sampled along one side of the image.\nIf None,
point_gridsmust provide explicit point sampling. \n - points_per_batch: The number of points run simultaneously by the model.\nHigher numbers may be faster but use more GPU memory. \n
- point_grids: A lisst over explicit grids of points used for sampling masks.\nNormalized to [0, 1] with respect to the image coordinate system. \n
- stability_score_offset: The amount to shift the cutoff when calculating the stability score. \n
Initialize image embeddings and masks for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - tile_shape: The tile shape for embedding prediction. \n
- halo: The overlap of between tiles. \n
- verbose: Whether to print computation progress. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Adapter to contain the UNETR decoder in a single module.
\n\nTo apply the decoder on top of pre-computed embeddings for\nthe segmentation functionality.\nSee also: https://github.com/constantinpape/torch-em/blob/main/torch_em/model/unetr.py
\n", "bases": "torch.nn.modules.module.Module"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.__init__", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.__init__", "kind": "function", "doc": "Initializes internal Module state, shared by both nn.Module and ScriptModule.
\n", "signature": "(unetr)"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.base", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.base", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.out_conv", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.out_conv", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv_out", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv_out", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.decoder_head", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.decoder_head", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.final_activation", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.final_activation", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.postprocess_masks", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.postprocess_masks", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.decoder", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.decoder", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv1", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv1", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv2", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv2", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv3", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv3", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.deconv4", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.deconv4", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.instance_segmentation.DecoderAdapter.forward", "modulename": "micro_sam.instance_segmentation", "qualname": "DecoderAdapter.forward", "kind": "function", "doc": "Defines the computation performed at every call.
\n\nShould be overridden by all subclasses.
\n\nAlthough the recipe for forward pass needs to be defined within\nthis function, one should call the Module instance afterwards\ninstead of this since the former takes care of running the\nregistered hooks while the latter silently ignores them.
Get UNETR model for automatic instance segmentation.
\n\nArguments:
\n\n- \n
- image_encoder: The image encoder of the SAM model.\nThis is used as encoder by the UNETR too. \n
- decoder_state: Optional decoder state to initialize the weights\nof the UNETR decoder. \n
- device: The device. \n
Returns:
\n\n\n\n", "signature": "(\timage_encoder: torch.nn.modules.module.Module,\tdecoder_state: Optional[collections.OrderedDict[str, torch.Tensor]] = None,\tdevice: Union[str, torch.device, NoneType] = None) -> torch.nn.modules.module.Module:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.get_decoder", "modulename": "micro_sam.instance_segmentation", "qualname": "get_decoder", "kind": "function", "doc": "The UNETR model.
\n
Get decoder to predict outputs for automatic instance segmentation
\n\nArguments:
\n\n- \n
- image_encoder: The image encoder of the SAM model. \n
- decoder_state: State to initialize the weights of the UNETR decoder. \n
- device: The device. \n
Returns:
\n\n\n\n", "signature": "(\timage_encoder: torch.nn.modules.module.Module,\tdecoder_state: collections.OrderedDict[str, torch.Tensor],\tdevice: Union[str, torch.device, NoneType] = None) -> micro_sam.instance_segmentation.DecoderAdapter:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.get_predictor_and_decoder", "modulename": "micro_sam.instance_segmentation", "qualname": "get_predictor_and_decoder", "kind": "function", "doc": "The decoder for instance segmentation.
\n
Load the SAM model (predictor) and instance segmentation decoder.
\n\nThis requires a checkpoint that contains the state for both predictor\nand decoder.
\n\nArguments:
\n\n- \n
- model_type: The type of the image encoder used in the SAM model. \n
- checkpoint_path: Path to the checkpoint from which to load the data. \n
- device: The device. \n
Returns:
\n\n\n\n", "signature": "(\tmodel_type: str,\tcheckpoint_path: Union[str, os.PathLike],\tdevice: Union[str, torch.device, NoneType] = None) -> Tuple[segment_anything.predictor.SamPredictor, micro_sam.instance_segmentation.DecoderAdapter]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder", "kind": "class", "doc": "The SAM predictor.\n The decoder for instance segmentation.
\n
Generates an instance segmentation without prompts, using a decoder.
\n\nImplements the same interface as AutomaticMaskGenerator.
Use this class as follows:
\n\nsegmenter = InstanceSegmentationWithDecoder(predictor, decoder)\nsegmenter.initialize(image) # Predict the image embeddings and decoder outputs.\nmasks = segmenter.generate(center_distance_threshold=0.75) # Generate the instance segmentation.\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- decoder: The decoder to predict intermediate representations\nfor instance segmentation. \n
Whether the mask generator has already been initialized.
\n"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder.initialize", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder.initialize", "kind": "function", "doc": "Initialize image embeddings and decoder predictions for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - verbose: Whether to be verbose. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Generate instance segmentation for the currently initialized image.
\n\nArguments:
\n\n- \n
- center_distance_threshold: Center distance predictions below this value will be\nused to find seeds (intersected with thresholded boundary distance predictions). \n
- boundary_distance_threshold: Boundary distance predictions below this value will be\nused to find seeds (intersected with thresholded center distance predictions). \n
- foreground_smoothing: Sigma value for smoothing the foreground predictions, to avoid\ncheckerboard artifacts in the prediction. \n
- foreground_threshold: Foreground predictions above this value will be used as foreground mask. \n
- distance_smoothing: Sigma value for smoothing the distance predictions. \n
- min_size: Minimal object size in the segmentation result. \n
- output_mode: The form masks are returned in. Pass None to directly return the instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\tself,\tcenter_distance_threshold: float = 0.5,\tboundary_distance_threshold: float = 0.5,\tforeground_threshold: float = 0.5,\tforeground_smoothing: float = 1.0,\tdistance_smoothing: float = 1.6,\tmin_size: int = 0,\toutput_mode: Optional[str] = 'binary_mask') -> List[Dict[str, Any]]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder.get_state", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder.get_state", "kind": "function", "doc": "The instance segmentation masks.
\n
Get the initialized state of the instance segmenter.
\n\nReturns:
\n\n\n\n", "signature": "(self) -> Dict[str, Any]:", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.InstanceSegmentationWithDecoder.set_state", "modulename": "micro_sam.instance_segmentation", "qualname": "InstanceSegmentationWithDecoder.set_state", "kind": "function", "doc": "Instance segmentation state.
\n
Set the state of the instance segmenter.
\n\nArguments:
\n\n- \n
- state: The instance segmentation state \n
Clear the state of the instance segmenter.
\n", "signature": "(self):", "funcdef": "def"}, {"fullname": "micro_sam.instance_segmentation.TiledInstanceSegmentationWithDecoder", "modulename": "micro_sam.instance_segmentation", "qualname": "TiledInstanceSegmentationWithDecoder", "kind": "class", "doc": "Same as InstanceSegmentationWithDecoder but for tiled image embeddings.
Initialize image embeddings and decoder predictions for an image.
\n\nArguments:
\n\n- \n
- image: The input image, volume or timeseries. \n
- image_embeddings: Optional precomputed image embeddings.\nSee
util.precompute_image_embeddingsfor details. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - verbose: Dummy input to be compatible with other function signatures. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Get the automatic mask generator class.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- is_tiled: Whether tiled embeddings are used. \n
- decoder: Decoder to predict instacne segmmentation. \n
- kwargs: The keyword arguments for the amg class. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tis_tiled: bool,\tdecoder: Optional[torch.nn.modules.module.Module] = None,\t**kwargs) -> Union[micro_sam.instance_segmentation.AMGBase, micro_sam.instance_segmentation.InstanceSegmentationWithDecoder]:", "funcdef": "def"}, {"fullname": "micro_sam.multi_dimensional_segmentation", "modulename": "micro_sam.multi_dimensional_segmentation", "kind": "module", "doc": "The automatic mask generator.
\n
Multi-dimensional segmentation with segment anything.
\n"}, {"fullname": "micro_sam.multi_dimensional_segmentation.PROJECTION_MODES", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "PROJECTION_MODES", "kind": "variable", "doc": "\n", "default_value": "('box', 'mask', 'points', 'points_and_mask', 'single_point')"}, {"fullname": "micro_sam.multi_dimensional_segmentation.segment_mask_in_volume", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "segment_mask_in_volume", "kind": "function", "doc": "Segment an object mask in in volumetric data.
\n\nArguments:
\n\n- \n
- segmentation: The initial segmentation for the object. \n
- predictor: The segment anything predictor. \n
- image_embeddings: The precomputed image embeddings for the volume. \n
- segmented_slices: List of slices for which this object has already been segmented. \n
- stop_lower: Whether to stop at the lowest segmented slice. \n
- stop_upper: Wheter to stop at the topmost segmented slice. \n
- iou_threshold: The IOU threshold for continuing segmentation across 3d. \n
- projection: The projection method to use. One of 'box', 'mask', 'points', 'points_and_mask' or 'single point'.\nPass a dictionary to choose the excact combination of projection modes. \n
- update_progress: Callback to update an external progress bar. \n
- box_extension: Extension factor for increasing the box size after projection. \n
- verbose: Whether to print details about the segmentation steps. \n
Returns:
\n\n\n\n", "signature": "(\tsegmentation: numpy.ndarray,\tpredictor: segment_anything.predictor.SamPredictor,\timage_embeddings: Dict[str, Any],\tsegmented_slices: numpy.ndarray,\tstop_lower: bool,\tstop_upper: bool,\tiou_threshold: float,\tprojection: Union[str, dict],\tupdate_progress: Optional[<built-in function callable>] = None,\tbox_extension: float = 0.0,\tverbose: bool = False) -> Tuple[numpy.ndarray, Tuple[int, int]]:", "funcdef": "def"}, {"fullname": "micro_sam.multi_dimensional_segmentation.merge_instance_segmentation_3d", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "merge_instance_segmentation_3d", "kind": "function", "doc": "Array with the volumetric segmentation.\n Tuple with the first and last segmented slice.
\n
Merge stacked 2d instance segmentations into a consistent 3d segmentation.
\n\nSolves a multicut problem based on the overlap of objects to merge across z.
\n\nArguments:
\n\n- \n
- slice_segmentation: The stacked segmentation across the slices.\nWe assume that the segmentation is labeled consecutive across z. \n
- beta: The bias term for the multicut. Higher values lead to a larger\ndegree of over-segmentation and vice versa. \n
- with_background: Whether this is a segmentation problem with background.\nIn that case all edges connecting to the background are set to be repulsive. \n
- gap_closing: If given, gaps in the segmentation are closed with a binary closing\noperation. The value is used to determine the number of iterations for the closing. \n
- min_z_extent: Require a minimal extent in z for the segmented objects.\nThis can help to prevent segmentation artifacts. \n
- verbose: Verbosity flag. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Returns:
\n\n\n\n", "signature": "(\tslice_segmentation: numpy.ndarray,\tbeta: float = 0.5,\twith_background: bool = True,\tgap_closing: Optional[int] = None,\tmin_z_extent: Optional[int] = None,\tverbose: bool = True,\tpbar_init: Optional[<built-in function callable>] = None,\tpbar_update: Optional[<built-in function callable>] = None) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.multi_dimensional_segmentation.automatic_3d_segmentation", "modulename": "micro_sam.multi_dimensional_segmentation", "qualname": "automatic_3d_segmentation", "kind": "function", "doc": "The merged segmentation.
\n
Segment volume in 3d.
\n\nFirst segments slices individually in 2d and then merges them across 3d\nbased on overlap of objects between slices.
\n\nArguments:
\n\n- \n
- volume: The input volume. \n
- predictor: The SAM model. \n
- segmentor: The instance segmentation class. \n
- embedding_path: The path to save pre-computed embeddings. \n
- with_background: Whether the segmentation has background. \n
- gap_closing: If given, gaps in the segmentation are closed with a binary closing\noperation. The value is used to determine the number of iterations for the closing. \n
- min_z_extent: Require a minimal extent in z for the segmented objects.\nThis can help to prevent segmentation artifacts. \n
- verbose: Verbosity flag. \n
- kwargs: Keyword arguments for the 'generate' method of the 'segmentor'. \n
Returns:
\n\n\n\n", "signature": "(\tvolume: numpy.ndarray,\tpredictor: segment_anything.predictor.SamPredictor,\tsegmentor: micro_sam.instance_segmentation.AMGBase,\tembedding_path: Union[os.PathLike, str, NoneType] = None,\twith_background: bool = True,\tgap_closing: Optional[int] = None,\tmin_z_extent: Optional[int] = None,\tverbose: bool = True,\t**kwargs) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.precompute_state", "modulename": "micro_sam.precompute_state", "kind": "module", "doc": "The segmentation.
\n
Precompute image embeddings and automatic mask generator state for image data.
\n"}, {"fullname": "micro_sam.precompute_state.cache_amg_state", "modulename": "micro_sam.precompute_state", "qualname": "cache_amg_state", "kind": "function", "doc": "Compute and cache or load the state for the automatic mask generator.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- raw: The image data. \n
- image_embeddings: The image embeddings. \n
- save_path: The embedding save path. The AMG state will be stored in 'save_path/amg_state.pickle'. \n
- verbose: Whether to run the computation verbose. \n
- i: The index for which to cache the state. \n
- kwargs: The keyword arguments for the amg class. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\traw: numpy.ndarray,\timage_embeddings: Dict[str, Any],\tsave_path: Union[str, os.PathLike],\tverbose: bool = True,\ti: Optional[int] = None,\t**kwargs) -> micro_sam.instance_segmentation.AMGBase:", "funcdef": "def"}, {"fullname": "micro_sam.precompute_state.cache_is_state", "modulename": "micro_sam.precompute_state", "qualname": "cache_is_state", "kind": "function", "doc": "The automatic mask generator class with the cached state.
\n
Compute and cache or load the state for the automatic mask generator.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- decoder: The instance segmentation decoder. \n
- raw: The image data. \n
- image_embeddings: The image embeddings. \n
- save_path: The embedding save path. The AMG state will be stored in 'save_path/amg_state.pickle'. \n
- verbose: Whether to run the computation verbose. \n
- i: The index for which to cache the state. \n
- skip_load: Skip loading the state if it is precomputed. \n
- kwargs: The keyword arguments for the amg class. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tdecoder: torch.nn.modules.module.Module,\traw: numpy.ndarray,\timage_embeddings: Dict[str, Any],\tsave_path: Union[str, os.PathLike],\tverbose: bool = True,\ti: Optional[int] = None,\tskip_load: bool = False,\t**kwargs) -> Optional[micro_sam.instance_segmentation.AMGBase]:", "funcdef": "def"}, {"fullname": "micro_sam.precompute_state.precompute_state", "modulename": "micro_sam.precompute_state", "qualname": "precompute_state", "kind": "function", "doc": "The instance segmentation class with the cached state.
\n
Precompute the image embeddings and other optional state for the input image(s).
\n\nArguments:
\n\n- \n
- input_path: The input image file(s). Can either be a single image file (e.g. tif or png),\na container file (e.g. hdf5 or zarr) or a folder with images files.\nIn case of a container file the argument
keymust be given. In case of a folder\nit can be given to provide a glob pattern to subselect files from the folder. \n - output_path: The output path were the embeddings and other state will be saved. \n
- pattern: Glob pattern to select files in a folder. The embeddings will be computed\nfor each of these files. To select all files in a folder pass \"*\". \n
- model_type: The SegmentAnything model to use. Will use the standard vit_h model by default. \n
- checkpoint_path: Path to a checkpoint for a custom model. \n
- key: The key to the input file. This is needed for contaner files (e.g. hdf5 or zarr)\nor to load several images as 3d volume. Provide a glob pattern, e.g. \"*.tif\", for this case. \n
- ndim: The dimensionality of the data. \n
- tile_shape: Shape of tiles for tiled prediction. By default prediction is run without tiling. \n
- halo: Overlap of the tiles for tiled prediction. \n
- precompute_amg_state: Whether to precompute the state for automatic instance segmentation\nin addition to the image embeddings. \n
Functions for prompt-based segmentation with Segment Anything.
\n"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_points", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_points", "kind": "function", "doc": "Segmentation from point prompts.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- points: The point prompts given in the image coordinate system. \n
- labels: The labels (positive or negative) associated with the points. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
- use_best_multimask: Whether to use multimask output and then choose the best mask.\nBy default this is used for a single positive point and not otherwise. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tpoints: numpy.ndarray,\tlabels: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tmultimask_output: bool = False,\treturn_all: bool = False,\tuse_best_multimask: Optional[bool] = None):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_mask", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_mask", "kind": "function", "doc": "The binary segmentation mask.
\n
Segmentation from a mask prompt.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- mask: The mask used to derive prompts. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- use_box: Whether to derive the bounding box prompt from the mask. \n
- use_mask: Whether to use the mask itself as prompt. \n
- use_points: Whether to derive point prompts from the mask. \n
- original_size: Full image shape. Use this if the mask that is being passed\ndownsampled compared to the original image. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
- box_extension: Relative factor used to enlarge the bounding box prompt. \n
- box: Precomputed bounding box. \n
- points: Precomputed point prompts. \n
- labels: Positive/negative labels corresponding to the point prompts. \n
- use_single_point: Whether to derive just a single point from the mask.\nIn case use_points is true. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tmask: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tuse_box: bool = True,\tuse_mask: bool = True,\tuse_points: bool = False,\toriginal_size: Optional[Tuple[int, ...]] = None,\tmultimask_output: bool = False,\treturn_all: bool = False,\treturn_logits: bool = False,\tbox_extension: float = 0.0,\tbox: Optional[numpy.ndarray] = None,\tpoints: Optional[numpy.ndarray] = None,\tlabels: Optional[numpy.ndarray] = None,\tuse_single_point: bool = False):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_box", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_box", "kind": "function", "doc": "The binary segmentation mask.
\n
Segmentation from a box prompt.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- box: The box prompt. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
- box_extension: Relative factor used to enlarge the bounding box prompt. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tbox: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tmultimask_output: bool = False,\treturn_all: bool = False,\tbox_extension: float = 0.0):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_based_segmentation.segment_from_box_and_points", "modulename": "micro_sam.prompt_based_segmentation", "qualname": "segment_from_box_and_points", "kind": "function", "doc": "The binary segmentation mask.
\n
Segmentation from a box prompt and point prompts.
\n\nArguments:
\n\n- \n
- predictor: The segment anything predictor. \n
- box: The box prompt. \n
- points: The point prompts, given in the image coordinates system. \n
- labels: The point labels, either positive or negative. \n
- image_embeddings: Optional precomputed image embeddings.\n Has to be passed if the predictor is not yet initialized.\ni: Index for the image data. Required if the input data has three spatial dimensions\n or a time dimension and two spatial dimensions. \n
- multimask_output: Whether to return multiple or just a single mask. \n
- return_all: Whether to return the score and logits in addition to the mask. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: segment_anything.predictor.SamPredictor,\tbox: numpy.ndarray,\tpoints: numpy.ndarray,\tlabels: numpy.ndarray,\timage_embeddings: Optional[Dict[str, Any]] = None,\ti: Optional[int] = None,\tmultimask_output: bool = False,\treturn_all: bool = False):", "funcdef": "def"}, {"fullname": "micro_sam.prompt_generators", "modulename": "micro_sam.prompt_generators", "kind": "module", "doc": "The binary segmentation mask.
\n
Classes for generating prompts from ground-truth segmentation masks.\nFor training or evaluation of prompt-based segmentation.
\n"}, {"fullname": "micro_sam.prompt_generators.PromptGeneratorBase", "modulename": "micro_sam.prompt_generators", "qualname": "PromptGeneratorBase", "kind": "class", "doc": "PromptGeneratorBase is an interface to implement specific prompt generators.
\n"}, {"fullname": "micro_sam.prompt_generators.PointAndBoxPromptGenerator", "modulename": "micro_sam.prompt_generators", "qualname": "PointAndBoxPromptGenerator", "kind": "class", "doc": "Generate point and/or box prompts from an instance segmentation.
\n\nYou can use this class to derive prompts from an instance segmentation, either for\nevaluation purposes or for training Segment Anything on custom data.\nIn order to use this generator you need to precompute the bounding boxes and center\ncoordiantes of the instance segmentation, using e.g. util.get_centers_and_bounding_boxes.
Here's an example for how to use this class:
\n\n# Initialize generator for 1 positive and 4 negative point prompts.\nprompt_generator = PointAndBoxPromptGenerator(1, 4, dilation_strength=8)\n\n# Precompute the bounding boxes for the given segmentation\nbounding_boxes, _ = util.get_centers_and_bounding_boxes(segmentation)\n\n# generate point prompts for the objects with ids 1, 2 and 3\nseg_ids = (1, 2, 3)\nobject_mask = np.stack([segmentation == seg_id for seg_id in seg_ids])[:, None]\nthis_bounding_boxes = [bounding_boxes[seg_id] for seg_id in seg_ids]\npoint_coords, point_labels, _, _ = prompt_generator(object_mask, this_bounding_boxes)\nArguments:
\n\n- \n
- n_positive_points: The number of positive point prompts to generate per mask. \n
- n_negative_points: The number of negative point prompts to generate per mask. \n
- dilation_strength: The factor by which the mask is dilated before generating prompts. \n
- get_point_prompts: Whether to generate point prompts. \n
- get_box_prompts: Whether to generate box prompts. \n
Generate point prompts from an instance segmentation iteratively.
\n", "bases": "PromptGeneratorBase"}, {"fullname": "micro_sam.sam_annotator", "modulename": "micro_sam.sam_annotator", "kind": "module", "doc": "The interactive annotation tools.
\n"}, {"fullname": "micro_sam.sam_annotator.annotator_2d", "modulename": "micro_sam.sam_annotator.annotator_2d", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.annotator_2d.Annotator2d", "modulename": "micro_sam.sam_annotator.annotator_2d", "qualname": "Annotator2d", "kind": "class", "doc": "Base class for micro_sam annotation plugins.
\n\nImplements the logic for the 2d, 3d and tracking annotator.\nThe annotators differ in their data dimensionality and the widgets.
\n", "bases": "micro_sam.sam_annotator._annotator._AnnotatorBase"}, {"fullname": "micro_sam.sam_annotator.annotator_2d.Annotator2d.__init__", "modulename": "micro_sam.sam_annotator.annotator_2d", "qualname": "Annotator2d.__init__", "kind": "function", "doc": "Create the annotator GUI.
\n\nArguments:
\n\n- \n
- viewer: The napari viewer. \n
- ndim: The number of spatial dimension of the image data (2 or 3). \n
Start the 2d annotation tool for a given image.
\n\nArguments:
\n\n- \n
- image: The image data. \n
- embedding_path: Filepath where to save the embeddings. \n
- segmentation_result: An initial segmentation to load.\nThis can be used to correct segmentations with Segment Anything or to save and load progress.\nThe segmentation will be loaded as the 'committed_objects' layer. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- precompute_amg_state: Whether to precompute the state for automatic mask generation.\nThis will take more time when precomputing embeddings, but will then make\nautomatic mask generation much faster. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- device: The computational device to use for the SAM model. \n
- prefer_decoder: Whether to use decoder based instance segmentation if\nthe model used has an additional decoder for instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\timage: numpy.ndarray,\tembedding_path: Optional[str] = None,\tsegmentation_result: Optional[numpy.ndarray] = None,\tmodel_type: str = 'vit_l',\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\treturn_viewer: bool = False,\tviewer: Optional[napari.viewer.Viewer] = None,\tprecompute_amg_state: bool = False,\tcheckpoint_path: Optional[str] = None,\tdevice: Union[str, torch.device, NoneType] = None,\tprefer_decoder: bool = True) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.annotator_3d", "modulename": "micro_sam.sam_annotator.annotator_3d", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.annotator_3d.Annotator3d", "modulename": "micro_sam.sam_annotator.annotator_3d", "qualname": "Annotator3d", "kind": "class", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Base class for micro_sam annotation plugins.
\n\nImplements the logic for the 2d, 3d and tracking annotator.\nThe annotators differ in their data dimensionality and the widgets.
\n", "bases": "micro_sam.sam_annotator._annotator._AnnotatorBase"}, {"fullname": "micro_sam.sam_annotator.annotator_3d.Annotator3d.__init__", "modulename": "micro_sam.sam_annotator.annotator_3d", "qualname": "Annotator3d.__init__", "kind": "function", "doc": "Create the annotator GUI.
\n\nArguments:
\n\n- \n
- viewer: The napari viewer. \n
- ndim: The number of spatial dimension of the image data (2 or 3). \n
Start the 3d annotation tool for a given image volume.
\n\nArguments:
\n\n- \n
- image: The volumetric image data. \n
- embedding_path: Filepath for saving the precomputed embeddings. \n
- segmentation_result: An initial segmentation to load.\nThis can be used to correct segmentations with Segment Anything or to save and load progress.\nThe segmentation will be loaded as the 'committed_objects' layer. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- precompute_amg_state: Whether to precompute the state for automatic mask generation.\nThis will take more time when precomputing embeddings, but will then make\nautomatic mask generation much faster. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- device: The computational device to use for the SAM model. \n
- prefer_decoder: Whether to use decoder based instance segmentation if\nthe model used has an additional decoder for instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\timage: numpy.ndarray,\tembedding_path: Optional[str] = None,\tsegmentation_result: Optional[numpy.ndarray] = None,\tmodel_type: str = 'vit_l',\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\treturn_viewer: bool = False,\tviewer: Optional[napari.viewer.Viewer] = None,\tprecompute_amg_state: bool = False,\tcheckpoint_path: Optional[str] = None,\tdevice: Union[str, torch.device, NoneType] = None,\tprefer_decoder: bool = True) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.annotator_tracking", "modulename": "micro_sam.sam_annotator.annotator_tracking", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.annotator_tracking.AnnotatorTracking", "modulename": "micro_sam.sam_annotator.annotator_tracking", "qualname": "AnnotatorTracking", "kind": "class", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Base class for micro_sam annotation plugins.
\n\nImplements the logic for the 2d, 3d and tracking annotator.\nThe annotators differ in their data dimensionality and the widgets.
\n", "bases": "micro_sam.sam_annotator._annotator._AnnotatorBase"}, {"fullname": "micro_sam.sam_annotator.annotator_tracking.AnnotatorTracking.__init__", "modulename": "micro_sam.sam_annotator.annotator_tracking", "qualname": "AnnotatorTracking.__init__", "kind": "function", "doc": "Create the annotator GUI.
\n\nArguments:
\n\n- \n
- viewer: The napari viewer. \n
- ndim: The number of spatial dimension of the image data (2 or 3). \n
Start the tracking annotation tool fora given timeseries.
\n\nArguments:
\n\n- \n
- raw: The image data. \n
- embedding_path: Filepath for saving the precomputed embeddings. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- device: The computational device to use for the SAM model. \n
Returns:
\n\n\n\n", "signature": "(\timage: numpy.ndarray,\tembedding_path: Optional[str] = None,\tmodel_type: str = 'vit_l',\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\treturn_viewer: bool = False,\tviewer: Optional[napari.viewer.Viewer] = None,\tcheckpoint_path: Optional[str] = None,\tdevice: Union[str, torch.device, NoneType] = None) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.image_series_annotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "image_series_annotator", "kind": "function", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Run the annotation tool for a series of images (supported for both 2d and 3d images).
\n\nArguments:
\n\n- \n
- images: List of the file paths or list of (set of) slices for the images to be annotated. \n
- output_folder: The folder where the segmentation results are saved. \n
- model_type: The Segment Anything model to use. For details on the available models check out\nhttps://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models. \n
- embedding_path: Filepath where to save the embeddings. \n
- tile_shape: Shape of tiles for tiled embedding prediction.\nIf
Nonethen the whole image is passed to Segment Anything. \n - halo: Shape of the overlap between tiles, which is needed to segment objects on tile boarders. \n
- viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- precompute_amg_state: Whether to precompute the state for automatic mask generation.\nThis will take more time when precomputing embeddings, but will then make\nautomatic mask generation much faster. \n
- checkpoint_path: Path to a custom checkpoint from which to load the SAM model. \n
- is_volumetric: Whether to use the 3d annotator. \n
- prefer_decoder: Whether to use decoder based instance segmentation if\nthe model used has an additional decoder for instance segmentation. \n
Returns:
\n\n\n\n", "signature": "(\timages: Union[List[Union[os.PathLike, str]], List[numpy.ndarray]],\toutput_folder: str,\tmodel_type: str = 'vit_l',\tembedding_path: Optional[str] = None,\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\tviewer: Optional[napari.viewer.Viewer] = None,\treturn_viewer: bool = False,\tprecompute_amg_state: bool = False,\tcheckpoint_path: Optional[str] = None,\tis_volumetric: bool = False,\tdevice: Union[str, torch.device, NoneType] = None,\tprefer_decoder: bool = True) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.image_folder_annotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "image_folder_annotator", "kind": "function", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
Run the 2d annotation tool for a series of images in a folder.
\n\nArguments:
\n\n- \n
- input_folder: The folder with the images to be annotated. \n
- output_folder: The folder where the segmentation results are saved. \n
- pattern: The glob patter for loading files from
input_folder.\nBy default all files will be loaded. \n - viewer: The viewer to which the SegmentAnything functionality should be added.\nThis enables using a pre-initialized viewer. \n
- return_viewer: Whether to return the napari viewer to further modify it before starting the tool. \n
- kwargs: The keyword arguments for
micro_sam.sam_annotator.image_series_annotator. \n
Returns:
\n\n\n\n", "signature": "(\tinput_folder: str,\toutput_folder: str,\tpattern: str = '*',\tviewer: Optional[napari.viewer.Viewer] = None,\treturn_viewer: bool = False,\t**kwargs) -> Optional[napari.viewer.Viewer]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.ImageSeriesAnnotator", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "ImageSeriesAnnotator", "kind": "class", "doc": "The napari viewer, only returned if
\nreturn_viewer=True.
QWidget(parent: typing.Optional[QWidget] = None, flags: Union[Qt.WindowFlags, Qt.WindowType] = Qt.WindowFlags())
\n", "bases": "micro_sam.sam_annotator._widgets._WidgetBase"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.ImageSeriesAnnotator.__init__", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "ImageSeriesAnnotator.__init__", "kind": "function", "doc": "\n", "signature": "(viewer: napari.viewer.Viewer, parent=None)"}, {"fullname": "micro_sam.sam_annotator.image_series_annotator.ImageSeriesAnnotator.run_button", "modulename": "micro_sam.sam_annotator.image_series_annotator", "qualname": "ImageSeriesAnnotator.run_button", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.training_ui", "modulename": "micro_sam.sam_annotator.training_ui", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.training_ui.TrainingWidget", "modulename": "micro_sam.sam_annotator.training_ui", "qualname": "TrainingWidget", "kind": "class", "doc": "QWidget(parent: typing.Optional[QWidget] = None, flags: Union[Qt.WindowFlags, Qt.WindowType] = Qt.WindowFlags())
\n", "bases": "micro_sam.sam_annotator._widgets._WidgetBase"}, {"fullname": "micro_sam.sam_annotator.training_ui.TrainingWidget.__init__", "modulename": "micro_sam.sam_annotator.training_ui", "qualname": "TrainingWidget.__init__", "kind": "function", "doc": "\n", "signature": "(parent=None)"}, {"fullname": "micro_sam.sam_annotator.training_ui.TrainingWidget.run_button", "modulename": "micro_sam.sam_annotator.training_ui", "qualname": "TrainingWidget.run_button", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.util", "modulename": "micro_sam.sam_annotator.util", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.sam_annotator.util.point_layer_to_prompts", "modulename": "micro_sam.sam_annotator.util", "qualname": "point_layer_to_prompts", "kind": "function", "doc": "Extract point prompts for SAM from a napari point layer.
\n\nArguments:
\n\n- \n
- layer: The point layer from which to extract the prompts. \n
- i: Index for the data (required for 3d or timeseries data). \n
- track_id: Id of the current track (required for tracking data). \n
- with_stop_annotation: Whether a single negative point will be interpreted\nas stop annotation or just returned as normal prompt. \n
Returns:
\n\n\n\n", "signature": "(\tlayer: napari.layers.points.points.Points,\ti=None,\ttrack_id=None,\twith_stop_annotation=True) -> Optional[Tuple[numpy.ndarray, numpy.ndarray]]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.util.shape_layer_to_prompts", "modulename": "micro_sam.sam_annotator.util", "qualname": "shape_layer_to_prompts", "kind": "function", "doc": "The point coordinates for the prompts.\n The labels (positive or negative / 1 or 0) for the prompts.
\n
Extract prompts for SAM from a napari shape layer.
\n\nExtracts the bounding box for 'rectangle' shapes and the bounding box and corresponding mask\nfor 'ellipse' and 'polygon' shapes.
\n\nArguments:
\n\n- \n
- prompt_layer: The napari shape layer. \n
- shape: The image shape. \n
- i: Index for the data (required for 3d or timeseries data). \n
- track_id: Id of the current track (required for tracking data). \n
Returns:
\n\n\n\n", "signature": "(\tlayer: napari.layers.shapes.shapes.Shapes,\tshape: Tuple[int, int],\ti=None,\ttrack_id=None) -> Tuple[List[numpy.ndarray], List[Optional[numpy.ndarray]]]:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.util.prompt_layer_to_state", "modulename": "micro_sam.sam_annotator.util", "qualname": "prompt_layer_to_state", "kind": "function", "doc": "The box prompts.\n The mask prompts.
\n
Get the state of the track from a point layer for a given timeframe.
\n\nOnly relevant for annotator_tracking.
\n\nArguments:
\n\n- \n
- prompt_layer: The napari layer. \n
- i: Timeframe of the data. \n
Returns:
\n\n\n\n", "signature": "(prompt_layer: napari.layers.points.points.Points, i: int) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sam_annotator.util.prompt_layers_to_state", "modulename": "micro_sam.sam_annotator.util", "qualname": "prompt_layers_to_state", "kind": "function", "doc": "The state of this frame (either \"division\" or \"track\").
\n
Get the state of the track from a point layer and shape layer for a given timeframe.
\n\nOnly relevant for annotator_tracking.
\n\nArguments:
\n\n- \n
- point_layer: The napari point layer. \n
- box_layer: The napari box layer. \n
- i: Timeframe of the data. \n
Returns:
\n\n\n\n", "signature": "(\tpoint_layer: napari.layers.points.points.Points,\tbox_layer: napari.layers.shapes.shapes.Shapes,\ti: int) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data", "modulename": "micro_sam.sample_data", "kind": "module", "doc": "The state of this frame (either \"division\" or \"track\").
\n
Sample microscopy data.
\n\nYou can change the download location for sample data and model weights\nby setting the environment variable: MICROSAM_CACHEDIR
\n\nBy default sample data is downloaded to a folder named 'micro_sam/sample_data'\ninside your default cache directory, eg:\n * Mac: ~/Library/Caches/
Download the sample images for the image series annotator.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_image_series", "modulename": "micro_sam.sample_data", "qualname": "sample_data_image_series", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides image series example image to napari.
\n\nOpens as three separate image layers in napari (one per image in series).\nThe third image in the series has a different size and modality.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_wholeslide_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_wholeslide_example_data", "kind": "function", "doc": "Download the sample data for the 2d annotator.
\n\nThis downloads part of a whole-slide image from the NeurIPS Cell Segmentation Challenge.\nSee https://neurips22-cellseg.grand-challenge.org/ for details on the data.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_wholeslide", "modulename": "micro_sam.sample_data", "qualname": "sample_data_wholeslide", "kind": "function", "doc": "The path of the downloaded image.
\n
Provides wholeslide 2d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_livecell_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_livecell_example_data", "kind": "function", "doc": "Download the sample data for the 2d annotator.
\n\nThis downloads a single image from the LiveCELL dataset.\nSee https://doi.org/10.1038/s41592-021-01249-6 for details on the data.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_livecell", "modulename": "micro_sam.sample_data", "qualname": "sample_data_livecell", "kind": "function", "doc": "The path of the downloaded image.
\n
Provides livecell 2d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_hela_2d_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_hela_2d_example_data", "kind": "function", "doc": "Download the sample data for the 2d annotator.
\n\nThis downloads a single image from the HeLa CTC dataset.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> Union[str, os.PathLike]:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_hela_2d", "modulename": "micro_sam.sample_data", "qualname": "sample_data_hela_2d", "kind": "function", "doc": "The path of the downloaded image.
\n
Provides HeLa 2d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_3d_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_3d_example_data", "kind": "function", "doc": "Download the sample data for the 3d annotator.
\n\nThis downloads the Lucchi++ datasets from https://casser.io/connectomics/.\nIt is a dataset for mitochondria segmentation in EM.
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_3d", "modulename": "micro_sam.sample_data", "qualname": "sample_data_3d", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides Lucchi++ 3d example image to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_tracking_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_tracking_example_data", "kind": "function", "doc": "Download the sample data for the tracking annotator.
\n\nThis data is the cell tracking challenge dataset DIC-C2DH-HeLa.\nCell tracking challenge webpage: http://data.celltrackingchallenge.net\nHeLa cells on a flat glass\nDr. G. van Cappellen. Erasmus Medical Center, Rotterdam, The Netherlands\nTraining dataset: http://data.celltrackingchallenge.net/training-datasets/DIC-C2DH-HeLa.zip (37 MB)\nChallenge dataset: http://data.celltrackingchallenge.net/challenge-datasets/DIC-C2DH-HeLa.zip (41 MB)
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_tracking", "modulename": "micro_sam.sample_data", "qualname": "sample_data_tracking", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides tracking example dataset to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_tracking_segmentation_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_tracking_segmentation_data", "kind": "function", "doc": "Download groundtruth segmentation for the tracking example data.
\n\nThis downloads the groundtruth segmentation for the image data from fetch_tracking_example_data.
Arguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.sample_data_segmentation", "modulename": "micro_sam.sample_data", "qualname": "sample_data_segmentation", "kind": "function", "doc": "The folder that contains the downloaded data.
\n
Provides segmentation example dataset to napari.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.synthetic_data", "modulename": "micro_sam.sample_data", "qualname": "synthetic_data", "kind": "function", "doc": "Create synthetic image data and segmentation for training.
\n", "signature": "(shape, seed=None):", "funcdef": "def"}, {"fullname": "micro_sam.sample_data.fetch_nucleus_3d_example_data", "modulename": "micro_sam.sample_data", "qualname": "fetch_nucleus_3d_example_data", "kind": "function", "doc": "Download the sample data for 3d segmentation of nuclei.
\n\nThis data contains a small crop from a volume from the publication\n\"Efficient automatic 3D segmentation of cell nuclei for high-content screening\"\nhttps://doi.org/10.1186/s12859-022-04737-4
\n\nArguments:
\n\n- \n
- save_directory: Root folder to save the downloaded data. \n
Returns:
\n\n\n\n", "signature": "(save_directory: Union[str, os.PathLike]) -> str:", "funcdef": "def"}, {"fullname": "micro_sam.training", "modulename": "micro_sam.training", "kind": "module", "doc": "The path of the downloaded image.
\n
Functionality for training Segment Anything.
\n"}, {"fullname": "micro_sam.training.joint_sam_trainer", "modulename": "micro_sam.training.joint_sam_trainer", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.training.joint_sam_trainer.JointSamTrainer", "modulename": "micro_sam.training.joint_sam_trainer", "qualname": "JointSamTrainer", "kind": "class", "doc": "Trainer class for training the Segment Anything model.
\n\nThis class is derived from torch_em.trainer.DefaultTrainer.\nCheck out https://github.com/constantinpape/torch-em/blob/main/torch_em/trainer/default_trainer.py\nfor details on its usage and implementation.
Arguments:
\n\n- \n
- convert_inputs: The class that converts outputs of the dataloader to the expected input format of SAM.\nThe class
micro_sam.training.util.ConvertToSamInputscan be used here. \n - n_sub_iteration: The number of iteration steps for which the masks predicted for one object are updated.\nIn each sub-iteration new point prompts are sampled where the model was wrong. \n
- n_objects_per_batch: If not given, we compute the loss for all objects in a sample.\nOtherwise the loss computation is limited to n_objects_per_batch, and the objects are randomly sampled. \n
- mse_loss: The regression loss to compare the IoU predicted by the model with the true IoU. \n
- prompt_generator: The iterative prompt generator which takes care of the iterative prompting logic for training \n
- mask_prob: The probability of using the mask inputs in the iterative prompting (per
n_sub_iteration) \n - **kwargs: The keyword arguments of the DefaultTrainer super class. \n
Trainer class for training the Segment Anything model.
\n\nThis class is derived from torch_em.trainer.DefaultTrainer.\nCheck out https://github.com/constantinpape/torch-em/blob/main/torch_em/trainer/default_trainer.py\nfor details on its usage and implementation.
Arguments:
\n\n- \n
- convert_inputs: The class that converts outputs of the dataloader to the expected input format of SAM.\nThe class
micro_sam.training.util.ConvertToSamInputscan be used here. \n - n_sub_iteration: The number of iteration steps for which the masks predicted for one object are updated.\nIn each sub-iteration new point prompts are sampled where the model was wrong. \n
- n_objects_per_batch: If not given, we compute the loss for all objects in a sample.\nOtherwise the loss computation is limited to n_objects_per_batch, and the objects are randomly sampled. \n
- mse_loss: The regression loss to compare the IoU predicted by the model with the true IoU. \n
- prompt_generator: The iterative prompt generator which takes care of the iterative prompting logic for training \n
- mask_prob: The probability of using the mask inputs in the iterative prompting (per
n_sub_iteration) \n - **kwargs: The keyword arguments of the DefaultTrainer super class. \n
Wrapper to make the SegmentAnything model trainable.
\n\nArguments:
\n\n- \n
- sam: The SegmentAnything Model. \n
- device: The device for training. \n
Initializes internal Module state, shared by both nn.Module and ScriptModule.
\n", "signature": "(\tsam: segment_anything.modeling.sam.Sam,\tdevice: Union[str, torch.device])"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.sam", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.sam", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.device", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.device", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.transform", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.transform", "kind": "variable", "doc": "\n"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.preprocess", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.preprocess", "kind": "function", "doc": "Resize, normalize pixel values and pad to a square input.
\n\nArguments:
\n\n- \n
- x: The input tensor. \n
Returns:
\n\n\n\n", "signature": "(self, x: torch.Tensor) -> Tuple[torch.Tensor, Tuple[int, int]]:", "funcdef": "def"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.image_embeddings_oft", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.image_embeddings_oft", "kind": "function", "doc": "\n", "signature": "(self, batched_inputs):", "funcdef": "def"}, {"fullname": "micro_sam.training.trainable_sam.TrainableSAM.forward", "modulename": "micro_sam.training.trainable_sam", "qualname": "TrainableSAM.forward", "kind": "function", "doc": "The resized, normalized and padded tensor.\n The shape of the image after resizing.
\n
Forward pass.
\n\nArguments:
\n\n- \n
- batched_inputs: The batched input images and prompts. \n
- image_embeddings: The precompute image embeddings. If not passed then they will be computed. \n
- multimask_output: Whether to predict mutiple or just a single mask. \n
Returns:
\n\n\n\n", "signature": "(\tself,\tbatched_inputs: List[Dict[str, Any]],\timage_embeddings: torch.Tensor,\tmultimask_output: bool = False) -> List[Dict[str, Any]]:", "funcdef": "def"}, {"fullname": "micro_sam.training.training", "modulename": "micro_sam.training.training", "kind": "module", "doc": "\n"}, {"fullname": "micro_sam.training.training.FilePath", "modulename": "micro_sam.training.training", "qualname": "FilePath", "kind": "variable", "doc": "\n", "default_value": "typing.Union[str, os.PathLike]"}, {"fullname": "micro_sam.training.training.train_sam", "modulename": "micro_sam.training.training", "qualname": "train_sam", "kind": "function", "doc": "The predicted segmentation masks and iou values.
\n
Run training for a SAM model.
\n\nArguments:
\n\n- \n
- name: The name of the model to be trained.\nThe checkpoint and logs wil have this name. \n
- model_type: The type of the SAM model. \n
- train_loader: The dataloader for training. \n
- val_loader: The dataloader for validation. \n
- n_epochs: The number of epochs to train for. \n
- early_stopping: Enable early stopping after this number of epochs\nwithout improvement. \n
- n_objects_per_batch: The number of objects per batch used to compute\nthe loss for interative segmentation. If None all objects will be used,\nif given objects will be randomly sub-sampled. \n
- checkpoint_path: Path to checkpoint for initializing the SAM model. \n
- with_segmentation_decoder: Whether to train additional UNETR decoder\nfor automatic instance segmentation. \n
- freeze: Specify parts of the model that should be frozen, namely:\nimage_encoder, prompt_encoder and mask_decoder\nBy default nothing is frozen and the full model is updated. \n
- device: The device to use for training. \n
- lr: The learning rate. \n
- n_sub_iteration: The number of iterative prompts per training iteration. \n
- save_root: Optional root directory for saving the checkpoints and logs.\nIf not given the current working directory is used. \n
- mask_prob: The probability for using a mask as input in a given training sub-iteration. \n
- n_iterations: The number of iterations to use for training. This will over-ride n_epochs if given. \n
- scheduler_class: The learning rate scheduler to update the learning rate.\nBy default, ReduceLROnPlateau is used. \n
- scheduler_kwargs: The learning rate scheduler parameters.\nIf passed None, the chosen default parameters are used in ReduceLROnPlateau. \n
- save_every_kth_epoch: Save checkpoints after every kth epoch separately. \n
- pbar_signals: Controls for napari progress bar. \n
Create a PyTorch Dataset for training a SAM model.
\n\nArguments:
\n\n- \n
- raw_paths: The path(s) to the image data used for training.\nCan either be multiple 2D images or volumetric data. \n
- raw_key: The key for accessing the image data. Internal filepath for hdf5-like input\nor a glob pattern for selecting multiple files. \n
- label_paths: The path(s) to the label data used for training.\nCan either be multiple 2D images or volumetric data. \n
- label_key: The key for accessing the label data. Internal filepath for hdf5-like input\nor a glob pattern for selecting multiple files. \n
- patch_shape: The shape for training patches. \n
- with_segmentation_decoder: Whether to train with additional segmentation decoder. \n
- with_channels: Whether the image data has RGB channels. \n
- sampler: A sampler to reject batches according to a given criterion. \n
- n_samples: The number of samples for this dataset. \n
- is_train: Whether this dataset is used for training or validation. \n
Returns:
\n\n\n\n", "signature": "(\traw_paths: Union[List[Union[os.PathLike, str]], str, os.PathLike],\traw_key: Optional[str],\tlabel_paths: Union[List[Union[os.PathLike, str]], str, os.PathLike],\tlabel_key: Optional[str],\tpatch_shape: Tuple[int],\twith_segmentation_decoder: bool,\twith_channels: bool = False,\tsampler=None,\tn_samples: Optional[int] = None,\tis_train: bool = True,\t**kwargs) -> torch.utils.data.dataset.Dataset:", "funcdef": "def"}, {"fullname": "micro_sam.training.training.default_sam_loader", "modulename": "micro_sam.training.training", "qualname": "default_sam_loader", "kind": "function", "doc": "\n", "signature": "(**kwargs) -> torch.utils.data.dataloader.DataLoader:", "funcdef": "def"}, {"fullname": "micro_sam.training.training.CONFIGURATIONS", "modulename": "micro_sam.training.training", "qualname": "CONFIGURATIONS", "kind": "variable", "doc": "The dataset.
\n
Best training configurations for given hardware resources.
\n", "default_value": "{'Minimal': {'model_type': 'vit_t', 'n_objects_per_batch': 4, 'n_sub_iteration': 4}, 'CPU': {'model_type': 'vit_b', 'n_objects_per_batch': 10}, 'gtx1080': {'model_type': 'vit_t', 'n_objects_per_batch': 5}, 'rtx5000': {'model_type': 'vit_b', 'n_objects_per_batch': 10}, 'V100': {'model_type': 'vit_b'}, 'A100': {'model_type': 'vit_h'}}"}, {"fullname": "micro_sam.training.training.train_sam_for_configuration", "modulename": "micro_sam.training.training", "qualname": "train_sam_for_configuration", "kind": "function", "doc": "Run training for a SAM model with the configuration for a given hardware resource.
\n\nSelects the best training settings for the given configuration.\nThe available configurations are listed in CONFIGURATIONS.
Arguments:
\n\n- \n
- name: The name of the model to be trained.\nThe checkpoint and logs wil have this name. \n
- configuration: The configuration (= name of hardware resource). \n
- train_loader: The dataloader for training. \n
- val_loader: The dataloader for validation. \n
- checkpoint_path: Path to checkpoint for initializing the SAM model. \n
- with_segmentation_decoder: Whether to train additional UNETR decoder\nfor automatic instance segmentation. \n
- kwargs: Additional keyword parameterts that will be passed to
train_sam. \n
Identity transformation.
\n\nThis is a helper function to skip data normalization when finetuning SAM.\nData normalization is performed within the model and should thus be skipped as\na preprocessing step in training.
\n", "signature": "(x):", "funcdef": "def"}, {"fullname": "micro_sam.training.util.require_8bit", "modulename": "micro_sam.training.util", "qualname": "require_8bit", "kind": "function", "doc": "Transformation to require 8bit input data range (0-255).
\n", "signature": "(x):", "funcdef": "def"}, {"fullname": "micro_sam.training.util.get_trainable_sam_model", "modulename": "micro_sam.training.util", "qualname": "get_trainable_sam_model", "kind": "function", "doc": "Get the trainable sam model.
\n\nArguments:
\n\n- \n
- model_type: The segment anything model that should be finetuned.\nThe weights of this model will be used for initialization, unless a\ncustom weight file is passed via
checkpoint_path. \n - device: The device to use for training. \n
- checkpoint_path: Path to a custom checkpoint from which to load the model weights. \n
- freeze: Specify parts of the model that should be frozen, namely: image_encoder, prompt_encoder and mask_decoder\nBy default nothing is frozen and the full model is updated. \n
- return_state: Whether to return the full checkpoint state. \n
Returns:
\n\n\n\n", "signature": "(\tmodel_type: str = 'vit_l',\tdevice: Union[str, torch.device, NoneType] = None,\tcheckpoint_path: Union[os.PathLike, str, NoneType] = None,\tfreeze: Optional[List[str]] = None,\treturn_state: bool = False) -> micro_sam.training.trainable_sam.TrainableSAM:", "funcdef": "def"}, {"fullname": "micro_sam.training.util.ConvertToSamInputs", "modulename": "micro_sam.training.util", "qualname": "ConvertToSamInputs", "kind": "class", "doc": "The trainable segment anything model.
\n
Convert outputs of data loader to the expected batched inputs of the SegmentAnything model.
\n\nArguments:
\n\n- \n
- transform: The transformation to resize the prompts. Should be the same transform used in the\nmodel to resize the inputs. If
Nonethe prompts will not be resized. \n - dilation_strength: The dilation factor.\nIt determines a \"safety\" border from which prompts are not sampled to avoid ambiguous prompts\ndue to imprecise groundtruth masks. \n
- box_distortion_factor: Factor for distorting the box annotations derived from the groundtruth masks. \n
Helper functions for downloading Segment Anything models and predicting image embeddings.
\n"}, {"fullname": "micro_sam.util.get_cache_directory", "modulename": "micro_sam.util", "qualname": "get_cache_directory", "kind": "function", "doc": "Get micro-sam cache directory location.
\n\nUsers can set the MICROSAM_CACHEDIR environment variable for a custom cache directory.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.util.microsam_cachedir", "modulename": "micro_sam.util", "qualname": "microsam_cachedir", "kind": "function", "doc": "Return the micro-sam cache directory.
\n\nReturns the top level cache directory for micro-sam models and sample data.
\n\nEvery time this function is called, we check for any user updates made to\nthe MICROSAM_CACHEDIR os environment variable since the last time.
\n", "signature": "() -> None:", "funcdef": "def"}, {"fullname": "micro_sam.util.models", "modulename": "micro_sam.util", "qualname": "models", "kind": "function", "doc": "Return the segmentation models registry.
\n\nWe recreate the model registry every time this function is called,\nso any user changes to the default micro-sam cache directory location\nare respected.
\n", "signature": "():", "funcdef": "def"}, {"fullname": "micro_sam.util.get_device", "modulename": "micro_sam.util", "qualname": "get_device", "kind": "function", "doc": "Get the torch device.
\n\nIf no device is passed the default device for your system is used.\nElse it will be checked if the device you have passed is supported.
\n\nArguments:
\n\n- \n
- device: The input device. \n
Returns:
\n\n\n\n", "signature": "(\tdevice: Union[str, torch.device, NoneType] = None) -> Union[str, torch.device]:", "funcdef": "def"}, {"fullname": "micro_sam.util.get_sam_model", "modulename": "micro_sam.util", "qualname": "get_sam_model", "kind": "function", "doc": "The device.
\n
Get the SegmentAnything Predictor.
\n\nThis function will download the required model or load it from the cached weight file.\nThis location of the cache can be changed by setting the environment variable: MICROSAM_CACHEDIR.\nThe name of the requested model can be set via model_type.\nSee https://computational-cell-analytics.github.io/micro-sam/micro_sam.html#finetuned-models\nfor an overview of the available models
Alternatively this function can also load a model from weights stored in a local filepath.\nThe corresponding file path is given via checkpoint_path. In this case model_type\nmust be given as the matching encoder architecture, e.g. \"vit_b\" if the weights are for\na SAM model with vit_b encoder.
By default the models are downloaded to a folder named 'micro_sam/models'\ninside your default cache directory, eg:
\n\n- \n
- Mac: ~/Library/Caches/
- Unix: ~/.cache/
or the value of the XDG_CACHE_HOME environment variable, if defined. \n - Windows: C:\\Users<user>\\AppData\\Local<AppAuthor><AppName>\\Cache\nSee the pooch.os_cache() documentation for more details:\nhttps://www.fatiando.org/pooch/latest/api/generated/pooch.os_cache.html \n
Arguments:
\n\n- \n
- model_type: The SegmentAnything model to use. Will use the standard vit_h model by default.\nTo get a list of all available model names you can call
get_model_names. \n - device: The device for the model. If none is given will use GPU if available. \n
- checkpoint_path: The path to a file with weights that should be used instead of using the\nweights corresponding to
model_type. If given,model_typemust match the architecture\ncorresponding to the weight file. E.g. if you use weights for SAM with vit_b encoder\nthenmodel_typemust be given as \"vit_b\". \n - return_sam: Return the sam model object as well as the predictor. \n
- return_state: Return the unpickled checkpoint state. \n
Returns:
\n\n\n\n", "signature": "(\tmodel_type: str = 'vit_l',\tdevice: Union[str, torch.device, NoneType] = None,\tcheckpoint_path: Union[str, os.PathLike, NoneType] = None,\treturn_sam: bool = False,\treturn_state: bool = False) -> mobile_sam.predictor.SamPredictor:", "funcdef": "def"}, {"fullname": "micro_sam.util.export_custom_sam_model", "modulename": "micro_sam.util", "qualname": "export_custom_sam_model", "kind": "function", "doc": "The segment anything predictor.
\n
Export a finetuned segment anything model to the standard model format.
\n\nThe exported model can be used by the interactive annotation tools in micro_sam.annotator.
Arguments:
\n\n- \n
- checkpoint_path: The path to the corresponding checkpoint if not in the default model folder. \n
- model_type: The SegmentAnything model type corresponding to the checkpoint (vit_h, vit_b, vit_l or vit_t). \n
- save_path: Where to save the exported model. \n
Compute the image embeddings (output of the encoder) for the input.
\n\nIf 'save_path' is given the embeddings will be loaded/saved in a zarr container.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- input_: The input data. Can be 2 or 3 dimensional, corresponding to an image, volume or timeseries. \n
- save_path: Path to save the embeddings in a zarr container. \n
- lazy_loading: Whether to load all embeddings into memory or return an\nobject to load them on demand when required. This only has an effect if 'save_path' is given\nand if the input is 3 dimensional. \n
- ndim: The dimensionality of the data. If not given will be deduced from the input data. \n
- tile_shape: Shape of tiles for tiled prediction. By default prediction is run without tiling. \n
- halo: Overlap of the tiles for tiled prediction. \n
- verbose: Whether to be verbose in the computation. \n
- pbar_init: Callback to initialize an external progress bar. Must accept number of steps and description.\nCan be used together with pbar_update to handle napari progress bar in other thread.\nTo enables using this function within a threadworker. \n
- pbar_update: Callback to update an external progress bar. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: mobile_sam.predictor.SamPredictor,\tinput_: numpy.ndarray,\tsave_path: Union[str, os.PathLike, NoneType] = None,\tlazy_loading: bool = False,\tndim: Optional[int] = None,\ttile_shape: Optional[Tuple[int, int]] = None,\thalo: Optional[Tuple[int, int]] = None,\tverbose: bool = True,\tpbar_init: Optional[<built-in function callable>] = None,\tpbar_update: Optional[<built-in function callable>] = None) -> Dict[str, Any]:", "funcdef": "def"}, {"fullname": "micro_sam.util.set_precomputed", "modulename": "micro_sam.util", "qualname": "set_precomputed", "kind": "function", "doc": "The image embeddings.
\n
Set the precomputed image embeddings for a predictor.
\n\nArguments:
\n\n- \n
- predictor: The SegmentAnything predictor. \n
- image_embeddings: The precomputed image embeddings computed by
precompute_image_embeddings. \n - i: Index for the image data. Required if
imagehas three spatial dimensions\nor a time dimension and two spatial dimensions. \n - tile_id: Index for the tile. This is required if the embeddings are tiled. \n
Returns:
\n\n\n\n", "signature": "(\tpredictor: mobile_sam.predictor.SamPredictor,\timage_embeddings: Dict[str, Any],\ti: Optional[int] = None,\ttile_id: Optional[int] = None) -> mobile_sam.predictor.SamPredictor:", "funcdef": "def"}, {"fullname": "micro_sam.util.compute_iou", "modulename": "micro_sam.util", "qualname": "compute_iou", "kind": "function", "doc": "The predictor with set features.
\n
Compute the intersection over union of two masks.
\n\nArguments:
\n\n- \n
- mask1: The first mask. \n
- mask2: The second mask. \n
Returns:
\n\n\n\n", "signature": "(mask1: numpy.ndarray, mask2: numpy.ndarray) -> float:", "funcdef": "def"}, {"fullname": "micro_sam.util.get_centers_and_bounding_boxes", "modulename": "micro_sam.util", "qualname": "get_centers_and_bounding_boxes", "kind": "function", "doc": "The intersection over union of the two masks.
\n
Returns the center coordinates of the foreground instances in the ground-truth.
\n\nArguments:
\n\n- \n
- segmentation: The segmentation. \n
- mode: Determines the functionality used for computing the centers. \n
- If 'v', the object's eccentricity centers computed by vigra are used. \n
- If 'p' the object's centroids computed by skimage are used. \n
Returns:
\n\n\n\n", "signature": "(\tsegmentation: numpy.ndarray,\tmode: str = 'v') -> Tuple[Dict[int, numpy.ndarray], Dict[int, tuple]]:", "funcdef": "def"}, {"fullname": "micro_sam.util.load_image_data", "modulename": "micro_sam.util", "qualname": "load_image_data", "kind": "function", "doc": "A dictionary that maps object ids to the corresponding centroid.\n A dictionary that maps object_ids to the corresponding bounding box.
\n
Helper function to load image data from file.
\n\nArguments:
\n\n- \n
- path: The filepath to the image data. \n
- key: The internal filepath for complex data formats like hdf5. \n
- lazy_loading: Whether to lazyly load data. Only supported for n5 and zarr data. \n
Returns:
\n\n\n\n", "signature": "(\tpath: str,\tkey: Optional[str] = None,\tlazy_loading: bool = False) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.util.segmentation_to_one_hot", "modulename": "micro_sam.util", "qualname": "segmentation_to_one_hot", "kind": "function", "doc": "The image data.
\n
Convert the segmentation to one-hot encoded masks.
\n\nArguments:
\n\n- \n
- segmentation: The segmentation. \n
- segmentation_ids: Optional subset of ids that will be used to subsample the masks. \n
Returns:
\n\n\n\n", "signature": "(\tsegmentation: numpy.ndarray,\tsegmentation_ids: Optional[numpy.ndarray] = None) -> torch.Tensor:", "funcdef": "def"}, {"fullname": "micro_sam.util.get_block_shape", "modulename": "micro_sam.util", "qualname": "get_block_shape", "kind": "function", "doc": "The one-hot encoded masks.
\n
Get a suitable block shape for chunking a given shape.
\n\nThe primary use for this is determining chunk sizes for\nzarr arrays or block shapes for parallelization.
\n\nArguments:
\n\n- \n
- shape: The image or volume shape. \n
Returns:
\n\n\n\n", "signature": "(shape: Tuple[int]) -> Tuple[int]:", "funcdef": "def"}, {"fullname": "micro_sam.visualization", "modulename": "micro_sam.visualization", "kind": "module", "doc": "The block shape.
\n
Functionality for visualizing image embeddings.
\n"}, {"fullname": "micro_sam.visualization.compute_pca", "modulename": "micro_sam.visualization", "qualname": "compute_pca", "kind": "function", "doc": "Compute the pca projection of the embeddings to visualize them as RGB image.
\n\nArguments:
\n\n- \n
- embeddings: The embeddings. For example predicted by the SAM image encoder. \n
Returns:
\n\n\n\n", "signature": "(embeddings: numpy.ndarray) -> numpy.ndarray:", "funcdef": "def"}, {"fullname": "micro_sam.visualization.project_embeddings_for_visualization", "modulename": "micro_sam.visualization", "qualname": "project_embeddings_for_visualization", "kind": "function", "doc": "PCA of the embeddings, mapped to the pixels.
\n
Project image embeddings to pixel-wise PCA.
\n\nArguments:
\n\n- \n
- image_embeddings: The image embeddings. \n
Returns:
\n\n\n\n", "signature": "(\timage_embeddings: Dict[str, Any]) -> Tuple[numpy.ndarray, Tuple[float, ...]]:", "funcdef": "def"}]; // mirrored in build-search-index.js (part 1) // Also split on html tags. this is a cheap heuristic, but good enough.The PCA of the embeddings.\n The scale factor for resizing to the original image size.
\n