The package provides scripts for training an image classifier using active learning (AL). It attempts to improve the efficiency of AL algorithms in an image processing context by training an embedding model that reduces images to a low dimensional space before active learning is performed. This approach has already proven useful in a study by Norouzaddeh et al. in training animal classifiers for camera trap data (see acknowledgements for more details), however their methods are applicable to any image classficiation problem.
Norouzaddeh et al. provided the code that they used in their experiment, however it lacked several features required to be used in a practical setting. Most notably, it was neither able to handle unlabelled data nor label that data. The package provided here is an adaptation of their code which addresses these issues and provides a pipeline for employing active learning with some additional quality of life features.
The main pipeline provided by this package is a function called "run_active_learning". It trains a model using the approach in the paper by Norouzaddeh et al. with some additional features such as providing checkpoints, testing the model and saving the model in an interoperable format. If you wish to use your own custom model architecture and training algorithm then this package could still prove useful by providing objects that can load-in unlabelled data, label data and keeping track of what images belong to which datasets (more information can be found in Designing your own model).
- Why use active learning?
- Quick start
- Using Timelapse
- Training algorithm
- Active learning files folder
- Long processing times
- Designing your own model
- Citation
- Contact
- Acknowledgements
Training neural networks to classify images typically require very large databases of manually labelled images. In some contexts, labelling these images can be far more expensive than the computational resources that are required to train the model. One way to reduce this cost is to label the images as the model is being trained. The idea behind this is to choose the best images to label at each iteration of the training algorithm as to provide the model with the most information at each stage. Algorithms that employ this technique are often classed under "active learning".
This package can be installed locally using PIP. To do so, download the GitHub repository and run the following code, where the folder given by "path\to\package" should contain the setup.py file
pip install "path\to\package"This package provides a function for training a model using active learning. The only information that it requires to run are the file paths to the training and unlabelled data, which must lie in separate directories. An example of how to run the main program can be seen below.
from camera_trap_al.main_scripts import run_active_learning
run_active_learning(
train_data = "path\to\train\data",
unlabelled_data = "path\to\unlabelled\data",
)The program uses PyTorch's ImageFolder class to read the data. As such, it requires that images be sorted into subfolders by class. The training data must contain all of the classes that could appear in the unlabelled dataset as there is currently no way to add classes to the data as the model is being trained. All unlabelled data must also appear in subfolders of its root directory although the names of these folders will be ignored and this data will not be considered to have been labelled. It is important to note that any image data that lies directly in the data directories and not inside a subfolder will be ignored. An example on how to structure the data can be found on their website.
The program will display loading bars for parts of the training algorithm that are known to take a long time, such as during finetuning and when extracting the image embedding. However, there are other parts of the program that might take some time but where progress bars cannot be generated. A log of when these processes start and stop can be generated by running the following code before the main function is run.
import logging
logging.getLogger().setLevel(logging.INFO)As per the premise of active learning, the program will ask for labels as the model is being trained. To do this, it will generate a CSV file in the top level of the Active Learning Files folder. The CSV will have the file name "timelapse_selector.csv" and have the following columns:
- File The file name of the image.
- RelativePath path to the image relative root dir for unlabelled data (i.e. path passed to "unlabelled_data" parameter) but without the file name.
- Selected Boolean. Entry is true if it needs a label and will be False otherwise.
The CSV will contain the file paths for all images in the unlabelled folder even if they have already been labelled by the program. The "Selected" column shows which images the program has highlighted as needing labels. All selected images will need to be labelled before the program can continue training the model. It should also be noted that the program will ignore any labels for images that weren't selected (i.e. where "Selected" was False). It is also advised that you keep a record of any and all labels that you make in case they are improperly recorded by the program.
To label an image, add a column to the CSV called "Species" that contains the class name for that image. The class name must exactly match one of those given by the sub folders of the "train_data" directory. After you have labelled all of the selected images in the CSV, save it in the "new_labels_bin" folder of the "Active Learning Files" directory and press enter on the command line. The saved file can have any file name as the program will simply read the first CSV file that it finds in there. If the program finds no issues with reading in the CSV then it will display the message "Labels were loaded successfully" and continue training the model.
The timelapse selector CSV was designed to be easily read by an image labelling software known as Timelapse, although any image labelling tool that adds a column called "Species" to a CSV would work. Timelapse is an open-source program that is commonly used by ecologists and animal conservation groups for analysing camera trap data. This package was orginally designed to help ecologists build models that can sort databases of animal images by automating large parts of the training algorithm, which is why it was designed to work with Timelapse and why image labels need to be placed in a "Species" column.
Details on how to install and set up Timelapse for this package can be found on this GitHub page. You will need to make sure that your template has the columns 'Species' and 'Selected' and that it is saved in the top level of the unlabelled data folder. To label the images you will have to load in the CSV, filter images by the "Selected" column, label the selected images and then export the CSV to the labels bin. When loading in the CSV, please make sure that you deselect your current filter if any otherwise Timelapse will not update the whole "Selected" column.
There is a known error when trying to use a GPU with this package. When the packages PyTorch and Torchvision are installed while this package is being installed through pip, it will not install the extension that is required to use these packages with a GPU. As such, the program will not be able to find your GPU should you have one in your machine. To work around this, please uninstall torch and torchvision through pip and re-install it with the required CUDA extention, which can be found on PyTorch's website. Of course, you will also need to install CUDA before you can use the PyTorch with CUDA extension, which can be found on NVIDIA's website. If your computer does not have an NVIDIA GPU then it currently cannot be used by this package.
Timelapse is a GUI that allows users to process and label images or video. Timelapse also allows users to review image recognition and classification with Microsoft Megadetector.
Installation link: Timelapse: Download and Installation
After installation, you need to first create a Timelapse Template before labelling the photos.
-
Open “Timelapse2TemplateEditor” in Timelapse file
-
Click “File” and “New template…” to create the template.
-
Save your template in the top level of the folder for unlabelled data and not in any subfolders of it.
-
Click "Choice" under "Add" and change "Data Label" to "Species".
- Click on "Define List" under "List" to add the species you want to label. The names of the classes must match those that appear in the train dataset exactly.
-
Again under "Add", click the "Flag" button and change the "Data Label" to "Selected".
- You can also add other labels such as "Count", which allows you to count how many animsals are present in the image. However, this will not be used by the package.
-
Quit Timelapse Template Editor after you finish editing, the template will save automatically after you close the window.
Warning A known error occurs when you set up the timelapse template before the run_active_learning program is run. This happens because Timelapse will produce a folder called "backup" in the same directory as the template which causes the PyTorch ImageFolder class to raise an error as it expects that all subfolders contain at least one image. To work around this, please close Timelapse and delete the backup folder that it creates before you first run the run_active_learning script. This error does not occur after the model has been initially trained on the train data folder or when the process is started from a checkpoint as by that point the program has already located all of the data.
After you created the template, you are now able to label your images/videos using Timelapse. To set up the Timelapse labeler, perform the following steps.
- Open "Timelapse2" in Timelapse file.
- Click "File template" and "Load template, images, and video files..." to open the template you created.
Once the Timelapse program has been set up, simply run the run_active_learning program and wait for the following message to appear.
Timelapse data selector file has been updated.
Please import it into the Timelapse database.
Please place exported Timelapse CSV in the labels bin at
"path\to\labels\bin"
Press enter when you have placed the labels in the labels binThis message tells you that the program has chosen some images that need labelling. To import this selection into Timelapse, do the following steps.
- Under the "File" tab, click on "Import data from a .csv file ...". Import the timelapse_selector.csv file in the top level of the Active Learning Folder. Timelapse will likely raise a warning saying that the DateTime column was not updated but that can be safely ignored as that is not used by the package.
- Click on the "Select" tab and select "Custom Selection". Check the box for the "Selected" column and check the Value checkbox for this variable (i.e. filter images such that "Selected" is True).
- Press the "Okay" button in the bottom-right to confirm your selection. The number of files that match your query should be precisely that of the active_batch size of the run_active_learning program.
At this point, the selected images can be viewed from either the "Image set" or "Data table" tab in the lower window. The images can be labelled by scrolling through them in the "Image set" tab and assigning their label in the Species drop-down menu. It is important to note that no images can be deleted or else the package will not continue the training algorithm. After labeling all the images for this batch, click "File", "export data in the current selection as a CSV file" and save the file to the new_labels_bin folder of the Active Learning Files folder.
Finally, the data must be deselected before the next batch of images are chosen or else Timelapse will not update the "Selected" column for the whole dataset when the new timelapse_selector.csv file is loaded in. To do so, simply return to the "custom_selection..." option in the "Select" tab and click on the "Reset to All Images" in the bottom-left and press "Okay". After this, press enter on the Python command line to submit the labels and, if they can be loaded successfully, wait for the program to request the next batch of labels.
This section on using Timelapse was written in collaboration with Lo Ching Hei while he worked at the the Remote Sensing and Global Ecology Lab at the University of Hong Kong.
The model that is trained by this package is actually two models that are executed in sequence when they are evaluated. Images are first passed through the embedding model which performs features extraction and those features are then passed to the classifier which makes the final decision. The reason behind splitting the model into two is to reduce the computational cost of the active learning process.
The embedding model is the more complicated model and takes its architecture and initial weights from a pre-trained neural network. The classifier on the other hand is a much simpler network with only two hidden layers. The embedding model is periodically finetuned during training whereas the classifier is retrained from scratch every time a batch of images have been labelled. From Norouzaddeh et al's study, performance tends to improve greatly whenever the embedding model is updated however there are only incrementaly gains in performance when the classifier is re-trained.
Almost all of the default values for the hyperparameters are the same as those found in Norouzaddeh et al.'s code for their experiment. The following code snippet shows all of the parts of the training algorithm that can be modified without changing the source code. The rest of this section describes in more detail what each of the parameters change. If the program starts from a checkpoint, parameters that change the data sets and the model's architecture will be ignored.
def run_active_learning(
train_data,
unlabelled_data,
validation_data = None,
validate_model : bool = True,
use_checkpoints : bool = True,
num_workers : int = 0,
output_dir = 'default',
active_batch : int = 100,
active_learning_strategy : str = 'margin',
use_pretrained : bool = True,
embedding_arch : str = 'resnet18',
embedding_loss_type : str = 'triplet',
embedding_loss_margin : float = 1.0,
embedding_loss_data_strategy : str = 'random',
feat_dim : int = 256,
normalize_embedding : bool = True,
extract_embedding_batch_size : int = 256,
embedding_finetuning_period = 2000,
embedding_finetuning_lr : float = 0.0001,
embedding_finetuning_weight_decay = 0,
embedding_finetuning_num_epochs : int = 20,
embedding_finetuning_loader_type : str = 'balanced',
embedding_train_lr : float = 0.00001,
embedding_train_weight_decay = 0.0005,
embedding_train_num_epochs : int = 5,
embedding_train_loader_type : str = 'single',
balanced_loader_num_classes : int = 20,
balanced_loader_num_samples : int = 10,
)Ignored when loaded from checkpoint : True
All image files must lie in subfolders of their data directory. Any images that lie in the top level of their directory will be ignored. For labelled data, the names of the subfolders in the top-level of the directory will be taken as the class name for all images that are in those folders. The names of the subfolders for the unlabelled directory will be ignored and no top-level subfolders can contain no images. Additionally, There is currently no way to add data to the AL program after the model has been initially trained.
Parameters:
-
train_data : Data that the model is intially trained on before active learning is performed. Expects an absolute file path to a directory of images Subfolders of the top level of this directory must contain all of the classes that could be found in the unlabelled dataset. If you think that you might find images that do not belong to any of your classes then it is advised to add an "Other" class that these images could be assigned to.
-
unlabelled_data : Data that has yet to be labelled. Expects an absolute file path to a directory of images Images that are labelled through the active learning process will not be moved from this folder even if they would technically be considered as labelled at that point. Therefore, if a checkpoint is deleted and external record of the new labels have been made then those labels be lost. Furthermore, if you wish to retrain the model from scratch using labels that you had made from a previous execution of the program then an external program would be required to move those labelled images from the unlabelled dataset to the train dataset.
-
validation_data : Data used to test the model as it is being trained. Expects an absolute file path to a directory of images. Classes of this dataset can be a subset of those in the train data but the names of the classes that are included must exactly match those in the train data. The model is tested every time new labels are added to the data and the test results can be found in the test results folder of the Active Learning Files folder.
Ignored when loaded from checkpoint : False
Parameters:
-
validate_model : Boolean. If True and if a test set exists, test results will be generated during training. Otherwise, no tests will be performed.
-
use_checkpoints : If True if a checkpoint exists, function will load model and data mappings from that checkpoint. Otherwise, it will train a model from scratch. Checkpoints are always overwritten by this program and so any progress made by old checkpoints will be lost if this is set to False.
-
num_workers : Number of workers that will train the embeddingmodel and extract features in parallel. WARNING num_workers must be set to 0 if running on a Windows machine or else the program will freeze indefinitely. This is because Windows OS blocks multi-processing requests from PyTorch.
-
output_dir : Directory where all files that are generated by the program will be stored. This includes checkpoints, label requests and the trained model. If 'default' is passed to the parameters then the program will create a folder in the working directory called "Active Learning Files" within which the files will be stored.
Ignored when loaded from checkpoint : False
Parameters that change how active learning is performed.
-
active_batch : Number of images that the program will ask to be labelled at each pass of the training loop.
-
active_learning_strategy : Strategy for choosing which images to label when active learning is performed. can choose from the following list: 'uniform', 'graph_density', 'entropy', 'confidence', 'kcenter', 'margin', 'informative_diverse', 'margin_cluster_mean', 'hierarchical'. Note that only the 'margin' strategy has been tested.
Ignored when loaded from checkpoint : True
Parameters that define the architecture and loss function of the embedding model.
-
use_pretrained : Boolean. If True, the program will implement transfer learning by initialising the embedding model with the pre-trained weights of the base model.
-
embedding_arch : Architecture of the embedding model. Only resnet, inception, densenet, vgg and alexnet architectures can be used by this package and using other architectures will likely cause errors to occur. When choosing an architecture, please pass the name of the architecture with its version number as a string. The name should be exactly the same as its model builder's name in the Torchvision models library. For a full list of the models in this library, pleas see their website.
-
embedding_loss_type : Loss function of embedding model. Can either be 'softmax', 'triplet' or 'siamese'. Only 'softmax' and 'triplet' have been tested.
-
embedding_loss_margin : Margin for triplet and siamese loss. Ignored if softmax loss is used.
-
embedding_loss_data_strategy : Data selection strategy for triplet and siamese loss. Can either be 'hardest', 'random', 'semi_hard' or 'hard_pair'. Ignored if loss type is 'softmax'. Note that if loss type is 'siamese' and data selection strategy is not 'hard_pair' then triplet loss will be used instead.
-
feat_dim : Number of features that the embedding model should extract from the images.
Ignored when loaded from checkpoint : False
-
normalize_embedding : Boolean. If True, embedding values will be normalised. This avoids bias caused by dominating features when active learning is performed. It is highly recommended that this parameter be True unless you are certain that the features extracted by the embedding model will be of the same order of magnitude.
-
extract_embedding_batch_size : Batch Size when features are extracted from images.
-
embedding_finetuning_period : Number of images to add via active learning before the embedding is finetuned. The counter that keeps track of the number of images that has been added is reset after every finetuning.
The parameters that are titled embedding_finetuning_* and embedding_train_* relate to the hyperparameters of the algorithm that trains the embedding model. The difference between them is that the embedding_train_* parameters are used when the model is initially trained on the training set and the embedding_finetuning_* parameters are used after data has been added through active learning. The leading asterisk in the following parameters should be replaced with either "embedding_train" or "embedding_finetuning" before the parameters are used.
-
*_lr : Learning rate of Adam optimiser.
-
*_weight_decay : Weight decay for Adam optimiser.
-
*_num_epochs : Number of epochs of the data to train/finetune the model on.
-
*_loader_type : Data sampling method during training/finetuning. Can either be 'single' or 'balanced'. If 'single', the loader simply shuffles the data with a batch size of 128 and then sequentially loads the data into the model. If 'balanced', images are sampled so that the number of images from each class in a batch are the same and, if necessary, it reuses images from a class if that class does not have enough images.
-
balanced_loader_num_classes : Number of classes to sample from at each batch of the balanced loader. Capped at number of classes in the training data.
-
balanced_loader_num_samples : Number of images to sample from each class per batch of the balanced loader.
Note that the batch size of balanced loader is num_classes * num_samples.
Unfortunately, there is currently no way to change the hyperparameters of the classifier model as they are hard-coded into the program. If you wish to change its architecture and how it is trained then you will have to edit the source code.
By default, the classifier is a Neural Network with two hidden layers. The first hidden layer consists of 230 nodes and the second has only 100. The size of the input layer is the same as the dimension of the embedding and the output layer has the same number of nodes as there are image classes. The program will continue to train the model until either 2000 epochs have been reached or the difference in consecutive values of the loss function is below 10-6. Furethermore, it is optimised using the Adam method with a learning rate of 0.0001. All other values are left the same as the default values in Scikit-Learn's MLPClassifier object.
The program performs several image transformations to the data during training to improve the robustness of the model. Unfortunately, these transformations cannot be changed as they are hard-coded into the program. The transformations are precisely as follows:
-
Resize image to a 256*256 pixel square.
-
Crop the image with a 224*224 pixel square at a random part of the image.
-
Convert the image to grayscale with a 10% probability.
-
Apply all of the following transformations in a random order.
- Flip the image horizontally with a probability of 50%.
- Randomly change the brightness, contrast and other aspects of the colour of the image according to PyTorch's ColourJitter object.
- Randomly rotate the image up to 20° in either direction.
-
Convert the image to a tensor.
-
Standardise the image pixel values according to the mean and the standard deviation of the the un-transformed pixels from all datasets.
Should you apply the model on a new dataset after it has been trained, it is advised that you resize the image as per Step-1, crop the image at the centre to the same size as Step-2, convert the image to a tensor like in Step-5 and then standardise the pixel values as in Step-6 before passing the data through the model. The pixel mean and standard deviation matrices can be found with the saved model weights in the export folder.
The main program creates several files and directories while it trains the model. These are required to checkpoint its progress, load-in new labels and save the model in a format that can be exported to another machine. Descriptions of these folders, as well as those for some other features, can be found in this section.
The 'classifier', 'data' and 'embedding' folders contain all of the files that are required to load the training process from a checkpoint. Please do not remove any files from these folders or from the 'export' folder or else the program will restart the training process from scratch.
The classifier folder contains the saved classifier model and a record of which images lie in which dataset (i.e. train, unlabelled or validation). The classifier model file in this folder is saved using Python's pickle library and so it cannot be guaranteed to be possible to load it on a different machine.
The data folder contains the pickled PyTorch dataset object which is used to handle the data and the labels. It also contains a record of the what images need labels when chosen through active learning and a CSV file that lists the labels for all images. If an image has yet to be labelled, its class is listed as "unlabelled". Thus, to avoid confusion it is recommended that no classes in the training data should be called "unlabelled".
The embedding folder contains two files. The first is a matrix which is the embedding values for all images across all datasets. The second contains the model weights and other parameters that define the embedding model. The file for the embedding model is actually in a machine interoperable format but for convenience is saved in both this folder and in the export folder.
An initially empty folder whose sole purpose is to read in labels (see the section on labelling images for more details). The program does not remove any files in this folder and so when labelling images it is recommended that you overwrite the same CSV file every time. Furthermore, the program will only look at files in the top level of the directory and so CSV files contained in subfolders will be safely ignored.
If a validation dataset is provided, the program will test the model every time before it asks for labels. The results of those tests will be placed in subfolders of this folder. The title of the folder details the total number of labelled images at the time of the test and contained inside that folder is the confusion matrix and several test metrics.
The rows of the confusion matrix correspond to the images' true labels and the columns are the labels that were predicted by the model. The classes are ordered in alphabetical order from left to right for columns and top to bottom for rows. If the validation dataset contains only a subset of the classes in the train set then it is possible that the confusion matrix will not have rows and columns for all of the classes as it will omit classes that have neither a true or predicted label. To see what classes appear in the matrix, see the 'conf_mat_classes' section of the metrics file. For more information on how the confusion matrix is generated and formatted, see the documentation on the Scikit-Learn function that the program uses.
The metrics file contains not only the classes of the confusion matrix but also several performance metrics. These metrics are precisely: accuracy, micro precision, micro recall, macro precision and macro recall. Accuracy, precision and recall are calculated with their usual definitions. The need to distinguish between "micro" and "macro" precision and recall arises from an ambiguity in their definitions in multi-class classification tasks. Descriptions on how these metrics are calculated can be found in the documentation of Scikit-Learn learn's precision_score and recall_score functions but in short, macro metrics are a simple average of the score across the classes and micro metrics are the same as the accuracy metric.
This folder contains all of the files that you need to apply the trained model on a new dataset. The folder contains four files: dataset_mean.npy, dataset_std.npy, classifier.onnx and embedding_model_weights.pt. The files dataset_mean.npy and dataset_std.npy are used to standardise the data before it gets fed into the model and should be loaded with NumPy's load function; see the section on transforming the data for more information. The file classifier.onnx is the classifier model saved in ONNX format. Instructions for using it can be found in ONNX package's documentation. The last file is called embedding_model_weights.pt and can be loaded using PyTorch's load function. It is a dictionary that contains all of the information required to recover the embedding model including: the trained weights, dimension of the embedding and the model that it uses as a base.
It is recommended that you use the package "Animal Classifier Pipeline" to apply the model on a new dataset as it already has the functions for doing so. However, that package relies on the output of an animal detection model known as Megadetector and therefore cannot be directly used on non-animal data. If your model is trained outside of an animal classification context then you will have to design your own program that can apply the model. In that case, it may prove useful to look at the source code of the animal classifier pipeline to see how the model can be applied.
There are several processes in the training loop that are known to typically take a long time. Some of these include those that train the embedding model and extract the embedding and so the program will display progress bars while they run. However, there are some processes that are known to take a long time to run but where progress bars cannot be generated. Most notably, those that prepare the dataset before the model is initially trained usually takes a long time as the program has to calculate the mean and standard deviation of the images across all data sets. Additionally, loading the program from a checkpoint can take a long time as the program has to recover the state of the dataset using Python's pickle library. These processes are performed before the main active learning loop and so usually do not have a large impact on the overall training time.
One part of the process that may become a problem is when the classifier is trained. So far, the program that trains the classifier has only been tested on training datasets of up to 5000 images. It may be possible that training the model for larger datasets may be prohibitively expensive. In that case, the only way to reduce the training time would be to change the classifier model's architecture or training algorithm in the source code.
There are several Python packages that let you train a model with active learning. The pytorch_active_learning github repository provides scripts for training PyTorch models with AL and the modAL package provides a modular framework for training Scikit-Learn models with it. If you wish to train a model on a low-dimensional dataset then these two packages would likely prove to be more useful that this one.
The package provided here attempts to improve the efficiency of the AL training algorithm in an image processing context. It does so by reducing the dimensionality of the data that the AL algorithms query from and by training only the final classification layers of the model after each new batch of labels. The primary purpose of the "run_active_learning" function is to provide a way for people who are less familiar with machine learning models a way to easily train a model while only needing to manually label a small portion of their dataset.
Some of the functions and classes in this package can still prove useful for those who would like to design their own PyTorch image classifier using AL. In particular, the classes "ExtendedImageFolder", "ActiveLearningEnvironment" and "LabelRetriever" can be used to relabel datasets, keep track of which images are labelled and interpret new labels respectively. Additionally, the source code of the "run_active_learning" function provides an example of how to train a model with active learning using checkpoints.
The following code can be used to import these classes after the package has been installed.
from camera_trap_al.deep_learning.data_loader import ExtendedImageFolder
from camera_trap_al.deep_learning.active_learning_manager import ActiveLearningEnvironment
from camera_trap_al.utils.objects import LabelRetrieverThis package is licensed under the Apache Licence 2.0. As such, please cite it if you use it in your own project. when doing so, please include the name of the author, Gareth Lamb, a link to the GitHub page and the NOTICE file in this repository.
Several models have been trained with this package for classifying animals that can be found in Hong Kong. If you would like to use these models, or if you are interested in camera trap projects in Hong Kong, please contact Dr. Calvin Lee of the Global Ecology and Remote Sensing Lab at the University of Hong Kong at his email address, [email protected].
We would like to thank the team behind the animal detection algorithm, MegaDetector, for designing many of the functions that are used in this package to build and train the embedding model. The GitHub page for their active learning program also includes the classes that the "ExtendedImageFolder" and "ActiveLearningEnvironment" classes in this package were adapted from. We would also like to thank them for writing the framework for the algorithm that the "run_active_learning" function employs to efficiently train the model. The framework can be found in their paper, the details of which are as follows.
- Title - A deep active learning system for species identification and counting in camera trap images
- Author - Mohammad Sadegh Norouzzadeh, Dan Morris, Sara Beery, Neel Joshi, Nebojsa Jojic, Jeff Clune
- Journal - Methods in Ecology and Evolution
- Publisher - British Ecological Society
- Year - 2020
We would also like to thank the developers from Google for writing the active learning algorithms that this program uses.