To faciliate the initial use of SimBA, we provide several use scenarios. We have created these scenarios around a hypothetical experiment that take a user from initial use (completely new start) all the way through analyzing a complete experiment and then adding additional experimental datasets to an initial project.
All scenarios assume that the videos have been pre-processed and that DLC behavioral tracking .CSV dataframes have been created.
Three days of resident-intruder testing between aggressive CD-1 mice and subordinante C57 intruders. Each day of testing has 10 pairs of mice, for a total of 30 videos recorded across 3 days. Recordings are 3 minutes in duration, in color, at 30fps.
In this Scenario you have generated a classifier using SimBA, but its possible (or, likely) that the classifier could be improved by including more training data. We suggest that you iterate between Scenario 1 and Scenerio 3 a few times before using a classifier on Experimental data, as done in Scenario 2. The goal is that you will only need to create a classifier once, so the better it is before experimental use, the less work you will have to do in the future!
Note 1: If you have received a classifier from another lab, or have notably changed the experimental recording configuration in your own lab, you will likely need create a new version of your classifier that now includes training examples from your own lab's behavioral setup or from the new recording conditions. While classifiers can genreralize well, they will always do better if the training dataset includes examples from your own behavioral setup! If your predictions are not going well with someone else's classifier, or after you have changed your behavioral equipment, this is a very likely reason and and easy fix.
Note 2: For the above reason, it is very important that you create clear operational definitions of annotated behaviors. Please see these examples for mouse and rat predictive classifiers for aggressive behaviors in resident-intruder protocols.
Note 3: If you plan to share a predictive classifier with other users, you will need to share the operationalized definitions, the
project_folder/csv/targets_insertedCSV files for all the videos used in the initial training set, and the video_info.csv file generated by SimBA which is located in theproject_folder\logsdirectory (further detailed below in Part 1).
HINT: SimBA inludes several model evaluation tools that cant be selected by the user when generating the predictive classfiers. Among other things, this includes learning curves, which indicates how the predictive classifier performs with increasing amounts of data. For more information on how to use SimBa to generate learning curves for your predictive classifier, click here. For more general information on learning curves in machine learning and their potential for informing on the benefit of additional data, click here.
In this tutorial, we will use SimBA to add further annotations to the predictive classifier generated in Scenario 1, and create an new classifier that now includes the training data from Scenario 1 as well as the new additional data.
Note: In practice, what we are doing in this Scenario is to generate a brand new classifier that contains previously annotated data as well as new annotated data, rather than building on an already generated classifier.
1. Locate the previously annotated CSV files. We will need to start with a project directory tree that does not contain any other data than the data we want to use to create the new predictive classifier. If you are coming along from Scenario 1 or Scenario 2 you will have one or more project directory trees already. The project tree from Scenario 1 will contain critical files that we want to use when creating the new predictive classifier in this current Scenario 3. The critical files are located in the project_folder/csv/targets_inserted directory of the Scenario 1 project. In this folder, there is one CSV file for each video, and each CSV file represents a video. Each CSV file contains the (i) pose-estimation data, (ii) the extracted features, and (iii) annotated behaviours that were created using the SimBA behavioral annotation GUI. Copy these CSV files to a safe memorable place, or alternatively, just be aware of their location. We will use them in later in this Scenario 3.
Note: If a predictive classifier has been generated by someone else, and you want to add further data to this classifer in order to generate a new classifier with more training data, you will need three things:
-
You will need the operational definitions of the behaviors, so that you can score the behaviors identically.
-
You will need the files from the targets_inserted folder from the project where the classifier was generated.
-
You will also need information on the resolution, pixels per mm, and fps from the videos used to create the original predictive classifier. This infomation is stored in the video_info.csv file generated by SimBA which is located in the
project_folder\logsdirectory. -
You do not have to have the frames, or the video files, for the videos that were used to generate the original classifier.
To create a new project, follow the instructions for creating a new project in either of these tutorials: 1, 2. Instead of importing the 'pilot' data as indicated in the tutorial from Scenario 1, import the pose estimation data and videos you want to 'add' to the predictive classifier.
Important I: In the final portion of the tutorials for creating a new project (Step 4), we extract the frames from the videos we have imported into the project. Having the frames is necessery in this current Scenario 3, as we will render them for annotating behaviors in the SimBA behavioral annotation GUI. However, you do not have to have the frames for the videos that were used to generate the original classifier.
Important II: Make sure the file names of the new videos, and new DeepLabCut pose estimation data, are not identical to the file names of the previously annotated videos.
Important III: Make sure the the classifier name(s), which is defined when creating the project in the SML Settings menu, is the same as the classifier name used to create the previous classifier.
In Part 2 above, we created a project that contains the data you wish to add to the predictive classifier. To continue working with this project, we must load it. To load the project and process your new data, follow the instructions for Step 1-6 in either of these tutorials: 1, 2.
When you have completed Step 1-6, you will have (i) corrected the outliers present in your new data, (ii) extracted features from your new data, (iii) and annotated behaviours in your new data. For example, if you annotated an additional, new, dataset contain 5 videos, you should now see 5 CSV files in the project_folder\csv\targets_inserted directory.
Part 4: Add the previously annotated CSV files to your project_folder\csv\targets_inserted directory.
We have now created a new project and have annotated data in this project, and we now want to import the annotations used to create the previous classifier into this project. We will do this manually.
1. Navigate to the folder that stores the previously annotated CSV files (see Part 1). Copy and paste them into your current current project into the project_folder\csv\targets_inserted directory.
You now have all the annotated files that will be used to create your predictive classifier in one place.
2. Create a new, updated, video_info.csv file. In the the current project directory, the video_info.csv file generated by SimBA, which is located in the project_folder\logs directory, will so far only include the information from the new videos that you wish to add to the classifier. It does not contain the information from the videos that were used to create the previous predictive classifier. To update the video_info.csv file so that it contains both the new and the older video information, first open the video_info.csv file from the earlier project (see Step 1) and copy its content. Next, open the video_info.csv file for the current project and paste the content into the file. You should now have a video_info.csv that contains a row of information for each of the CSV files represented in the project_folder\csv\targets_inserted directory. The content of the file may look similar to this:
Note I: In the example screenshoot above the videos have the same resolution, pixels per millimeter, and fps. However, they can be in any different combination of values across different videos.
Note II: Make sure that the column Video (the first column on the left), has the names of the video files in the
project_folder\csv\targets_inserteddirectory (excluding the file format, .csv).
You now have all the necessery data organised and you are ready to generate a new predictive classifier that includes the data used to generate the original classfier, plus the data from new annotated videos. To generate a new predictive classifier with these data, see Step 7 and onwards in the Scenario 1 tutorial.
Once you have completed Part 5 of the current Scenario 3, you may want to proceed to either:
-
Visualize and validate the new classifier on a video not used during training of the classifier. For validating a classifier on a new video, see Scenario 1 - Critical validation step before running machine model on new data.
-
Run the new classifier on new data. To run the classifier on new data, see Scenario 2 - Using a classifier on new experimental data.
-
Visualize machine predictions on one or several videos. To create visualizations, see Scenario 2 - Visualizing machine predictions. Note: You can only visualize the predictions on videos for which you have extracted the frames and have feature data for. For more information on extracting frames - click here.
Congratulations! You now have a new classifier of your behavior that is better than the original, or that has been updated from another lab's behavioral setup to work with your own! If you are following the Scenarios, you also now are ready to add the next two days of Experimental data to your project. Please see Scenario 4 for instructions.