All the following scripts are meant to be used with our modified version of the Matlab COCO API, cloned into dataset/COCO/ as dataset/COCO/cocoapi/.
This directory contains multiple scripts to evaluate the accuracy of the trained models. It is split into 2 sections:
- Auto-Validation during Training: Used to find the best models of each training session.
- Validation of Specific Models: Used to debug or analyze the best models accross different training sessions.
All the following scripts are meant to be used with our modified version of the Matlab COCO API, cloned into dataset/COCO/ as dataset/COCO/cocoapi/.
In my case, I duplicate this repo once on each one of my servers. Then, I move the models and other useful files into a central NAS (i.e., storage) server. Finally, I process it. Script (a) is run on each server. Scripts (b)-(h) are run from the central server. This is why the following scripts:
- validation/a_copyModels.sh.example: While training OpenPose, this script automatically copies the important files (each model snapshot, all prototxt files, and the txt log file) into the desired folder with the desired name. Only
EXPERIMENTandSHARED_FOLDERshould be modified between experiments. This script should be duplicated and edited asvalidation/a_copyModels.shon each training folder in order to be used (which is included in the gitignore file so it can be added in multiple training servers). - validation/a_copyModelsCar.sh.example: Analog to
validation/a_copyModels.sh.example, but applied to the car dataset. Experimental and not released work. See car/README.md for more details. - validation/b_emptyCocoJsons.sh: If there is not enough time to test all models, this script will generate empty files for the desired models (e.g., if not enough time to test them all, useful for the models generated during the first 150k iterations).
- validation/b_getCocoJsons.sh.example: This script automatically test each model snapshot with OpenPose and generates the JSON file that can be read with the official COCO API. It should also be renamed as
validation/b_getCocoJsons.shon each training duplicate of this repo. - validation/c_plotTrainLoss.m: Given the txt file auto-generated during training, this Matlab script displays a graph where the x-axis represents the number of iterations and the y-axis the loss during training. Usefull to make sure the network is behaving properly. Note that after about 20k-30k iterations, this graph does not seem to improve anymore. However, the accuracy during testing/validation still improves an additional 5-10% after that. If displayed in log-log scale, then you can appreciate it keeps improving.
- validation/d_plotAccuracies.m: Given all the generated JSON files, this Matlab script will display them all into the same figure. It allows multiple training sequences that will be displayed in different figures.
- validation/e_getValidations.m: Given specific JSON files, it prints the accuracy of that model on the command line.
- validation/f_getFalsePosNeg.m: Given a specific JSON file, it looks for images with false positives or negatives for model debugging.
- validation/g_getLowAccurateImages.m: Given a specific JSON file, it looks for images with low accuracy for model debugging.
- validation/h_readCaffeNet.ipynb: Given a specific JSON file, it reads and displays the value of its layers.