Creation of a held-out test data for active learning training phase validation

[rohanbanerjee]

Description

I've been thinking a ways to evaluate out active learning training phases (mentioned in #29). We have currently trained a baseline model based on the good quality (✅ ) segmentations and we are using this model for predicting the SC on other images and then manually correct them on need basis.

Our overall goal to do active learning was to see if the model is able to adapt on not-great quality images and we expect performance improvement after each active learning phase. For this imo, we should have a separate held-out test set that we can test out trained models on.

How do we build this test set:

1. We divide all the datasets into a 80:20 ratio in which 80% is for training and 20% is for testing.
2. We concatenate all the 20% sets into one Heldout-test set (illustrated in the figure below)
3. After each and every active learning training phase, we test the performance on the test set created in step 2.

Figure: Creation of held-out test set for testing active learning phases

Ref: SCIseg paper

Tasks:

• Create the 80:20 train and test splits using the script in utilities/dataset_conversion/create_dataset_split.py
• Use these splits to create the held-out test set. Can be found at: duke/temp/rohan/fmri_sc_seg/datasets/
• Remove ⚠️ (artifact) data from the held-out test data as per the QC done in the issue Systematic review of binary ground truth quality #25 (comment)
• Generate QC for the test set
• Remove the artifacts images from this set based on Systematic review of binary ground truth quality #25
• Manually correct the GTs for the over/under-segmentations
• Test and report the performance of the already trained baseline model

[rohanbanerjee]

Uploading the QC for the held-out test set
qc_held_out_test.zip

I went through the QC and below is the report
20240305_held_out_qc_report-RB.zip

[rohanbanerjee]

Update:
Currently using a small subset of the held-out test data. Below is the list of the subjects:
participant_held-out.tsv

This dataset which includes the manual corrections can be found at: https://drive.google.com/file/d/18SiXA8RWzCo6TBovC8umZxWcsZW1soIV/view?usp=sharing

The QC for the above data can be found below:

qc_held_out_test_manually_corrected.zip

[rohanbanerjee]

• Upload the corrected data to Openneuro