Instance Segmentation Experiment

[lichtefeld]

The goal of this experiment is to test various ways of performing instance segmentation and the impact this has on stroke extraction and then training the GNN. While a complete experiment with this may desire a complex grid search to optimize parameters this could be tuned later in the process. For now we're mostly interested in the broad impacts of changing our segmentation process (Additional point of experimentation: Color / internal object segmentation). This issue aims to outline the experiment design & necessary technical changes to support it.

Experiment Layout

Curriculum

This experiment will use two different base sets of curriculum enabling three easy-to-produce variants. A set of ~400 images from the month 4 object demonstration rendered in Unreal Engine 4, a set of ~400 images from the Unreal Engine 5 rendering, and a final curriculum which combines both of the above source data (maybe this is worth a variant with double the data and a variant with a 50/50 split for each object but the same number of training instances?)

Instance Segmentation

The month 4 curriculum already has instance segmentation provided from the legacy AirSim integration. This will be the only curriculum for which results over this segmentation method will be available.

All curriculums will undergo instance segmentation on a base-model Mask R-CNN via a server API as well as a STEGO segmentation model. It's not quite in the spirit of adam's no labeled training data but given our minimal set of labeled data we could use that to fine-tune the top layer of the Mask R-CNN to improve it's performance (Expected outcome by discussions with Blake)

Stroke Extraction

We'll perform stroke extraction using the default parameters provided to us by Sheng. If we discover issues with these extractions we may consider additional tuning metrics to train over but I would like to keep this as only one configuration if possible. That said, I may consider a few of the improvements we've discussed internally including:

• Maximum stroke-length
• short-stroke combination
• others?

GNN Training

I'll use the provided GNN script to train the model. There may be parameters of the GNN training which we can tailor but this would require coordinating with Sheng in some capacity.

Technical Changes

STEGO & Mask R-CNN API

Right now the scripts only can support running one of the two of these models at a time, I'd like to adapt Blake's scripts to support running both in either two different servers (this seems simpler) or a single node with two different endpoints.

Curriculum generation script

Right now we have some source images in a desired curriculum layout. My goal is to write a unified preprocessing script which is designed to invoke a series of jobs to preprocess the required elements as well as manage the placement of files into the correct directories.

This requires determining how to integrate scripts from ASU into our processing steps. This currently requires utilizing different python environments. It would be nice to design this system in a way that minimizes the amount of multi-conda setup each of us needs to do to experiment on the cluster. I may borrow from other configuration of using Local python venvs for each of these modules.