Can proofreading tracks make inferences more accurate?

[tserpling]

Hello!

I’m working on a project to determine if SLEAP can accurately track octopus arms. I believe it can because I have seen the video where SLEAP was used to track the arms of a hydra. I am treating each arm as a separate instance and using 7 nodes from the base of the arm to the tip (picture attached).

I trained the software on two octopus videos (total: 2590 frames) and got it to accurately track the arms after labeling 418 frames. When I ran inference, however, the identities for the eight arms were incorrect. I expected this since the tutorial said I would have to proofread tracks after inference, but I experienced a few issues when proofreading identities. After making corrections, I ran inference again. The identities were still incorrect and some identities were swapped in the previously proofread frames. When fixing these, identities would sometimes switch in a frame I already corrected after I proofread another frame.

Does the proofreading impact subsequent inferences or am I supposed to correct the identities each time I do an inference? Is there a way I can make inferences as accurate as possible prior to proofreading? I already tried suggestions referenced in other discussions pertaining to switched/mistaken identities (eg. set target instance, cull instance count, single break tracks, propagate track labels) yet none of them have significantly improved the results.

A few other questions: Is there a way to combine separate instances if they have been split up? What’s considered successful for a model trained on SLEAP, and does that include a successful tracking of identities?

Thanks,
Tara

[roomrys]

Hi @tserpling,

Whoa, cool project! Thanks for sharing the neat photos.

Proofreading is usually done at the final stage when you are happy with predictions and have run inference on the entire video; thus, when you correct tracks, that is the last step before exporting to analysis. Correcting the tracks on predictions will not be saved when running inference again - hence why proofreading is the final step.

You can set the track for a user-labeled instance which will be remembered (the track will not change) after running training/inference. To make predictions more accurate, you can ensure tracks are assigned to your user-labeled instances (as only user-labeled instances are used/retained in training/inference).

Unfortunately, there is currently no way to combine instances that have been predicted as two separate instances. A successful SLEAP model predicts the location of body parts accurately/precisely - the tracking still needs user correction through proofreading. We are working on a feature to add tracking as a classifier, but that has yet to be implemented.

Let us know if this helps!

Thanks,
Liezl

[tserpling]

Hello @roomrys,

Thank you for your help! I will update you on how it goes.

Best,
Tara