BrainHack 2024

[talmo]

Overview

Goal: Be able to store SLEAP package files in PoseTraining objects so we can embed images used for training models together with the ground truth poses.

Data model equivalences

sleap-io	ndx-pose
Labels	PoseTraining
LabeledFrame	TrainingFrame
Skeleton	Skeleton
Instance	SkeletonInstance
Video	ImageSeries
Labels.skeletons	Skeletons
Labels.labeled_frames	TrainingFrames
Labels.videos	SourceVideos

TODO

MVP

• Make TrainingFrame.skeleton_instance into TrainingFrame.skeleton_instances so it can hold multiple instances per frame
• Put SkeletonInstance creation inside of TrainingFrame loop in the README example
• Store image data in Image as 3-D array (Add tests for using Image in TrainingFrame.source_image #2)
• Make TrainingFrame.source_video optional so we don't need to keep the source video together with the NWB file (Add tests for using Image in TrainingFrame.source_image #2)
• Integrate into sleap_io.io.nwb
• Update docs/readme
• Finish tests

Nice to have

• Add a plain string field to TrainingFrame to point to source video as provenance but without triggering validation on DANDI
• Add support for PNG string encoding representation for Image
• Convert and upload reference SLEAP training datasets to DANDI
• Add support for directly opening SLEAP Dandisets in sleap-io via dandi API