including skeleton

[bendichter]

ndx-pose has a place for "edges", which I think maps to "skeleton" in DLC. However, the example config.yml does not contain skeleton information and the converter does not handle skeleton info. Including this information would allow us to provide much better visualizations of DLC output in NWB Widgets.

[AlexEMG]

Good point. Skeleton can have two roles in DLC:

• for visualization (in the provided example it wasn't populated)
• for inference in the case of multiple animals, in which case it is functionally important (see Lauer et al.).

We should probably distinguish those cases. In general right now we are not putting much metadata from the DLC model that makes the predictions into the NWB file. Perhaps it would be best to also provide the relevant entries from pose-cfg.yaml.

[bendichter]

@AlexEMG how would you want to distinguish between these cases? What other types of data are in pose-cfg.yml that you think might be useful here?

[AlexEMG]

Depends on the philosophy. The most extreme case is having all metadata to document how the data was created; the minimalist version could be just the network name (scorer name as it is already included)

What are the essential values to store to understand the dataset and the process by which the dataset was produced?

• training/test dataset files
• Frames and labeled coordinates on frames
• Config settings
• Network architecture and training parameters
• ImageNet pretrained names, e.g., Resnet-50 https://github.com/DeepLabCut/DeepLabCut/blob/master/deeplabcut/pose_estimation_tensorflow/models/pretrained/pretrained_model_urls.yaml
• Model Zoo
• Version of DLC code
• settings for augmentation pipeline
• settings for inference code (e.g. thresholds, graph, etc.)
• one could even put the log files (yaml, tensor flow output, …)

For a given project, this metadata is likely going to be identical and could be put in a central NWB project file? On the other hand, the metadata is relatively small to store and could go with each video's output.

For the skeleton file,

• if it is used for inference, then I feel it should go into the metadata outlined above
• if it is for the sake of visualizing in NWB (or other downstream applications), then we could maybe give a specialized name? E.g. 'visualization_skeleton'

[AlexEMG]

Opened this issue more towards general metadata: #4

[AlexEMG]

The skeleton is now in!