Selecting model architecture for speed of realtime inference

[Kelechi-Akalaonu]

Hi -- thank you for this great resource!

I'm interested in using SLEAP for real-time inference in a closed-loop experiment and I would like to optimize inference speed. I will test different parameters like stride length and filter number, but I was wondering if you had recommendations for choosing among the various backbones and encoder models.

Our image size is 2048x1296, and the experimental arena will be populated by ~25 animals. We only need to extract 2 keypoints per animal because we are primarily interested in position and orientation.

[talmo]

Hi @Kelechi-Akalaonu,

Oh wow, that's going to be a tough one. That's pretty high resolution and quite a large number of animals.

My recommendations would be to:

1. Try top-down, but be aware that it'll scale almost linearly with the number of animals since it does a forward pass per animal for the second stage (centered instance model)
2. See how low you can turn down the input scale on the centroid model (0.50 is the default, maybe 0.25 would work)
3. On the centered instance model, use the UNet backbone, set the max stride to 16 , filters to 16, filter rate to 1.5.

Start off with that and let's see where you get!

Cheers,

Talmo

[Kelechi-Akalaonu]

Thank you for your quick reply! I'm looking forward to trying out your recommendations. Would you suggest using a UNet backbone for the centroid model as well? If so, do you have any initial parameter recommendations? Thanks again!