Run predictions using python script. Parameters documentation?

[anas-masood]

Hi. Thanks for implementing SLEAP and its been working great so far. I'm trying to use the data structures documentation to create a python script in order predict videos sequentially from our data server. We have trained a top down model and the first minor tests seem to be working fine.

However, I was having trouble finding a documentation on how to pass some parameters to the predictor. The sequence of commands I follow are:
predictor = sleap.load_model([centroidPath, centered_InstancePath])
predictions = predictor.predict(myVideo)

I would like to increase the batch size of the predictor, and see if I can change any other parameters for fast processing (perhaps GPU allocation and for post tracking). I would like to set batch size to 64, and for the post tracker to assert tracking a single instance.

Update:
I tried (lazily I would admit) checking the code and found this at nn.Inference

def sleap_model

and changed the predictor to sleap.loadmodel([centroidPath,centered_InstancePath],batch_size=64,tracker_max_instances=1) but it doesn't seem to make my inferences any faster. The GPU usage also doesnt go higher than 10% and in rare cases goes up to 40%. Any ideas?
Any help would be appreciated.

Thanks,
Anas

[jmdelahanty]

For the last part re GPU utilization I would maybe check out upping your batch size if you haven't tried that yet! And also see this answer from the legend Talmo himself.

Edit:

I mean your batch size is already pretty big. I've noticed that performing tracking after inferences can yield low GPU use since the tracking piece can take some time whereas running just predictions first and then tracking after could help get some predictions out faster.

[anas-masood]

Thanks for the fast reply. Indeed cuda usage was at maximum and increasing the batch size further caused it to crash. I will stay at 64.

Without the tracking, each video (~210,000 frames at 1408x1408 pix) takes about 1 hour 40 mins with an average fps of 30. With tracking it is doubled to approximately 4 hours at 15 fps.

As a side question also, is there a way to provide tracker arguments (e.g.
def make_tracker_by_name( cls, tracker: str = "flow", similarity: str = "instance", match: str = "greedy", track_window: int = 5, min_new_track_points: int = 0, min_match_points: int = 0, # Optical flow options img_scale: float = 1.0, of_window_size: int = 21, of_max_levels: int = 3, save_shifted_instances: bool = False, # Pre-tracking options to cull instances target_instance_count: int = 0, pre_cull_to_target: bool = False, pre_cull_iou_threshold: Optional[float] = None, # Post-tracking options to connect broken tracks post_connect_single_breaks: bool = False, # TODO: deprecate these post-tracking cleaning options clean_instance_count: int = 0, clean_iou_threshold: Optional[float] = None, # Kalman filter options kf_init_frame_count: int = 0, kf_node_indices: Optional[list] = None, **kwargs, ) -> BaseTracker:
I noticed that sleap.load_model does not pass extra arguments to the tracker. I would definitely like to add the cull argument, and which node_indices to use for kalman filter.

[jmdelahanty]

Interesting! From what Talmo and his team have taught me, there's a few reasons stuff might slow down that they've been helping me benchmark/check out here:

• I/O of reading a video especially if it's over a network connection
• Video decoding might be hard for some reason on the video you have (maybe try encoding using recommended settings if it's not that way already)
• The computer is busy doing some other thing at the same time
• Maybe the model is really big and takes a while to run.

With the responses Talmo and his team sent along in that benchmarking thread, maybe that can be a good way to start diagnosing the problem! I'm guessing you have a pretty beefy card to handle a big batch size like that, so hopefully the things in that thread can help you use it to the most!

The last part I'm not sure about unfortunately.

Edit:

The sleap-track CLI lets you add a ton of parameters to a given call, but I don't know if it lets you add all of the ones you're talking about. In our lab people are running shell scripts mostly and there's a whole bunch of parameters people throw into the call.

[talmo]

Hi @anas-masood,

@jmdelahanty's answer has lots of great pointers.

To add to that, with regards to tracker customization, check out our notebook on standalone tracking which shows how to customize some of the tracker parameters that might not be exposed by the CLI.

Let me know if you have any questions!

Cheers,

Talmo