Configurable Object Detection Model

[PaulHax]

Short term motivation

Demo general usefullness of app by allowing user to quickly configure wide variarty of dataset+model combos.

Could have a GUI to select a hugging face COCO dataset, then all related models are shown in model dropdown. Or vice versa, first select model, then related datasets are shown.

Where do the models come from?

• https://huggingface.co/models <- current source
• Users' hard drive

How to configure models?

• CLI, like how we do COCO datasets. I.e. nrtk-explorer --models ../my-model/some-file
• GUI to enter model name or pick file/directory

Need to convert model object detection output to COCO annotation data structure?

Code touch points

• https://github.com/Kitware/nrtk-explorer/blob/main/src/nrtk_explorer/library/object_detector.py#L50-L60
• https://github.com/Kitware/nrtk-explorer/blob/main/src/nrtk_explorer/app/ui/layout.py#L83-L98
• https://github.com/Kitware/nrtk-explorer/blob/main/src/nrtk_explorer/app/transforms.py#L150-L152

Embeddings too?

• https://github.com/Kitware/nrtk-explorer/blob/main/src/nrtk_explorer/library/embeddings_extractor.py#L50-L57
• https://github.com/Kitware/nrtk-explorer/blob/main/src/nrtk_explorer/app/embeddings.py#L67-L71

Shortest putt

Define list of Hugging Face model names via CLI.

Alternative

Support for visualization of offline model predictions (e.g. also loaded in COCO format), which allows for faster analysis and does not require on-device model inference.

• Support for multiple models with different labels (e.g. “fasterrcnn”, “yolo”, “ground truth”)

Related

Expanding set of datasets/perturbations/models available, switching to JATIC protocols

A PR: #117