Add RKNN object detection docs (#325)

* add rknn docs * Update index.rst * Update about-object-detection.rst * Update romi.rst * address review
PhotonVision · Jan 21, 2024 · 653960d · 653960d
1 parent 66cfd77
commit 653960d
Show file tree

Hide file tree

Showing 11 changed files with 61 additions and 8 deletions.
diff --git a/source/_static/assets/objdetectFiltering.mp4 b/source/_static/assets/objdetectFiltering.mp4
diff --git a/source/docs/calibration/calibration.rst b/source/docs/calibration/calibration.rst
@@ -114,7 +114,7 @@ Investigating Calibration Data with mrcal
 
 `mrcal <https://mrcal.secretsauce.net/tour.html>`_ is a command-line tool for camera calibration and visualization. PhotonVision has the option to use the mrcal backend during camera calibration to estimate intrinsics. mrcal can also be used post-calibration to inspect snapshots and provide feedback. These steps will closely follow the `mrcal tour <https://mrcal.secretsauce.net/tour-initial-calibration.html>`_ -- I'm aggregating commands and notes here, but the mrcal documentation is much more thorough.
 
-Start by `Installing mrcal <https://mrcal.secretsauce.net/install.html>`_. Note that while mrcal *calibration* using Photon is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using `WSL 2 on Windows <https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps>`ap_ as well. You may also need to install ``feedgnuplot``. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one `built into vscode <https://github.com/ros2/ros2/issues/1406>`_.
+Start by `Installing mrcal <https://mrcal.secretsauce.net/install.html>`_. Note that while mrcal *calibration* using PhotonVision is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using `WSL 2 on Windows <https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps>`ap_ as well. You may also need to install ``feedgnuplot``. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one `built into vscode <https://github.com/ros2/ros2/issues/1406>`_.
 
 Let's run ``calibrationUtils.py`` as described above, and then cd into the output folder. From here, you can follow the mrcal tour, just replacing the VNL filename and camera imager size as necessary. My camera calibration was at 1280x720, so I've set the XY limits to that below.
 

diff --git a/source/docs/examples/simposeest.rst b/source/docs/examples/simposeest.rst
@@ -47,7 +47,7 @@ For both simulation and on-robot code, we create objects to represent the physic
          :lineno-start: 83
 
 
-To incorporate Photon Vision, we need to create a :code:`PhotonCamera`:
+To incorporate PhotonVision, we need to create a :code:`PhotonCamera`:
 
 .. tab-set::
 

diff --git a/source/docs/installation/sw_install/romi.rst b/source/docs/installation/sw_install/romi.rst
@@ -8,7 +8,7 @@ Installation
 
 The WPILibPi image includes FRCVision, which reserves USB cameras; to use PhotonVision, we need to edit the `/home/pi/runCamera` script to disable it.  First we will need to make the file system writeable; the easiest way to do this is to go to ``10.0.0.2`` and choose "Writable" at the top.
 
-SSH into the Raspberry Pi (using a tool like `Putty <https://www.putty.org/>`_ ) at the Romi's default address ``10.0.0.2``.  The default user is ``pi``, and the password is ``raspberry``.
+SSH into the Raspberry Pi (using Windows command line, or a tool like `Putty <https://www.chiark.greenend.org.uk/~sgtatham/putty/>`_ ) at the Romi's default address ``10.0.0.2``.  The default user is ``pi``, and the password is ``raspberry``.
 
 Follow the process for installing PhotonVision on :ref:`"Other Debian-Based Co-Processor Installation" <docs/installation/sw_install/other-coprocessors:Other Debian-Based Co-Processor Installation>`.  As it mentions this will require an internet connection so plugging into the ethernet jack on the Raspberry Pi will be the easiest solution. The pi must remain writable!
 

diff --git a/source/docs/objectDetection/about-object-detection.rst b/source/docs/objectDetection/about-object-detection.rst
@@ -0,0 +1,44 @@
+About Object Detection
+======================
+
+How does it work?
+^^^^^^^^^^^^^^^^^
+
+PhotonVision supports object detection using neural network accelerator hardware built into Orange Pi 5/5+ coprocessors. The Neural Processing Unit, or NPU, is `used by PhotonVision <https://github.com/PhotonVision/rknn_jni/tree/main>`_ to massively accelerate certain math operations like those needed for running ML-based object detection.
+
+For the 2024 season, PhotonVision ships with a **pre-trained NOTE detector** (shown above), as well as a mechanism for swapping in custom models. Future development will focus on enabling lower friction management of multiple custom models.
+
+.. image:: images/notes-ui.png
+
+Tracking Objects
+^^^^^^^^^^^^^^^^
+
+Before you get started with object detection, ensure that you have followed the previous sections on installation, wiring and networking. Next, open the Web UI, go to the top right card, and swtich to the “Object Detection” type. You should see a screen similar to the image above.
+
+Tuning and Filtering
+^^^^^^^^^^^^^^^^^^^^
+
+Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain).
+
+.. raw:: html
+
+        <video width="85%" controls>
+            <source src="../../_static/assets/objdetectFiltering.mp4" type="video/mp4">
+            Your browser does not support the video tag.
+        </video>
+
+The same area, aspect ratio, and target orientation/sort parameters from :ref:`reflective pipelines <docs/reflectiveAndShape/contour-filtering:Reflective>` are also exposed in the object detection card.
+
+Training Custom Models
+^^^^^^^^^^^^^^^^^^^^^^
+
+Coming soon!
+
+Uploading Custom Models
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning:: PhotonVision currently ONLY supports YOLOV5 models trained and converted to ``.rknn`` format for RK3588 CPUs! Other models require different post-processing code and will NOT work. The model conversion process is also highly particular. Proceed with care.
+
+Our `pre-trained NOTE model <https://github.com/PhotonVision/photonvision/blob/master/photon-server/src/main/resources/models/note-640-640-yolov5s.rknn>`_ is automatically extracted from the JAR when PhotonVision starts, only if a file named “note-640-640-yolov5s.rknn” and "labels.txt" does not exist in the folder ``photonvision_config/models/``. This technically allows power users to replace the model and label files with new ones without rebuilding Photon from source and uploading a new JAR.
+
+Use a program like WinSCP or FileZilla to access your coprocessor's filesystem, and copy the new ``.rknn`` model file into /home/pi. Next, SSH into the coprocessor and ``sudo mv /path/to/new/model.rknn /opt/photonvision/photonvision_config/models/note-640-640-yolov5s.rknn``. Repeat this process with the labels file, which should contain one line per label the model outputs with no training newline. Next, restart PhotonVision via the web UI.
diff --git a/source/docs/objectDetection/images/notes-ui.png b/source/docs/objectDetection/images/notes-ui.png
diff --git a/source/docs/objectDetection/index.rst b/source/docs/objectDetection/index.rst
@@ -0,0 +1,8 @@
+Object Detection
+================
+
+.. toctree::
+   :maxdepth: 0
+   :titlesonly:
+
+   about-object-detection
diff --git a/source/docs/programming/photonlib/getting-target-data.rst b/source/docs/programming/photonlib/getting-target-data.rst
@@ -6,7 +6,7 @@ Constructing a PhotonCamera
 
 What is a PhotonCamera?
 ^^^^^^^^^^^^^^^^^^^^^^^
-``PhotonCamera`` is a class in PhotonLib that allows a user to interact with one camera that is connected to hardware that is running Photon Vision. Through this class, users can retrieve yaw, pitch, roll, robot-relative pose, latency, and a wealth of other information.
+``PhotonCamera`` is a class in PhotonLib that allows a user to interact with one camera that is connected to hardware that is running PhotonVision. Through this class, users can retrieve yaw, pitch, roll, robot-relative pose, latency, and a wealth of other information.
 
 
 The ``PhotonCamera`` class has two constructors: one that takes a ``NetworkTable`` and another that takes in the name of the network table that PhotonVision is broadcasting information over. For ease of use, it is recommended to use the latter. The name of the NetworkTable (for the string constructor) should be the same as the camera's nickname (from the PhotonVision UI).

diff --git a/source/docs/reflectiveAndShape/index.rst b/source/docs/reflectiveAndShape/index.rst
@@ -1,5 +1,5 @@
-Colored Shape Detection
-=======================
+Colored Shape & Reflective
+==========================
 
 .. toctree::
    :maxdepth: 0

diff --git a/source/docs/reflectiveAndShape/thresholding.rst b/source/docs/reflectiveAndShape/thresholding.rst
@@ -8,7 +8,7 @@ In PhotonVision, HSV thresholds is available in the "Threshold" tab.
 .. raw:: html
 
         <video width="85%" controls>
-            <source src="../../../../_static/assets/tuningHueSatVal.mp4" type="video/mp4">
+            <source src="../../_static/assets/tuningHueSatVal.mp4" type="video/mp4">
             Your browser does not support the video tag.
         </video>
 
@@ -20,7 +20,7 @@ The color picker can be used to quickly adjust HSV values. "Set to average" will
 .. raw:: html
 
         <video width="85%" controls>
-            <source src="../../../../_static/assets/colorPicker.mp4" type="video/mp4">
+            <source src="../../_static/assets/colorPicker.mp4" type="video/mp4">
             Your browser does not support the video tag.
         </video>
 

diff --git a/source/index.rst b/source/index.rst
@@ -87,6 +87,7 @@ PhotonVision is licensed under the `GNU GPL v3 <https://www.gnu.org/licenses/gpl
    docs/pipelines/index
    docs/apriltag-pipelines/index
    docs/reflectiveAndShape/index
+   docs/objectDetection/index
    docs/calibration/calibration
 
 .. toctree::