Merge branch 'master' into Ascope

PhotonVision · Jan 21, 2024 · 5e6b683 · 5e6b683
2 parents af09b7d + 70a1f80
commit 5e6b683
Show file tree

Hide file tree

Showing 17 changed files with 123 additions and 15 deletions.
diff --git a/source/_static/assets/objdetectFiltering.mp4 b/source/_static/assets/objdetectFiltering.mp4
diff --git a/source/docs/apriltag-pipelines/coordinate-systems.rst b/source/docs/apriltag-pipelines/coordinate-systems.rst
@@ -1,10 +1,10 @@
-Coordiante Systems
+Coordinate Systems
 ==================
 
-Field and Robot Coordiante Frame
+Field and Robot Coordinate Frame
 --------------------------------
 
-PhotonVision follows the WPILib conventions for the robot and field coordinate-systems, as defined `here <https://docs.wpilib.org/en/stable/docs/software/advanced-controls/geometry/coordinate-systems.html>`_.
+PhotonVision follows the WPILib conventions for the robot and field coordinate systems, as defined `here <https://docs.wpilib.org/en/stable/docs/software/advanced-controls/geometry/coordinate-systems.html>`_.
 
 You define the camera to robot transform in the robot coordinate frame.
 
@@ -21,7 +21,7 @@ The camera coordinate system is defined as follows, relative to the camera senso
 AprilTag Coordinate Frame
 -------------------------
 
-The AprilTag coordinate system is defined as follows, relative to the center of the AprilTag itself, and when viewing the tag as a robot would.=:
+The AprilTag coordinate system is defined as follows, relative to the center of the AprilTag itself, and when viewing the tag as a robot would:
 
 * The origin is the center.
 * The x-axis points to your right

diff --git a/source/docs/apriltag-pipelines/images/field-layout.png b/source/docs/apriltag-pipelines/images/field-layout.png
diff --git a/source/docs/apriltag-pipelines/images/multitag-ui.png b/source/docs/apriltag-pipelines/images/multitag-ui.png
diff --git a/source/docs/apriltag-pipelines/multitag.rst b/source/docs/apriltag-pipelines/multitag.rst
@@ -1,4 +1,53 @@
 MultiTag Localization
 =====================
 
-Coming soon!
+PhotonVision can combine AprilTag detections from multiple simultaniously observed AprilTags from a particular camera wih information about where tags are expected to be located on the field to produce a better estimate of where the camera (and therefore robot) is located on the field. PhotonVision can calculate this multi-target result on your coprocessor, reducing CPU usage on your RoboRio. This result is sent over NetworkTables along with other detected targets as part of the ``PhotonPipelineResult`` provided by PhotonLib.
+
+.. warning:: MultiTag requires an accurate field layout JSON be uploaded! Differences between this layout and tag's physical location will drive error in the estimated pose output.
+
+Enabling MultiTag
+^^^^^^^^^^^^^^^^^
+
+Ensure that your camera is calibrated and 3D mode is enabled. Navigate to the Output tab and enable "Do Multi-Target Estimation". This enables MultiTag using the uploaded field layout JSON to calculate your camera's pose in the field. This 3D transform will be shown as an additional table in the "targets" tab, along with the IDs of AprilTags used to compute this transform.
+
+.. image:: images/multitag-ui.png
+   :width: 600
+   :alt: Multitarget enabled and running in the PhotonVision UI
+
+.. note:: By default, enabling multi-target will disable calculating camera-to-target transforms for each observed AprilTag target to increase performance; the X/Y/angle numbers shown in the target table of the UI are instead calculated using the tag's expected location (per the field layout JSON) and the field-to-camera transform calculated using MultiTag. If you additionally want the individual camera-to-target transform calculated using SolvePNP for each target, enable "Always Do Single-Target Estimation".
+
+This multi-target pose estimate can be accessed using PhotonLib. We suggest using :ref:`the PhotonPoseEstimator class <docs/programming/photonlib/robot-pose-estimator:AprilTags and PhotonPoseEstimator>` with the ``MULTI_TAG_PNP_ON_COPROCESSOR`` strategy to simplify code, but the transform can be directly accessed using ``getMultiTagResult``/``MultiTagResult()`` (Java/C++).
+
+
+
+.. tab-set-code::
+
+    .. code-block:: java
+
+        var result = camera.getLatestResult();
+        if (result.getMultiTagResult().estimatedPose.isPresent) {
+          Transform3d fieldToCamera = result.getMultiTagResult().estimatedPose.best;
+        }
+
+
+    .. code-block:: C++
+
+        auto result = camera.GetLatestResult();
+        if (result.MultiTagResult().result.isPresent) {
+          frc::Transform3d fieldToCamera = result.MultiTagResult().result.best;
+        }
+
+.. note:: The returned field to camera transform is a transform from the fixed field origin to the camera's coordinate system. This does not change based on alliance color, and by convention is on the BLUE ALLIANCE wall.
+
+Updating the Field Layout
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+PhotonVision ships by default with the `2024 field layout JSON <https://github.com/wpilibsuite/allwpilib/blob/main/apriltag/src/main/native/resources/edu/wpi/first/apriltag/2024-crescendo.json>`_. The layout can be inspected by navigating to the settings tab and scrolling down to the "AprilTag Field Layout" card, as shown below.
+
+.. image:: images/field-layout.png
+   :width: 600
+   :alt: The currently saved field layout in the Photon UI
+
+An updated field layout can be uploaded by navigating to the "Device Control" card of the Settings tab and clicking "Import Settings". In the pop-up dialog, select the "Apriltag Layout" type and choose a updated layout JSON (in the same format as the WPILib field layout JSON linked above) using the paperclip icon, and select "Import Settings". The AprilTag layout in the "AprilTag Field Layout" card below should update to reflect the new layout.
+
+.. note:: Currently, there is no way to update this layout using PhotonLib, although this feature is under consideration.
diff --git a/source/docs/calibration/calibration.rst b/source/docs/calibration/calibration.rst
@@ -114,7 +114,7 @@ Investigating Calibration Data with mrcal
 
 `mrcal <https://mrcal.secretsauce.net/tour.html>`_ is a command-line tool for camera calibration and visualization. PhotonVision has the option to use the mrcal backend during camera calibration to estimate intrinsics. mrcal can also be used post-calibration to inspect snapshots and provide feedback. These steps will closely follow the `mrcal tour <https://mrcal.secretsauce.net/tour-initial-calibration.html>`_ -- I'm aggregating commands and notes here, but the mrcal documentation is much more thorough.
 
-Start by `Installing mrcal <https://mrcal.secretsauce.net/install.html>`_. Note that while mrcal *calibration* using Photon is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using `WSL 2 on Windows <https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps>`ap_ as well. You may also need to install ``feedgnuplot``. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one `built into vscode <https://github.com/ros2/ros2/issues/1406>`_.
+Start by `Installing mrcal <https://mrcal.secretsauce.net/install.html>`_. Note that while mrcal *calibration* using PhotonVision is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using `WSL 2 on Windows <https://learn.microsoft.com/en-us/windows/wsl/tutorials/gui-apps>`ap_ as well. You may also need to install ``feedgnuplot``. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one `built into vscode <https://github.com/ros2/ros2/issues/1406>`_.
 
 Let's run ``calibrationUtils.py`` as described above, and then cd into the output folder. From here, you can follow the mrcal tour, just replacing the VNL filename and camera imager size as necessary. My camera calibration was at 1280x720, so I've set the XY limits to that below.
 

diff --git a/source/docs/examples/simposeest.rst b/source/docs/examples/simposeest.rst
@@ -47,7 +47,7 @@ For both simulation and on-robot code, we create objects to represent the physic
          :lineno-start: 83
 
 
-To incorporate Photon Vision, we need to create a :code:`PhotonCamera`:
+To incorporate PhotonVision, we need to create a :code:`PhotonCamera`:
 
 .. tab-set::
 

diff --git a/source/docs/installation/sw_install/romi.rst b/source/docs/installation/sw_install/romi.rst
@@ -8,7 +8,7 @@ Installation
 
 The WPILibPi image includes FRCVision, which reserves USB cameras; to use PhotonVision, we need to edit the `/home/pi/runCamera` script to disable it.  First we will need to make the file system writeable; the easiest way to do this is to go to ``10.0.0.2`` and choose "Writable" at the top.
 
-SSH into the Raspberry Pi (using a tool like `Putty <https://www.putty.org/>`_ ) at the Romi's default address ``10.0.0.2``.  The default user is ``pi``, and the password is ``raspberry``.
+SSH into the Raspberry Pi (using Windows command line, or a tool like `Putty <https://www.chiark.greenend.org.uk/~sgtatham/putty/>`_ ) at the Romi's default address ``10.0.0.2``.  The default user is ``pi``, and the password is ``raspberry``.
 
 Follow the process for installing PhotonVision on :ref:`"Other Debian-Based Co-Processor Installation" <docs/installation/sw_install/other-coprocessors:Other Debian-Based Co-Processor Installation>`.  As it mentions this will require an internet connection so plugging into the ethernet jack on the Raspberry Pi will be the easiest solution. The pi must remain writable!
 

diff --git a/source/docs/installation/sw_install/windows-pc.rst b/source/docs/installation/sw_install/windows-pc.rst
@@ -28,6 +28,8 @@ To run PhotonVision, open a terminal window of your choice and run the following
 
 If your computer has a compatible webcam connected, PhotonVision should startup without any error messages. If there are error messages, your webcam isn't supported or another issue has occurred. If it is the latter, please open an issue on the `PhotonVision issues page <https://github.com/PhotonVision/photonvision/issues>`_.
 
+.. warning:: Using an integrated laptop camera may cause issues when trying to run PhotonVision. If you are unable to run PhotonVision on a laptop with an integrated camera, try disabling the camera's driver in Windows Device Manager.
+
 Accessing the PhotonVision Interface
 ------------------------------------
 Once the Java backend is up and running, you can access the main vision interface by navigating to ``localhost:5800`` inside your browser.
diff --git a/source/docs/objectDetection/about-object-detection.rst b/source/docs/objectDetection/about-object-detection.rst
@@ -0,0 +1,48 @@
+About Object Detection
+======================
+
+How does it work?
+^^^^^^^^^^^^^^^^^
+
+PhotonVision supports object detection using neural network accelerator hardware built into Orange Pi 5/5+ coprocessors. The Neural Processing Unit, or NPU, is `used by PhotonVision <https://github.com/PhotonVision/rknn_jni/tree/main>`_ to massively accelerate certain math operations like those needed for running ML-based object detection.
+
+For the 2024 season, PhotonVision ships with a **pre-trained NOTE detector** (shown above), as well as a mechanism for swapping in custom models. Future development will focus on enabling lower friction management of multiple custom models.
+
+.. image:: images/notes-ui.png
+
+Tracking Objects
+^^^^^^^^^^^^^^^^
+
+Before you get started with object detection, ensure that you have followed the previous sections on installation, wiring and networking. Next, open the Web UI, go to the top right card, and swtich to the “Object Detection” type. You should see a screen similar to the image above.
+
+PhotonVision currently ships with a NOTE detector based on a `YOLOv5 model <https://docs.ultralytics.com/yolov5/>`_. This model is trained to detect one or more object "classes" (such as cars, stoplights, or in our case, NOTES) in an input image. For each detected object, the model outputs a bounding box around where in the image the object is located, what class the object belongs to, and a unitless confidence between 0 and 1.
+
+.... note:: This model output means that while its fairly easy to say that "this rectangle probably contains a NOTE", we doesn't have any information about the NOTE's orientation or location. Further math in user code would be required to make estimates about where an object is physically located relative to the camera.
+
+Tuning and Filtering
+^^^^^^^^^^^^^^^^^^^^
+
+Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain).
+
+.. raw:: html
+
+        <video width="85%" controls>
+            <source src="../../_static/assets/objdetectFiltering.mp4" type="video/mp4">
+            Your browser does not support the video tag.
+        </video>
+
+The same area, aspect ratio, and target orientation/sort parameters from :ref:`reflective pipelines <docs/reflectiveAndShape/contour-filtering:Reflective>` are also exposed in the object detection card.
+
+Training Custom Models
+^^^^^^^^^^^^^^^^^^^^^^
+
+Coming soon!
+
+Uploading Custom Models
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. warning:: PhotonVision currently ONLY supports YOLOv5 models trained and converted to ``.rknn`` format for RK3588 CPUs! Other models require different post-processing code and will NOT work. The model conversion process is also highly particular. Proceed with care.
+
+Our `pre-trained NOTE model <https://github.com/PhotonVision/photonvision/blob/master/photon-server/src/main/resources/models/note-640-640-yolov5s.rknn>`_ is automatically extracted from the JAR when PhotonVision starts, only if a file named “note-640-640-yolov5s.rknn” and "labels.txt" does not exist in the folder ``photonvision_config/models/``. This technically allows power users to replace the model and label files with new ones without rebuilding Photon from source and uploading a new JAR.
+
+Use a program like WinSCP or FileZilla to access your coprocessor's filesystem, and copy the new ``.rknn`` model file into /home/pi. Next, SSH into the coprocessor and ``sudo mv /path/to/new/model.rknn /opt/photonvision/photonvision_config/models/note-640-640-yolov5s.rknn``. Repeat this process with the labels file, which should contain one line per label the model outputs with no training newline. Next, restart PhotonVision via the web UI.
diff --git a/source/docs/objectDetection/images/notes-ui.png b/source/docs/objectDetection/images/notes-ui.png
diff --git a/source/docs/objectDetection/index.rst b/source/docs/objectDetection/index.rst
@@ -0,0 +1,8 @@
+Object Detection
+================
+
+.. toctree::
+   :maxdepth: 0
+   :titlesonly:
+
+   about-object-detection
diff --git a/source/docs/programming/photonlib/getting-target-data.rst b/source/docs/programming/photonlib/getting-target-data.rst
@@ -6,7 +6,7 @@ Constructing a PhotonCamera
 
 What is a PhotonCamera?
 ^^^^^^^^^^^^^^^^^^^^^^^
-``PhotonCamera`` is a class in PhotonLib that allows a user to interact with one camera that is connected to hardware that is running Photon Vision. Through this class, users can retrieve yaw, pitch, roll, robot-relative pose, latency, and a wealth of other information.
+``PhotonCamera`` is a class in PhotonLib that allows a user to interact with one camera that is connected to hardware that is running PhotonVision. Through this class, users can retrieve yaw, pitch, roll, robot-relative pose, latency, and a wealth of other information.
 
 
 The ``PhotonCamera`` class has two constructors: one that takes a ``NetworkTable`` and another that takes in the name of the network table that PhotonVision is broadcasting information over. For ease of use, it is recommended to use the latter. The name of the NetworkTable (for the string constructor) should be the same as the camera's nickname (from the PhotonVision UI).

diff --git a/source/docs/reflectiveAndShape/contour-filtering.rst b/source/docs/reflectiveAndShape/contour-filtering.rst
@@ -22,7 +22,7 @@ Finally, speckle rejection is an algorithm that can discard contours whose area
 .. raw:: html
 
         <video width="85%" controls>
-            <source src="../../../../_static/assets/AreaRatioFullness.mp4" type="video/mp4">
+            <source src="../../_static/assets/AreaRatioFullness.mp4" type="video/mp4">
             Your browser does not support the video tag.
         </video>
 
@@ -74,6 +74,6 @@ Finally, target sort defines how targets are ranked, from "best" to "worst." The
 .. raw:: html
 
         <video width="85%" controls>
-            <source src="../../../../_static/assets/groupingSorting.mp4" type="video/mp4">
+            <source src="../../_static/assets/groupingSorting.mp4" type="video/mp4">
             Your browser does not support the video tag.
         </video>
diff --git a/source/docs/reflectiveAndShape/index.rst b/source/docs/reflectiveAndShape/index.rst
@@ -1,5 +1,5 @@
-Colored Shape Detection
-=======================
+Colored Shape & Reflective
+==========================
 
 .. toctree::
    :maxdepth: 0

diff --git a/source/docs/reflectiveAndShape/thresholding.rst b/source/docs/reflectiveAndShape/thresholding.rst
@@ -8,7 +8,7 @@ In PhotonVision, HSV thresholds is available in the "Threshold" tab.
 .. raw:: html
 
         <video width="85%" controls>
-            <source src="../../../../_static/assets/tuningHueSatVal.mp4" type="video/mp4">
+            <source src="../../_static/assets/tuningHueSatVal.mp4" type="video/mp4">
             Your browser does not support the video tag.
         </video>
 
@@ -20,7 +20,7 @@ The color picker can be used to quickly adjust HSV values. "Set to average" will
 .. raw:: html
 
         <video width="85%" controls>
-            <source src="../../../../_static/assets/colorPicker.mp4" type="video/mp4">
+            <source src="../../_static/assets/colorPicker.mp4" type="video/mp4">
             Your browser does not support the video tag.
         </video>
 

diff --git a/source/index.rst b/source/index.rst
@@ -87,6 +87,7 @@ PhotonVision is licensed under the `GNU GPL v3 <https://www.gnu.org/licenses/gpl
    docs/pipelines/index
    docs/apriltag-pipelines/index
    docs/reflectiveAndShape/index
+   docs/objectDetection/index
    docs/calibration/calibration
 
 .. toctree::