diff --git a/source/_static/assets/objdetectFiltering.mp4 b/source/_static/assets/objdetectFiltering.mp4 new file mode 100644 index 00000000..4a328df9 Binary files /dev/null and b/source/_static/assets/objdetectFiltering.mp4 differ diff --git a/source/docs/apriltag-pipelines/coordinate-systems.rst b/source/docs/apriltag-pipelines/coordinate-systems.rst index 3316b98a..de40ab2e 100644 --- a/source/docs/apriltag-pipelines/coordinate-systems.rst +++ b/source/docs/apriltag-pipelines/coordinate-systems.rst @@ -1,10 +1,10 @@ -Coordiante Systems +Coordinate Systems ================== -Field and Robot Coordiante Frame +Field and Robot Coordinate Frame -------------------------------- -PhotonVision follows the WPILib conventions for the robot and field coordinate-systems, as defined `here `_. +PhotonVision follows the WPILib conventions for the robot and field coordinate systems, as defined `here `_. You define the camera to robot transform in the robot coordinate frame. @@ -21,7 +21,7 @@ The camera coordinate system is defined as follows, relative to the camera senso AprilTag Coordinate Frame ------------------------- -The AprilTag coordinate system is defined as follows, relative to the center of the AprilTag itself, and when viewing the tag as a robot would.=: +The AprilTag coordinate system is defined as follows, relative to the center of the AprilTag itself, and when viewing the tag as a robot would: * The origin is the center. * The x-axis points to your right diff --git a/source/docs/apriltag-pipelines/images/field-layout.png b/source/docs/apriltag-pipelines/images/field-layout.png new file mode 100644 index 00000000..1e1ec0a6 Binary files /dev/null and b/source/docs/apriltag-pipelines/images/field-layout.png differ diff --git a/source/docs/apriltag-pipelines/images/multitag-ui.png b/source/docs/apriltag-pipelines/images/multitag-ui.png new file mode 100644 index 00000000..0eab8f75 Binary files /dev/null and b/source/docs/apriltag-pipelines/images/multitag-ui.png differ diff --git a/source/docs/apriltag-pipelines/multitag.rst b/source/docs/apriltag-pipelines/multitag.rst index d55ab0b0..134ba195 100644 --- a/source/docs/apriltag-pipelines/multitag.rst +++ b/source/docs/apriltag-pipelines/multitag.rst @@ -1,4 +1,53 @@ MultiTag Localization ===================== -Coming soon! +PhotonVision can combine AprilTag detections from multiple simultaniously observed AprilTags from a particular camera wih information about where tags are expected to be located on the field to produce a better estimate of where the camera (and therefore robot) is located on the field. PhotonVision can calculate this multi-target result on your coprocessor, reducing CPU usage on your RoboRio. This result is sent over NetworkTables along with other detected targets as part of the ``PhotonPipelineResult`` provided by PhotonLib. + +.. warning:: MultiTag requires an accurate field layout JSON be uploaded! Differences between this layout and tag's physical location will drive error in the estimated pose output. + +Enabling MultiTag +^^^^^^^^^^^^^^^^^ + +Ensure that your camera is calibrated and 3D mode is enabled. Navigate to the Output tab and enable "Do Multi-Target Estimation". This enables MultiTag using the uploaded field layout JSON to calculate your camera's pose in the field. This 3D transform will be shown as an additional table in the "targets" tab, along with the IDs of AprilTags used to compute this transform. + +.. image:: images/multitag-ui.png + :width: 600 + :alt: Multitarget enabled and running in the PhotonVision UI + +.. note:: By default, enabling multi-target will disable calculating camera-to-target transforms for each observed AprilTag target to increase performance; the X/Y/angle numbers shown in the target table of the UI are instead calculated using the tag's expected location (per the field layout JSON) and the field-to-camera transform calculated using MultiTag. If you additionally want the individual camera-to-target transform calculated using SolvePNP for each target, enable "Always Do Single-Target Estimation". + +This multi-target pose estimate can be accessed using PhotonLib. We suggest using :ref:`the PhotonPoseEstimator class ` with the ``MULTI_TAG_PNP_ON_COPROCESSOR`` strategy to simplify code, but the transform can be directly accessed using ``getMultiTagResult``/``MultiTagResult()`` (Java/C++). + + + +.. tab-set-code:: + + .. code-block:: java + + var result = camera.getLatestResult(); + if (result.getMultiTagResult().estimatedPose.isPresent) { + Transform3d fieldToCamera = result.getMultiTagResult().estimatedPose.best; + } + + + .. code-block:: C++ + + auto result = camera.GetLatestResult(); + if (result.MultiTagResult().result.isPresent) { + frc::Transform3d fieldToCamera = result.MultiTagResult().result.best; + } + +.. note:: The returned field to camera transform is a transform from the fixed field origin to the camera's coordinate system. This does not change based on alliance color, and by convention is on the BLUE ALLIANCE wall. + +Updating the Field Layout +^^^^^^^^^^^^^^^^^^^^^^^^^ + +PhotonVision ships by default with the `2024 field layout JSON `_. The layout can be inspected by navigating to the settings tab and scrolling down to the "AprilTag Field Layout" card, as shown below. + +.. image:: images/field-layout.png + :width: 600 + :alt: The currently saved field layout in the Photon UI + +An updated field layout can be uploaded by navigating to the "Device Control" card of the Settings tab and clicking "Import Settings". In the pop-up dialog, select the "Apriltag Layout" type and choose a updated layout JSON (in the same format as the WPILib field layout JSON linked above) using the paperclip icon, and select "Import Settings". The AprilTag layout in the "AprilTag Field Layout" card below should update to reflect the new layout. + +.. note:: Currently, there is no way to update this layout using PhotonLib, although this feature is under consideration. diff --git a/source/docs/calibration/calibration.rst b/source/docs/calibration/calibration.rst index 10e4f8f2..1fbbff12 100644 --- a/source/docs/calibration/calibration.rst +++ b/source/docs/calibration/calibration.rst @@ -114,7 +114,7 @@ Investigating Calibration Data with mrcal `mrcal `_ is a command-line tool for camera calibration and visualization. PhotonVision has the option to use the mrcal backend during camera calibration to estimate intrinsics. mrcal can also be used post-calibration to inspect snapshots and provide feedback. These steps will closely follow the `mrcal tour `_ -- I'm aggregating commands and notes here, but the mrcal documentation is much more thorough. -Start by `Installing mrcal `_. Note that while mrcal *calibration* using Photon is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using `WSL 2 on Windows `ap_ as well. You may also need to install ``feedgnuplot``. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one `built into vscode `_. +Start by `Installing mrcal `_. Note that while mrcal *calibration* using PhotonVision is supported on all platforms, but investigation right now only works on Linux. Some users have also reported luck using `WSL 2 on Windows `ap_ as well. You may also need to install ``feedgnuplot``. On Ubuntu systems, these commands should be run from a standalone terminal and *not* the one `built into vscode `_. Let's run ``calibrationUtils.py`` as described above, and then cd into the output folder. From here, you can follow the mrcal tour, just replacing the VNL filename and camera imager size as necessary. My camera calibration was at 1280x720, so I've set the XY limits to that below. diff --git a/source/docs/examples/simposeest.rst b/source/docs/examples/simposeest.rst index 9c4492d1..e376a77a 100644 --- a/source/docs/examples/simposeest.rst +++ b/source/docs/examples/simposeest.rst @@ -47,7 +47,7 @@ For both simulation and on-robot code, we create objects to represent the physic :lineno-start: 83 -To incorporate Photon Vision, we need to create a :code:`PhotonCamera`: +To incorporate PhotonVision, we need to create a :code:`PhotonCamera`: .. tab-set:: diff --git a/source/docs/installation/sw_install/romi.rst b/source/docs/installation/sw_install/romi.rst index 74eefb6b..55e16c2f 100644 --- a/source/docs/installation/sw_install/romi.rst +++ b/source/docs/installation/sw_install/romi.rst @@ -8,7 +8,7 @@ Installation The WPILibPi image includes FRCVision, which reserves USB cameras; to use PhotonVision, we need to edit the `/home/pi/runCamera` script to disable it. First we will need to make the file system writeable; the easiest way to do this is to go to ``10.0.0.2`` and choose "Writable" at the top. -SSH into the Raspberry Pi (using a tool like `Putty `_ ) at the Romi's default address ``10.0.0.2``. The default user is ``pi``, and the password is ``raspberry``. +SSH into the Raspberry Pi (using Windows command line, or a tool like `Putty `_ ) at the Romi's default address ``10.0.0.2``. The default user is ``pi``, and the password is ``raspberry``. Follow the process for installing PhotonVision on :ref:`"Other Debian-Based Co-Processor Installation" `. As it mentions this will require an internet connection so plugging into the ethernet jack on the Raspberry Pi will be the easiest solution. The pi must remain writable! diff --git a/source/docs/installation/sw_install/windows-pc.rst b/source/docs/installation/sw_install/windows-pc.rst index 9f3c6798..33dda468 100644 --- a/source/docs/installation/sw_install/windows-pc.rst +++ b/source/docs/installation/sw_install/windows-pc.rst @@ -28,6 +28,8 @@ To run PhotonVision, open a terminal window of your choice and run the following If your computer has a compatible webcam connected, PhotonVision should startup without any error messages. If there are error messages, your webcam isn't supported or another issue has occurred. If it is the latter, please open an issue on the `PhotonVision issues page `_. +.. warning:: Using an integrated laptop camera may cause issues when trying to run PhotonVision. If you are unable to run PhotonVision on a laptop with an integrated camera, try disabling the camera's driver in Windows Device Manager. + Accessing the PhotonVision Interface ------------------------------------ Once the Java backend is up and running, you can access the main vision interface by navigating to ``localhost:5800`` inside your browser. diff --git a/source/docs/objectDetection/about-object-detection.rst b/source/docs/objectDetection/about-object-detection.rst new file mode 100644 index 00000000..0b4fcc83 --- /dev/null +++ b/source/docs/objectDetection/about-object-detection.rst @@ -0,0 +1,48 @@ +About Object Detection +====================== + +How does it work? +^^^^^^^^^^^^^^^^^ + +PhotonVision supports object detection using neural network accelerator hardware built into Orange Pi 5/5+ coprocessors. The Neural Processing Unit, or NPU, is `used by PhotonVision `_ to massively accelerate certain math operations like those needed for running ML-based object detection. + +For the 2024 season, PhotonVision ships with a **pre-trained NOTE detector** (shown above), as well as a mechanism for swapping in custom models. Future development will focus on enabling lower friction management of multiple custom models. + +.. image:: images/notes-ui.png + +Tracking Objects +^^^^^^^^^^^^^^^^ + +Before you get started with object detection, ensure that you have followed the previous sections on installation, wiring and networking. Next, open the Web UI, go to the top right card, and swtich to the “Object Detection” type. You should see a screen similar to the image above. + +PhotonVision currently ships with a NOTE detector based on a `YOLOv5 model `_. This model is trained to detect one or more object "classes" (such as cars, stoplights, or in our case, NOTES) in an input image. For each detected object, the model outputs a bounding box around where in the image the object is located, what class the object belongs to, and a unitless confidence between 0 and 1. + +.... note:: This model output means that while its fairly easy to say that "this rectangle probably contains a NOTE", we doesn't have any information about the NOTE's orientation or location. Further math in user code would be required to make estimates about where an object is physically located relative to the camera. + +Tuning and Filtering +^^^^^^^^^^^^^^^^^^^^ + +Compared to other pipelines, object detection exposes very few tuning handles. The Confidence slider changes the minimum confidence that the model needs to have in a given detection to consider it valid, as a number between 0 and 1 (with 0 meaning completely uncertain and 1 meaning maximally certain). + +.. raw:: html + + + +The same area, aspect ratio, and target orientation/sort parameters from :ref:`reflective pipelines ` are also exposed in the object detection card. + +Training Custom Models +^^^^^^^^^^^^^^^^^^^^^^ + +Coming soon! + +Uploading Custom Models +^^^^^^^^^^^^^^^^^^^^^^^ + +.. warning:: PhotonVision currently ONLY supports YOLOv5 models trained and converted to ``.rknn`` format for RK3588 CPUs! Other models require different post-processing code and will NOT work. The model conversion process is also highly particular. Proceed with care. + +Our `pre-trained NOTE model `_ is automatically extracted from the JAR when PhotonVision starts, only if a file named “note-640-640-yolov5s.rknn” and "labels.txt" does not exist in the folder ``photonvision_config/models/``. This technically allows power users to replace the model and label files with new ones without rebuilding Photon from source and uploading a new JAR. + +Use a program like WinSCP or FileZilla to access your coprocessor's filesystem, and copy the new ``.rknn`` model file into /home/pi. Next, SSH into the coprocessor and ``sudo mv /path/to/new/model.rknn /opt/photonvision/photonvision_config/models/note-640-640-yolov5s.rknn``. Repeat this process with the labels file, which should contain one line per label the model outputs with no training newline. Next, restart PhotonVision via the web UI. diff --git a/source/docs/objectDetection/images/notes-ui.png b/source/docs/objectDetection/images/notes-ui.png new file mode 100644 index 00000000..9993d5e0 Binary files /dev/null and b/source/docs/objectDetection/images/notes-ui.png differ diff --git a/source/docs/objectDetection/index.rst b/source/docs/objectDetection/index.rst new file mode 100644 index 00000000..0bb65ca5 --- /dev/null +++ b/source/docs/objectDetection/index.rst @@ -0,0 +1,8 @@ +Object Detection +================ + +.. toctree:: + :maxdepth: 0 + :titlesonly: + + about-object-detection diff --git a/source/docs/programming/photonlib/getting-target-data.rst b/source/docs/programming/photonlib/getting-target-data.rst index 5342ccc2..c6264b89 100644 --- a/source/docs/programming/photonlib/getting-target-data.rst +++ b/source/docs/programming/photonlib/getting-target-data.rst @@ -6,7 +6,7 @@ Constructing a PhotonCamera What is a PhotonCamera? ^^^^^^^^^^^^^^^^^^^^^^^ -``PhotonCamera`` is a class in PhotonLib that allows a user to interact with one camera that is connected to hardware that is running Photon Vision. Through this class, users can retrieve yaw, pitch, roll, robot-relative pose, latency, and a wealth of other information. +``PhotonCamera`` is a class in PhotonLib that allows a user to interact with one camera that is connected to hardware that is running PhotonVision. Through this class, users can retrieve yaw, pitch, roll, robot-relative pose, latency, and a wealth of other information. The ``PhotonCamera`` class has two constructors: one that takes a ``NetworkTable`` and another that takes in the name of the network table that PhotonVision is broadcasting information over. For ease of use, it is recommended to use the latter. The name of the NetworkTable (for the string constructor) should be the same as the camera's nickname (from the PhotonVision UI). diff --git a/source/docs/reflectiveAndShape/contour-filtering.rst b/source/docs/reflectiveAndShape/contour-filtering.rst index 84bc2ebe..1d4367cd 100644 --- a/source/docs/reflectiveAndShape/contour-filtering.rst +++ b/source/docs/reflectiveAndShape/contour-filtering.rst @@ -22,7 +22,7 @@ Finally, speckle rejection is an algorithm that can discard contours whose area .. raw:: html @@ -74,6 +74,6 @@ Finally, target sort defines how targets are ranked, from "best" to "worst." The .. raw:: html diff --git a/source/docs/reflectiveAndShape/index.rst b/source/docs/reflectiveAndShape/index.rst index 53f0c5a2..fab30635 100644 --- a/source/docs/reflectiveAndShape/index.rst +++ b/source/docs/reflectiveAndShape/index.rst @@ -1,5 +1,5 @@ -Colored Shape Detection -======================= +Colored Shape & Reflective +========================== .. toctree:: :maxdepth: 0 diff --git a/source/docs/reflectiveAndShape/thresholding.rst b/source/docs/reflectiveAndShape/thresholding.rst index 5e9826a6..df6e8860 100644 --- a/source/docs/reflectiveAndShape/thresholding.rst +++ b/source/docs/reflectiveAndShape/thresholding.rst @@ -8,7 +8,7 @@ In PhotonVision, HSV thresholds is available in the "Threshold" tab. .. raw:: html @@ -20,7 +20,7 @@ The color picker can be used to quickly adjust HSV values. "Set to average" will .. raw:: html diff --git a/source/index.rst b/source/index.rst index f6ee923c..e322c0a5 100644 --- a/source/index.rst +++ b/source/index.rst @@ -87,6 +87,7 @@ PhotonVision is licensed under the `GNU GPL v3