HYDRA-845 : Selection highlighting for point instancing of meshes

doc/selectionHighlightingArchitecture.md

@@ -8,7 +8,7 @@ supports Hydra rendering in this repository as the Flow Viewport Toolkit (name
 subject to change).
 
 This document will describe the state of Flow Viewport Toolkit selection
-highlighting as of 21-Sep-2023.
+highlighting as of 31-May-2024.
 
 ## Behavior
 
@@ -18,16 +18,18 @@ are shown differently in the viewport for ease of understanding.
 
 An application will provide a way to select an object, or to select components
 of an object.  For example, for a mesh object, these components may be points,
-edges, or faces.  At time of writing, only object selection highlighting is
-supported, and selection highlighting of components is unimplemented.
+edges, or faces.  Currently, only object selection highlighting and point
+instancing highlighting of meshes are supported.  Selection highlighting of
+components is unimplemented.
 
 ## Selection: Application versus Hydra
 
 The application maintains an edit-friendly version of the scene.  This scene is
 translated into a Hydra scene by scene indices.  Correspondingly, there are two
 versions of the selection, one in the application, with objects and their paths
 described with application-specific classes, and a version of the selection in
-the Hydra scene, described as prims and their `SdfPath`s
+the Hydra scene, described as prims and their `SdfPath`s, as well as their
+associated selection data sources.
 
 ## Requirements
 
@@ -45,8 +47,8 @@ Requirements for selection highlighting are:
   highlight appearance.
 
 - It must be possible to let a data injecting data model provide prims to Hydra
-  that already contain selection highlighting.  At time of writing
-  (19-Sep-2023), this is true of Maya native Dag data, where selection
+  that already contain selection highlighting.  Currently,
+  this is true of Maya native Dag data, where selection
   highlighting is done by OGS.
 
 ## Selection Highlighting Styles
@@ -59,16 +61,16 @@ special way, e.g. object contour, modified object color, or object overlay.
 
 The former approach is handled by having a plugin provide a selection
 highlighting filtering scene index to the Flow Viewport Toolkit, and is the
-topic of this document at time of writing (20-Sep-2023).  The
-latter is handled by having a plugin provide a selection highlighting 
-task to the Flow Viewport Toolkit, and is currently unimplemented.
+topic of this document at time of writing.  The latter is handled by having 
+a plugin provide a selection highlighting task to the Flow Viewport Toolkit, 
+and is currently unimplemented.
 
 ## Added Geometry Plugin Software Architecture Requirements
 
 A selection highlighting plugin that provides added geometry to scene must
 provide the following services:
 
-- A way to translate the application's selection path(s) into Hydra paths:
+- A way to translate the application's selection path(s) into Hydra paths and data sources:
     - So that the appropriate prims in Hydra can be dirtied on selection change.
     - So that selected prims in Hydra can have a data source added.
   This is embodied in a **Path interface**.
@@ -83,16 +85,16 @@ provide the following services:
 ### Selection Change
 
 This
-[selection change code](../lib/flowViewport/sceneIndex/fvpSelectionSceneIndex.cpp#L150-L167)
-shows the use of the *Path Interface*, through the *SceneIndexPath()* method,
+[selection change code](../lib/flowViewport/sceneIndex/fvpSelectionSceneIndex.cpp#L152-L173)
+shows the use of the *Path Interface*, through the *ConvertUfePathToHydraSelections()* method,
 called on the input scene index.  The path interface allows the selection scene
 index to translate selected application paths to selected Hydra scene index
 paths.
 
 ### Wireframe Selection Highlighting
 
 This
-[wireframe selection highlighting code](../lib/flowViewport/sceneIndex/fvpWireframeSelectionHighlightSceneIndex.cpp#L76-L97)
+[wireframe selection highlighting code](../lib/flowViewport/sceneIndex/fvpWireframeSelectionHighlightSceneIndex.cpp#L462-L465)
 shows the use of the *Selection*, through the
 *HasFullySelectedAncestorInclusive()* method, called on the input selection.
 The selection allows a selection highlighting filtering scene index to query
@@ -178,7 +180,7 @@ index is optional.
 
 The object modeling is the following:
 - **Selection**: builtin provided by the Flow Viewport Toolkit.
-    - Encapsulates the Hydra selection as scene index paths.
+    - Encapsulates the Hydra selection as scene index paths and selection data sources.
     - Is shared by the selection scene index and all selection highlighting
       scene indices.
 - **Selection scene index**: builtin provided by the Flow Viewport Toolkit.
@@ -232,7 +234,7 @@ class Selection{
 }
 
 class PathInterface{
-+SceneIndexPath(Ufe::Path) SdfPath
++ConvertUfePathToHydraSelections(Ufe::Path) SdfPath
 }
 
 class SelectionSceneIndex
@@ -287,3 +289,183 @@ WireframeSelectionHighlightSceneIndex o-- Selection : Read
   propagate across scene index inputs, so that if a Maya Dag ancestor is
   selected, a USD descendant's appearance can change.  This is the same
   situation as global transformation and visibility.
+
+## Mesh point instancing wireframe selection highlighting
+
+We currently support wireframe selection highlighting for point instancing of meshes
+for the three different point instancing selection modes :
+
+- Point Instancer
+
+![Point instancer selection highlight](images/pointInstancerSelectionHighlight.png)
+
+- Instance
+
+![Instance selection highlight](images/instanceSelectionHighlight.png)
+
+- Prototype
+
+![Prototype selection highlight](images/prototypeSelectionHighlight.png)
+
+Here is an overview of how point instancing works in Hydra, and how we implement 
+wireframe selection highlighting for it.
+
+### Scene index structure
+
+In Hydra, a point instancer is represented as a prim of type `instancer`,
+with an `instancerTopology` data source. 
+
+![instancerTopology data source](images/instancerTopology.png)
+
+This data source contains three relevant inner data sources :
+- The `prototypes` data source, of type `VtArray<SdfPath>`, lists the paths
+to each prototype this point instancer instances.
+- The `instanceIndices` data source, a vector data source where each element
+data source (`i0, i1, i2, etc.`) is of type `VtArray<int>` and contains
+which instances correspond to which prototype. For example, if `i1` contains
+`0, 3`, then the first and fourth instances will be using the second prototype.
+- The `mask` data source, of type `VtArray<bool>`, which can optionally be used
+to show/hide specific instances (e.g. if the 3rd element of the mask is `false`,
+then the 3rd instance will be hidden). If this array is empty, all instances will be
+shown.
+
+---
+
+Per-instance data is specified using primvar data sources, namely :
+- hydra:instanceTranslations
+- hydra:instanceRotations
+- hydra:instanceScales
+- hydra:instanceTransforms
+
+![instanceTranslations primvar data source](images/instanceTranslations.png)
+
+Where the corresponding primvarValue data source lists the instance-specific data.
+Note that while the first three are 3-dimensional vectors and `hydra:instanceTransforms`
+is a 4x4 matrix, they can all be used simultaneously (internally, they will all be
+converted to 4x4 matrices, and then multiplied together).
+
+---
+
+On the other end of instancing, prototype prims have an `instancedBy` data source.
+
+![instancedBy data source](images/instancedBy.png)
+
+This data source contains up to two inner data sources :
+- (required) : The `paths` data source, of type `VtArray<SdfPath>`, lists the paths
+to each instancer that instances this prototype.
+- (optional) : When a sub-hierarchy is prototyped, the `prototypeRoots`, of type 
+`VtArray<SdfPath>`, lists the paths to the roots of the sub-hierarchies that are being 
+prototyped. For example, if we are instancing an xform that has a child mesh,
+then the prototype xform and mesh prims will each have the same `instancedBy` data source,
+where the `paths` data source will point to the instancers that use this prototype, and
+where the `prototypeRoots` will point to the xform prim.
+
+---
+
+Some notes about the behavioral impacts of the hierarchical location of prims :
+- Prims that are rooted under an instancer will not be drawn unless instanced
+- Prototypes that are instanced will still be drawn as if they were not instanced
+(i.e. the instances will be drawn in addition to the base prim itself), unless as 
+mentioned they are rooted under an instancer.
+
+### Nested/Composed instancers
+
+It is possible for an instancer itself to be instanced by another, and thus have both the
+`instancerTopology` and the `instancedBy` data sources. Note that this does not preclude
+such a prototyped instancer from also drawing geometry itself. If the prototyped instancer 
+is a child of the instancing instancer, then yes, such a nested instancer will not draw by
+itself, and will be instance-drawn through the parent instancer. However, if the prototyped
+instancer has no parent instancer, but it is instanced by another instancer somewhere else
+in the hierarchy, then both the prototyped instancer will draw as if it were by itself, but
+also be instance-drawn by the other instancer.
+
+This nesting and composition of instancers is what leads to most of the complexity of point
+instancing selection highlighting. We can view such nesting and composition of instancers as 
+graphs, with the vertices being the instancer prims, and the edges being the paths contained 
+in the `instancerTopology/prototypes`, `instancedBy/paths` and `instancedBy/prototypeRoots` 
+data sources.
+
+### Implementation for point instancer and instance selection
+
+This section will focus on selection highlighting when trying to highlight point instancers
+as a whole or specific instances, as these require a more complicated workflow. Unlike standard
+selection, we cannot simply override instanced meshes to use the `refinedWireOnSurf`/`wireOnSurf`
+HdReprs, as that would lead to highlighting all instances of the prototype all the time. Instead, 
+we opt for the following approach : when an instancer is selected (entirely or only certain instances), 
+we will create a mirror of the instancing graph it is a part of, and make the mirror copies of the 
+instanced meshes draw with a wireframe representation. This mirror graph includes everything from 
+the most deeply buried prims to the topmost instancers; anything that this instancer affects or is 
+affected by, including itself. In practice, this means that each prototype and each instancer will 
+have a corresponding mirror prim for selection highlighting, that will be located alongside it as 
+a sibling. This way, any parent transforms affecting the original prim will also affect the selection 
+highlight mirror prim.
+
+For example, given the following scene structure : 
+```
+Root
+|__TopInstancer
+   |__NestedInstancer
+      |__Prototype
+```
+the resulting scene structure with selection highlighting would become :
+```
+Root
+|__TopInstancer
+|  |__NestedInstancer
+|  |  |__Prototype
+|  |  |__Prototype_SelectionHighlight
+|  |__NestedInstancer_SelectionHighlight
+|__TopInstancer_SelectionHighlight
+```
+where `TopInstancer_SelectionHighlight` would instance `NestedInstancer_SelectionHighlight`, which would in turn instance `Prototype_SelectionHighlight`.
+
+Note that in the case where a prototype is not a single prim but a sub-hierarchy, we only need to 
+create a single *explicit* selection highlight mirror for the whole prototype sub-hierarchy; the 
+child prims of the selection highlight mirror will simply be pulled from the corresponding original 
+prim, and thus implicitly be selection highlight mirrors as well.
+
+Another thing to be aware of is that a nested/composed instancer is not necessarily directly selected, 
+as it is not necessarily a prototype root itself. If an instancer is a child prim of another prim 
+that is itself selected or instanced by another instancer, these instancers are still composed 
+together, but will not point to each other directly. Such cases are an example of when we need to 
+use the `instancedBy/prototypeRoots` data source to properly construct the mirror graph of instancers.
+
+An example of this is the following :
+```
+Root
+|__TopInstancer
+   |__Prototype
+      |__ChildInstancer
+```
+for which we end up with :
+```
+Root
+|__TopInstancer
+|  |__Prototype
+|  |  |__ChildInstancer
+|  |__Prototype_SelectionHighlight
+|     |__ChildInstancer
+|__TopInstancer_SelectionHighlight
+```
+where `TopInstancer_SelectionHighlight` instances `Prototype_SelectionHighlight`, which implicitly draws the selection highlight version of `ChildInstancer`.
+
+Of note are the following selection highlighting scenarios and their corresponding behaviors :
+- Selecting a point instancer in its entirety
+  - If the instancer is a top-level instancer, all instances it draws WILL be highlighted
+  - If the instancer is a prototype, instances of itself drawn by other instancers will NOT be highlighted
+- Selecting specific instances of point instancer
+  - If the instancer is a top-level instancer, the selected instances it draws WILL be highlighted
+  - If the instancer is a prototype, the instances it would indirectly draw through instances of itself drawn by other instancers will NOT be highlighted
+- Selecting a parent prim of a point instancer
+  - (same as selecting a point instancer in its entirety)
+
+### Implementation for prototype selection
+
+We simply do as for a regular selection, and override the original prim's display style 
+to draw as a wireframe-on-surface representation.
+
+### Current limitations
+
+- The wireframe colors for point instancer & instance selections might not always be correct (with 
+  respect to the lead/active selection colors). However, prototype selections should be using the 
+  correct colors.

lib/flowViewport/sceneIndex/fvpBlockPrimRemovalPropagationSceneIndex.h

@@ -70,8 +70,8 @@ class BlockPrimRemovalPropagationSceneIndex : public PXR_NS::HdSingleInputFilter
 
     //from PathInterface
     FVP_API
-    PXR_NS::SdfPath SceneIndexPath(const Ufe::Path& appPath) const override{
-        return _pathInterface->SceneIndexPath(appPath);
+    PrimSelectionInfoVector ConvertUfePathToHydraSelections(const Ufe::Path& appPath) const override{
+        return _pathInterface->ConvertUfePathToHydraSelections(appPath);
     }
 
 protected:

lib/flowViewport/sceneIndex/fvpMergingSceneIndex.cpp

@@ -34,7 +34,7 @@ MergingSceneIndex::MergingSceneIndex() : HdMergingSceneIndex()
         .Msg("MergingSceneIndex::MergingSceneIndex() called.\n");
 }
 
-SdfPath MergingSceneIndex::SceneIndexPath(const Ufe::Path& appPath) const
+PrimSelectionInfoVector MergingSceneIndex::ConvertUfePathToHydraSelections(const Ufe::Path& appPath) const
 {
     // FLOW_VIEWPORT_TODO  May be able to use a caching scheme for app path to
     // scene index path conversion using the run-time ID of the UFE path, as it
@@ -50,13 +50,13 @@ SdfPath MergingSceneIndex::SceneIndexPath(const Ufe::Path& appPath) const
         // scene we know whether it supports the PathInterface or not.
         auto pathInterface = dynamic_cast<const PathInterface*>(&*inputScene);
         if (pathInterface) {
-            auto sceneIndexPath = pathInterface->SceneIndexPath(appPath);
-            if (!sceneIndexPath.IsEmpty()) {
-                return sceneIndexPath;
+            auto primSelections = pathInterface->ConvertUfePathToHydraSelections(appPath);
+            if (!primSelections.empty()) {
+                return primSelections;
             }
         }
     }
-    return SdfPath();
+    return PrimSelectionInfoVector();
 }
 
 }

lib/flowViewport/sceneIndex/fvpMergingSceneIndex.h

@@ -41,7 +41,7 @@ class MergingSceneIndex
     static MergingSceneIndexRefPtr New();
 
     FVP_API
-    PXR_NS::SdfPath SceneIndexPath(const Ufe::Path& appPath) const override;
+    PrimSelectionInfoVector ConvertUfePathToHydraSelections(const Ufe::Path& appPath) const override;
 
 private:
     MergingSceneIndex();

lib/flowViewport/sceneIndex/fvpPathInterface.h

@@ -20,21 +20,31 @@
 #include <ufe/ufe.h>
 
 #include <pxr/pxr.h>
+#include <pxr/base/tf/smallVector.h>
+#include <pxr/imaging/hd/dataSource.h>
+#include <pxr/usd/sdf/path.h>
 
 UFE_NS_DEF {
 class Path;
 }
 
-PXR_NAMESPACE_OPEN_SCOPE
-class SdfPath;
-PXR_NAMESPACE_CLOSE_SCOPE
-
 namespace FVP_NS_DEF {
 
+struct PrimSelectionInfo
+{
+    PXR_NS::SdfPath primPath;
+    PXR_NS::HdDataSourceBaseHandle selectionDataSource;
+};
+
+// Using TfSmallVector to optimize for selections that map to a few prims,
+// which is likely going to be the bulk of use cases.
+using PrimSelectionInfoVector = PXR_NS::TfSmallVector<PrimSelectionInfo, 8>;
+
 /// \class PathInterface
 ///
 /// A pure interface class to allow for conversion between an application's
-/// path, expressed as a Ufe::Path, into an SdfPath valid for a scene index.
+/// path, expressed as a Ufe::Path, into SdfPaths valid for a scene index
+/// and selection data sources.
 /// To be used as a mix-in class for scene indices.
 ///
 class PathInterface
@@ -45,7 +55,7 @@ class PathInterface
     //! If no such path exists, an empty SdfPath should be returned.
     //! \return scene index path.
     FVP_API
-    virtual PXR_NS::SdfPath SceneIndexPath(const Ufe::Path& appPath) const = 0;
+    virtual PrimSelectionInfoVector ConvertUfePathToHydraSelections(const Ufe::Path& appPath) const = 0;
 
 protected:
 

lib/flowViewport/sceneIndex/fvpPathInterfaceSceneIndex.h

@@ -27,7 +27,7 @@ namespace FVP_NS_DEF {
 ///
 /// A simple pass-through filtering scene index that adds support for the path
 /// interface.  Derived classes need only implement the 
-/// PathInterface::SceneIndexPath() virtual.
+/// PathInterface::ConvertUfePathToHydraSelections() virtual.
 ///
 class PathInterfaceSceneIndexBase 
     : public PXR_NS::HdSingleInputFilteringSceneIndexBase