Is there support for 3D soft constraints?

[ldmoser]

Problem or Feature

Would there be an easy way to add 3D markers as constraints? Let's say that there's a 3D survey of the environment, where some points have known 3D positions in the scene. We want the camera transforms to be relative to that scene. I believe we would need the solver to support 3D constraints, forcing each 3D bundle point to match the corresponding 3D marker position (in world space). Is there any way this could be achieved in the current version?

Expected behavior:
Allow registering 3D markers that are not parented to a camera and will be compared in world space to the corresponding bundle. The loss is computed in 3D space instead of 2D for those markers. And perhaps, in order to unify 2D and 3D constraints there could be a user parameter that scales the 3D losses to a compatible unit. For example, if 3D locations are in cm, this scalar perhaps should be 10 to make the errors in 3D be about mm, and perhaps be more comparable to the errors from 2D markers (in pixels).

[david-cattermole]

Hello @ldmoser,

Yes, mmSolver can absolutely do that!
You just need to lock the 3D Bundle translate attributes with the attribute values you need, in other software the locked values mean the 3D Bundle point is "exactly surveyed".

"Soft" constraints for the Bundle attribute values are not supported in they way that other software may use them, however with mmSolver you can add weight to each individual (2D) Marker - if you want a specific Marker to have a higher priority (at the expensive of other Markers), then you can set the Marker's weight to (for example) 10.0 - this will give an higher weight to the Marker, and will therefore mean that all other Markers will effectively have a reduced weight.

mmSolver uses the Maya DAG, if an attribute is locked it will not be adjusted during the solve.
However if the attribute is unlocked and does not have a keyframe, it is considered "static", and if you add a single keyframe to the attribute each frame will be solved with a different value.

We have some getting started video tutorials (and quite advanced tutorials as well), in the documentation:
https://david-cattermole.github.io/mayaMatchMoveSolver/tutorial.html

[david-cattermole]

For deep dive into the low-level design of the Solver, we have some documentation here:
https://david-cattermole.github.io/mayaMatchMoveSolver/solver_design.html

Feel free to ask any questions if it's not clear.

Thanks,
David

[david-cattermole]

Hello again @ldmoser,

Sorry, I re-read your question - no "3D Marker to 3D Bundle" is not supported currently, with or without soft constraints.

I have had a similar request in the past and the plan was to implement a "Bundle to Bundle" solver.

Right now I'm unable to write such a solver, however if you're ok with a proprietary software, I am told Peel Solve provides similar functionality:
https://peeldev.com/peelsolve

On the other hand if you'd like to add such a feature, PRs are most welcome ;)

David

[ldmoser]

Thank you for taking the time!
It would be a great feature to add here, and be able to have 2D and 3D soft constraints. From the code so far it looked feasible to extend the definition of a Marker, adding a flag to determine whether it is 2D or 3D, or simply using the fact that 3D markers would not have Cameras associated with them. And then in the solver, bypass all the camera projection logic for the corresponding bundle, and simply compute the loss directly on the difference of the two (in world space). It's going to be probably be tricky to change the logic and perhaps UI to support the situation of cameras not being involved perhaps.
You think that route would be easier than somehow create two types of bundles (source and target), or you favoured that because of naming conventions (ie: marker is 2D and bundle is 3D?).
Thank you so much for sharing your project!

[david-cattermole]

Hello @ldmoser,

My fear with changing the definition of a Marker is conceptual for users (I have always insisted Markers are "2D"), but also pragmatic for the code.

Inside mmSolver Markers have a few assumptions baked into them currently:

• Markers have translateX and translateY, weight and bundle attributes - translateZ is always hidden.
• Markers (almost always) parented under Marker Group and camera transform - there are no connections to the camera, instead the camera is "connected" via the implicit parenting relationship in the Maya DAG.

In v0.4.x (currently on the develop branch), I am replacing the standard locator shape node with a custom shape node, which will provide a more "2D" look.

Therefore my approach would be to allow "linking" Bundles together somehow. Bundles are free to move around wherever, and can be parented under any (transform) node, There are less assumptions for Bundles, from a code and user perspective.

We wouldn't need special "source" and "target/destination" types of Bundle objects, although user tools could create such a concept using naming, or special attributes to tag such nodes. I would assume the user tool would pass only the attributes to be adjusted. By only adding the attributes from one bundle, it would be assumed the other bundle is "locked" (even if the Maya attributes are not locked explicitly).

Rather than start adjusting the Solver UI and underlying maya.cmds.mmSolver() command, I would start by adding a new Maya command (MPxCommand class) that does what you need. This will help with understanding the use case and allow refactoring and re-using the underlying code in the mmSolver() command, without affecting the normal mmSolver() command.

Once the command is worked out and the usage is clear, I'd then consider how to combine this with the maya.cmds.mmSolver() command - if it is possible at all, and then the Solver UI.

Ideally I'd like to combine such a feature into the core of mmSolver, I am just fearful the code will become bloated even more than it is.

Finally, for performance, I'd hope MM SceneGraph (#114) could be utilised, although it currently only computes 2D reprojection error, but as you mentioned 3D-to-3D error could be added fairly easily. MM SceneGraph would definitely fail for complex Maya rigs, however for more simple 3D hierarchies, or no hierarchy at all, it would provide significant performance increases.

---

One use-case for this type of feature would be to align two 3D objects, such as two models of a building. We may have only parts of each 3D model (but the models must have overlapping features).

[ldmoser]

Hi! thank you again for looking closely into this!

I understand your concern about changing the meaning of marker. I don't disagree, but I also see that currently "bundle" is the thing that needs to match, and "marker" is the thing they match against. By using bundle for both uses also sort of changes the meaning of things. In my mind the shelf buttons that allow to switch between marker and corresponding bundle and vice-versa are exploring that relationship of source/target. If we do have "3D" markers that relationship and UI would remain.

In my mind the problem one could be solving could involve 2D and 3D constraints at the same time, so creating a new command only for 3D constraints would not necessarily address interesting use cases. Indeed, if only 3D constraints are required one could fork the project and specialize things for that case I would imagine.

I was not aware of the SceneGraph. Does it replicate the functionality of maya in order to accelerate computations on the graph? Does it mean the Maya setup can't use arbitrary types of nodes in the scene (only the types supported by the SceneGraph)? I was running with the assumption that the Maya setup could be quite complex, including with custom nodes, driving the bundles/cameras/markers.

[david-cattermole]

Hello @ldmoser,

Sorry for the late response, I've been thinking about this for a few days.

Before I continue I should say that I'm unable to implement this feature myself at the moment, for numerous reasons, but PRs are very welcome and I can offer advice to make this happen.

---

Regarding "3D Markers": I see your point, and I'm warming to the idea. I'm still not totally 100% sure how to implement it for users, to reduce confusion. To get a working implementation I think we can disregard the name of the objects for the user, for the moment.

---

In my mind the problem one could be solving could involve 2D and 3D constraints at the same time, so creating a new command only for 3D constraints would not necessarily address interesting use cases.

Yes, that would be an interesting use-case to support, and could be quite powerful - (not exactly sure how), but it would also be easier for users since they don't need to "enable" or "switch' the features. I was mostly suggesting a separate command as a proof of concept first, but if it's easy to change the underlying solver command (maya.cmds.mmSolver()), then it's fine to drop the proof-of-concept phase.

Indeed, if only 3D constraints are required one could fork the project and specialize things for that case I would imagine.

Obviously I'd prefer to avoid that, if possible ;) ... but that is one way forward.

---

And perhaps, in order to unify 2D and 3D constraints there could be a user parameter that scales the 3D losses to a compatible unit. For example, if 3D locations are in cm, this scalar perhaps should be 10 to make the errors in 3D be about mm, and perhaps be more comparable to the errors from 2D markers (in pixels).

Sure this should be simple to simply scale things for the "deviation" units. I think adding a user preference for this is easiest, simply with a scale factor value. Where to store this value can be worked out later.

---

I was not aware of the SceneGraph. Does it replicate the functionality of maya in order to accelerate computations on the graph?

MM SceneGraph will only be available in the upcoming v0.4.0 release (on the develop branch - which is currently in an unstable state). Yes, the purpose is to speed up computations. Most of the solving time is currently spent updating and evaluating the Maya DAG. Initial tests suggest the performance increase is quite drastic.

Does it mean the Maya setup can't use arbitrary types of nodes in the scene (only the types supported by the SceneGraph)? I was running with the assumption that the Maya setup could be quite complex, including with custom nodes, driving the bundles/cameras/markers.

If MM SceneGraph is used, then only a sub-set of the Maya DAG can be replicated. The idea is that if custom nodes are detected the Maya DAG will be used as a fall-back.

MM Scene Graph might not be be important to you really. If you use the general functions it will replicate the exact same behavior, and the user of the function won't detect the difference. If you dig around in the solver core source code on the develop branch you'll see: https://github.com/david-cattermole/mayaMatchMoveSolver/blob/develop/src/core/bundleAdjust_solveFunc.cpp#L308

---

The feature you are talking about requires the following:

• Two 3D transforms that are "linked" in some way.
• A (floating point) weight value (for flexibility, an attribute with the value is better, since we can animate and control it as needed).

The simplest way to add this feature to the solver core is add a new flag to the maya.cmds.mmSolver() command, such as this:

tfm_a = "transform1"
tfm_b = "transform2"
weight_node_attr = tfm_a + '.weight'
maya.cmds.mmSolver(newFlagName=(tfm_a, tfm_b, weight_node_attr))

("newFlagName" is a placeholder name)

The mmSolver command internals can then convert these transforms into data structures that can be solved with. Attributes should be added to the mmSolver command as usual.

# following on from above...
adjust_node_attr = tfm_b + '.translateX'
attr_min_value = -1.0
attr_max_value = 1.0
attr_offset_value = 0.0
attr_scale_value = 1.0
maya.cmds.mmSolver(
    distanceBetween=(tfm_a, tfm_b, weight_node_attr), 
    attr=(adjust_node_attr, attr_min_value, attr_max_value, attr_offset_value, attr_scale_value))

See the maya.cmds.mmSolver() command reference here:
https://david-cattermole.github.io/mayaMatchMoveSolver/commands.html#mmsolver-command

After the maya.cmds.mmSolver() has been updated with the needed flags the other parts that need to be worked out or updated are:

• mmSolver API ./python/mmSolver/_api/
  • the "solver step" and solver sub-classes.
  • the "marker"
• Solver UI: ./python/mmSolver/tools/solver/
• ... maybe user tools to help users use the new feature.

If you work on the develop_v0.3.x or master branches, I'll merge/port to the develop branch, and hope things aren't very different.

It's all a bit detailed, but hopefully it gives you some ideas.
David

[ldmoser]

Hi!
I'm not sure when I would have the time to try it out. Thank you for the detailed answer, it will definitely help speed things up!

[StanleyBalu]

I like this idea; I've been thinking about this type of feature for a while. I'm just putting my thoughts here. I believe we will need to introduce a Bundle with a different name (3d constrain bundle), and each bundle will require another 3d bundle linked to it. Like a source and a destination. All of the source bundles can be parented to the source object and solved to transform the source object to snap it closer to the destination bundles/object in 3D space (just in perspective, no camera required), similar to a 3 point snap. I just remember using a similar tool. Which was extremely useful for stitching lidars, aligning lidar and camera to 3d environments for layout task, and snapping objects with different topologies. I'm not sure of the difficulty in getting this feature added. But it would definitely be a good feature to have.