HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test improvements #108

debloip-adsk · 2024-03-25T19:40:21Z

This PR contains the following changes :

Fixed MtoA and LookdevX loading
Reworked plugin loading in tests to make it so failing to load a plugin fails the test instead of skipping it. This is to avoid the MtoA/LookdevX situation where the tests weren't working yet reported as passing.
Merged MtohTestCase into MayaHydraBaseTestCase, as the distinction was no longer really useful and was leading to code duplication
Converted absolute paths in test scenes to relative paths
Converted .usd files to .usda for readability

…scene

…gin can't load

…eckForPlugin functions

debloip-adsk · 2024-03-25T20:27:45Z

test/lib/mayaUsd/render/mayaToHydra/README.md

@@ -26,7 +26,7 @@ For example, to run testVisibility on RelWithDebInfo :

 ### Running tests with MayaUSD

-Some tests require the MayaUSD plugin to be loaded in order to be run. If MayaUSD cannot be loaded, those tests will be skipped. In order for the test framework to find and load MayaUSD, the `-DMAYAUSD_LOCATION` CMake flag must be set to your MayaUSD installation directory when building MayaHydra. For convenience, you can use the `--mayausd-location` parameter when using the [build.py](../../../../../build.py) script, which will set the CMake flag for you with the given argument. The specified path should point to the directory where your MayaUSD `.mod` file resides. Note that the test framework will not find MayaUSD using the MAYA_MODULE_PATH environment variable.
+The tests require MayaUSD in order to be run. In order for the test framework to find and load MayaUSD, the `-DMAYAUSD_LOCATION` CMake flag must be set to your MayaUSD installation directory when building MayaHydra. For convenience, you can use the `--mayausd-location` parameter when using the [build.py](../../../../../build.py) script, which will set the CMake flag for you with the given argument. The specified path should point to the directory where your MayaUSD `.mod` file resides. Note that the test framework will not find MayaUSD using the MAYA_MODULE_PATH environment variable.


(see mtohUtils.py for the corresponding code changes) I made it so we load MayaUSD by default for all tests, since we expect MayaHydra to be built with it anyways.

debloip-adsk · 2024-03-25T20:30:32Z

test/lib/mayaUsd/render/mayaToHydra/README.md

-4. Add the following snippet at the bottom of your file :
+2. Create a test class that derives from `mtohUtils.MayaHydraBaseTestCase`.
+3. Add the line `_file = __file__` in your class definition.
+4. If needed, set `_extraPluginsToLoad` to list any plugins that need to be loaded for your test. MayaUSD does not need to be specified, it will be loaded automatically.


(see mtohUtils.py for the corresponding code changes) The _extraPluginsToLoad variable will be used to list any plugins required by the test suite. As opposed to the previous method of skipping a test if it failed to load, this method will try to load the listed plugins, but fail the test suite immediately if one of them fails to load.

debloip-adsk · 2024-03-25T20:38:15Z

test/lib/mayaUsd/render/mayaToHydra/README.md

- By default, the mayaHydra plugin will not be loaded. An easy way to load it is to have your test class derive from `MayaHydraBaseTestCase`, which will automatically load mayaHydra when running tests. Don't forget to add the line `_file = __file__` in your class.
- After loading mayaHydra, the renderer will still be on Viewport 2.0. If you want to use another renderer, your test will have to switch to it. If you want to use HdStorm, an easy way to do it is to have your test class derive from `MayaHydraBaseTestCase` and call `self.setHdStormRenderer()`
+- Before each test, a new file will be opened and the renderer will be switched to Hydra. If you need to switch between renderers, you can use the `self.setHdStormRenderer()` and `self.setViewport2Renderer()` methods.


Removed most of the previous info as it was outdated, and removed the mention of "MayaHydra not being loaded by default unless deriving from MayaHydraBaseTestCase", since rarely (if ever) a test will want to derive from unittest.TestCase directly. I've also adjusted the steps to reflect that.

debloip-adsk · 2024-03-25T20:43:47Z

test/lib/mayaUsd/render/mayaToHydra/README.md

-1. Make your test class derive from `MtohTestCase`.
-2. Don't forget to add the line `_file = __file__` in your class.
-3. Create a folder called [TestName]Test (e.g. for testVisibility, create a folder VisibilityTest)
-4. Put the images you want to use for comparison in that folder
-5. Call `self.assertImagesClose`, `self.assertImagesEqual` and/or `self.assertSnapshotClose` with the file names of the images to compare with.
+1. Make sure your test class derives from `mtohUtils.MayaHydraBaseTestCase` (and has the line `_file = __file__`).


See mtohUtils.py for the corresponding code changes and explanation of merging MtohTestCase into MayaHydraBaseTestCase

debloip-adsk · 2024-03-25T20:45:13Z

test/lib/mayaUsd/render/mayaToHydra/README.md

-Utility files are located under [/test/testUtils](../../../../testUtils/). Note that at the time of writing this (2023/08/16), many of the utils files and their contents were inherited from the USD plugin, and are not all used. Here are some of the main utilities you might find useful :
-
- mtohutils.py :
-    - `checkForMayaUsdPlugin()` will try to load the MayaUsd plugin and return a boolean indicating if the plugin was loaded successfully. By decorating a test case with the following line, you can run the test only if the MayaUsd plugin is found and loaded : 
-    ```@unittest.skipUnless(mtohUtils.checkForMayaUsdPlugin(), "Requires Maya USD Plugin.")```
-    - `MayaHydraBaseTestCase` is a base class you can use for your test classes that will provide you with some useful functionality, such as automatically loading the mayaHydra plugin. It also provides a method to use the HdStorm renderer (`setHdStormRenderer`), and another to create a simple scene with a cube (`makeCubeScene`).
-    - `MtohTestCase` is a class you can use as a base for your test classes that will provide you with image comparison functionality (see [Image comparison](#Image-comparison) section). Note that this class already derives from MayaHydraBaseTestCase, so it will also automatically load the mayaHydra plugin.
+Utility files are located under [/test/testUtils](../../../../testUtils/). Note that at the time of writing this (2023/08/16), many of the utils files and their contents were inherited from the USD plugin, and are not all used.


Removed most info as it is outdated or not very relevant. The plugin loading info has been moved into the steps to add tests + the best practices section

debloip-adsk · 2024-03-25T20:57:23Z

test/testUtils/mtohUtils.py

Most differences here are merging the MtohTestCase-specific code into MayaHydraBaseTestCase. Historically, the MayaHydraBaseTestCase class was a read-only class for tests that didn't output anything, and MtohTestCase was the image-diffing subclass that set up an output directory. However, since code coverage was added, MayaHydraBaseTestCase now also sets up an output directory, which is kind of duplicated from MtohTestCase. The plugin loading rework code would also have needed to be duplicated in both setupClass methods of MayaHydraBaseTestCase and MtohTestCase, so it felt simpler to just merge the two. Additionally, the data-source dumping setup that should be coming soon will also need pretty much the same setup as MtohTestCase, so I feel like there is not much point to keeping separate classes, and just merging them into one will be easier to maintain.

debloip-adsk · 2024-03-25T20:59:06Z

test/testUtils/mtohUtils.py

    _file = None
-
-    DEFAULT_CAM_DIST = 24
+    _extraPluginsToLoad = []


New variable to list what plugins a test suite needs

debloip-adsk · 2024-03-25T21:00:51Z

test/testUtils/mtohUtils.py

+        if MAYAUSD_PLUGIN_NAME not in cls._extraPluginsToLoad:
+            cls._extraPluginsToLoad.append(MAYAUSD_PLUGIN_NAME)
+
+        for pluginToLoad in cls._extraPluginsToLoad:
+            # If a plugin fails to load, the entire test suite will be immediately aborted.
+            # Note that in the case of mtoa, the plugin might load successfully but not
+            # initialize properly, which means issues will only be caught in the actual tests.
+            cmds.loadPlugin(pluginToLoad)


Plugin-loading code that will fail the test suite if one fails to load (with one asterisk, see the comment). We add MayaUSD to the list if it is not specified, since we expect MayaHydra to have been built with it. This allows most tests to not have to define _extraPluginsToLoad if all they need is MayaUSD, which will be quite common.

debloip-adsk · 2024-03-25T21:20:16Z

test/testSamples/testMayaDisplayModes/testMayaDisplayModes.ma

Removed Arnold references as per HYDRA-932. Also removed the USD stage references as there is no associated stage.

debloip-adsk · 2024-03-25T21:45:04Z

FYI @ppt-adsk @vlasovi @lanierd-adsk @roopavr-adsk

lilike-adsk · 2024-03-25T22:23:40Z

test/lib/mayaUsd/render/mayaToHydra/README.md

-3. Create a folder called [TestName]Test (e.g. for testVisibility, create a folder VisibilityTest)
-4. Put the images you want to use for comparison in that folder
-5. Call `self.assertImagesClose`, `self.assertImagesEqual` and/or `self.assertSnapshotClose` with the file names of the images to compare with.
+1. Make sure your test class derives from `mtohUtils.MayaHydraBaseTestCase` (and has the line `_file = __file__`).


side question: is it possible to avoid the copying of "_file = file" each time?

So, this is used in order to set the _inputDir variable, which points to the input folder containing the reference images (or dumped data sources), by using the given file path and test class name. Since the input folders live in the same hierarchy as the test files, __file__ is used to get the path to test file, which can then be used to construct the path to its corresponding input folder. I've tried other means of obtaining the path to the test file/class from the generic test setup code, such as by using inspect.getfile(), but it does not seem to work. With the relative paths between the test setup code and test files being known, we could in theory use these to construct the path to the input folders, but that would need to be changed along with any directory structure change, and we'd also need to account for C++ tests being in a subdirectory. At this point I think it'd be more trouble than it's worth, so I suggest staying with what we have.

lilike-adsk · 2024-03-25T22:46:27Z

test/testUtils/mtohUtils.py

+        cls._testDir = os.path.abspath('.')
+
+        if MAYAUSD_PLUGIN_NAME not in cls._extraPluginsToLoad:
+            cls._extraPluginsToLoad.append(MAYAUSD_PLUGIN_NAME)


Is it better to set _extraPluginsToLoad = [MAYAUSD_PLUGIN_NAME] in MayaHydraBaseTestCase, so the derived test case has an opportunity to not load mayaUSDPlugin if needed?

Following our discussion in the scrum, I suggest leaving it as is currently, and adjust later if needed.

ppt-adsk

Great work, thanks for the cleanup!

test/testUtils/mtohUtils.py

lilike-adsk

Looks good!

…-tests

…remove a bunch of unused imports

debloip-adsk changed the title ~~HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Use relative paths in test scenes~~ HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test framework improvements Mar 25, 2024

debloip-adsk added 14 commits March 25, 2024 16:16

HYDRA-861 : Fix mtoa loading

108205e

HYDRA-861 : Make USD stage path relative in testArnoldLights scene

071ba07

HYDRA-932 : Remove Arnold/mtoa/ai references in testMayaDisplayModes …

7ab3980

…scene

HYDRA-861 : Fix LookdevX loading

76d1aee

HYDRA-861 : Rework plugin loading test infrastructure and fail if plu…

1f19933

…gin can't load

HYDRA-861 : Use relative paths in all test scene files

2fb3745

HYDRA-861 : Kill ADPClientService.exe on test suite end

80f6a95

HYDRA-861 : Add comment for killing ADPClientService

a8cfd80

HYDRA-861 : Update point instance picking test and remove specific ch…

0794aa0

…eckForPlugin functions

HYDRA-861 : Add back skipping testSceneBrowser on OSX

fa519c0

HYDRA-861 : Document force flag usage in taskkill

f889e4f

HYDRA-861 : Add best practices section in test README

5f9e5fb

HYDRA-861 : Document MAYA_MODULE_PATH usage for MtoA

c39d5dd

HYDRA-861 : Re-add existing line about MAYA_MODULE_PATH

18bcfdf

debloip-adsk force-pushed the debloip/HYDRA-861/HYDRA-932/fix-mtoa-loodkevx-tests branch from 776285a to 18bcfdf Compare March 25, 2024 20:16

HYDRA-861 : Add note on what happens before each test

ec4e6c0

debloip-adsk commented Mar 25, 2024

View reviewed changes

debloip-adsk changed the title ~~HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test framework improvements~~ HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test improvements Mar 25, 2024

HYDRA-861 : Add other best practices

758bf27

debloip-adsk marked this pull request as ready for review March 25, 2024 21:14

debloip-adsk self-assigned this Mar 25, 2024

debloip-adsk commented Mar 25, 2024

View reviewed changes

debloip-adsk requested a review from lilike-adsk March 25, 2024 21:44

lilike-adsk reviewed Mar 25, 2024

View reviewed changes

ppt-adsk reviewed Mar 26, 2024

View reviewed changes

test/testUtils/mtohUtils.py Outdated Show resolved Hide resolved

HYDRA-861 : Rename _extraPluginsToLoad to _requiredPlugins

3e1a73d

lilike-adsk previously approved these changes Mar 26, 2024

View reviewed changes

debloip-adsk assigned debloip-adsk and unassigned debloip-adsk Mar 26, 2024

ppt-adsk previously approved these changes Mar 26, 2024

View reviewed changes

debloip-adsk added 2 commits March 26, 2024 14:45

Merge branch 'dev' into debloip/HYDRA-861/HYDRA-932/fix-mtoa-loodkevx…

2772837

…-tests

HYDRA-861 : Adjust testRefinement and testUsdNativeInstancePicking + …

31354e8

…remove a bunch of unused imports

debloip-adsk dismissed stale reviews from ppt-adsk and lilike-adsk via 31354e8 March 26, 2024 19:44

debloip-adsk assigned debloip-adsk and unassigned debloip-adsk Mar 26, 2024

ppt-adsk approved these changes Mar 26, 2024

View reviewed changes

lilike-adsk approved these changes Mar 26, 2024

View reviewed changes

debloip-adsk added ready-for-merge Development process is finished, PR is ready for merge test labels Mar 26, 2024

lilike-adsk merged commit b5e22e1 into dev Mar 26, 2024
10 checks passed

lilike-adsk deleted the debloip/HYDRA-861/HYDRA-932/fix-mtoa-loodkevx-tests branch March 26, 2024 20:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test improvements #108

HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test improvements #108

debloip-adsk commented Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024

debloip-adsk Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024

debloip-adsk Mar 25, 2024

debloip-adsk Mar 25, 2024

debloip-adsk Mar 25, 2024

debloip-adsk Mar 25, 2024

debloip-adsk commented Mar 25, 2024

lilike-adsk Mar 25, 2024

debloip-adsk Mar 26, 2024

lilike-adsk Mar 25, 2024

debloip-adsk Mar 26, 2024

ppt-adsk left a comment

lilike-adsk left a comment

HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test improvements #108

HYDRA-861 & HYDRA-932 : Fix MtoA & LookdevX loading in tests + Related test improvements #108

Conversation

debloip-adsk commented Mar 25, 2024 • edited Loading

debloip-adsk Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

debloip-adsk Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debloip-adsk Mar 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

debloip-adsk commented Mar 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ppt-adsk left a comment

Choose a reason for hiding this comment

lilike-adsk left a comment

Choose a reason for hiding this comment

debloip-adsk commented Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024 •

edited

Loading

debloip-adsk Mar 25, 2024 •

edited

Loading