Unit Testable Engine Interactions

[ProbablePrime]

Is your feature request related to a problem? Please describe.

It is often very difficult to validate and verify behaviour within Resonite Development.

We do have unit tests, but these are usually Library Style code that is easy to isolate from the rest of the engine, we can also quite easily test ProtoFlux Nodes.

However, We do not have an easy way to unit test engine operations such as those found in Components or other parts of the engine that make heavy use of the state of the engine.

Describe the solution you'd like

A mechanism colloquially known as a Test Harness, to enable unit testing within a completely running FrooxEngine Instance.

I think we'll need several "Scenarios/Contexts" for tests:

1. Tests which do not require a Resonite World - For things like engine fundementals and loading etc.
2. Tests which require an empty Resonite World - Testing the creation or behavior of components
3. Tests which require a specific Resonite World - Bug isolation etc.
4. Test which require loading a specific asset - Bug isolation and reproduction etc

An example(but highly simplified and trivialized) test, is:

Which is designed to test that the ValueCopy component successfully copies a value from one field to another. This test is incredibly basic, but it is not something we can do right now.

Describe alternatives you've considered

What I currently do which is sometimes to launch Resonite 10-20 times a day 😱

Additional Context

This is a development only feature. So please don't assume it will allow for unit testing your community creations.

Requesters

YDMS Dev Team

[ProbablePrime]

I looked at this yesterday.

I started off in MSTest, but that became hard. We need to highly couple the order of tests, how they're run what happens before and after etc.

To try and mitigate this I moved to XUnit which has slightly better lifecycle controls, but still struggled.

My new plan is to migrate once more to Fixie, which allows for ENTIRELY CUSTOM test execution. For an example see(https://github.com/fixie/fixie/wiki/Customizing-the-Test-Project-Lifecycle#recipe-infrequent-construction).

The benefit of Fixie is that we can smartly, start the engine, start any pre-requisites... run the test and then smartly close down the engine. We can also abstract away items into attributes. So for example:

[UsesWorld("resrec://u-probableprime/cheese")
public async Task TestWorldBug() 

Could be picked up automatically and start up the world for the test, and shut it down afterwards.

[ProbablePrime]

Found: https://github.com/VerifyTests/Verify in my travels too. I want to try using it too.

Its a really cool library, that makes it easy to save and compare snapshotted data. The basic pattern would be:

1. Setup some world/state
2. Do some action
3. Dump the world/area to JSON/another format(verify supports a lot).
4. Compare it to the stored verify state.

This could be used to verify that a certain state is reproducible under a certain set of steps.

So an example use case might be: A bug is discovered that causes a certain 3D model to load "weird" with wrong coordinates or something.

You can verify that the fix for this bug works by creating a test that:

1. Loads the model into a world, from its resrec.
2. Dumps the world state to file that Verify can use.
3. Verify would then be like YUP the expected World State Matches the saved verified world state.

[ProbablePrime]

using FrooxEngine.Tests.Harness;
using Shouldly;

namespace FrooxEngine.Tests;

[FrooxEngineTestClass]
[UseWorld(TestWorldPreset.Blank)]
public class ValueCopyTests
{
    [FrooxEngineTestMethod]
    public void TestValueCopy(FrooxEngineTestContext context)
    {
        var slot = context.RootSlot.AddSlot("Test");
        var field1 = slot.AttachComponent<ValueField<float>>();
        var field2 = slot.AttachComponent<ValueField<float>>();

        var valueCopy = slot.AttachComponent<ValueCopy<float>>();

        field1.Value.Value = 7;
        valueCopy.Source.Target = field1.Value;
        valueCopy.Target.Target = field2.Value;

        context.World.RunInUpdates(1, () =>
        {
            field2.Value.Value.ShouldBe(field1.Value.Value);
        });
    }
}

This isn't successfully running yet and im still making tweaks, but I wanted to share. With Fixie, I've managed to reduce a lot of the clutter that's usually required. There's no await ToWorld();(Tests start in the world context).

I will be replacing the Assertion Library, that by default comes with Fixie though. It makes sense for most cases but due to the Sync infrastructure its making tests really hard to read. So we'll back back to Assert.AreEqual soon or something similar.