Test generated by IFixtureBuilder is skipped the first time when its name changes

[loop-evgeny]

NUnit 3.13.2, NUnit3TestAdapter 4.2.1, Visual Studio 2022 Pro, 17.0.5, .NET Core 3.1

I have a custom IFixtureBuilder that generates tests with variable names. I noticed that under the VS test runner any name change is not picked up the first time, but only on the subsequent run. For test names that always change this means they never get run at all! All tests run every time under dotnet test or the ReSharper test runner, however.

To reproduce open NUnitDiscoveryNameChange.zip in VS 2022 and select Test, Run All Tests. The tests output window shows:

========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in C:\W ork\Bugs\NUnitDiscoveryNameChange\bin\Debug\netcoreapp3.1\NUnitDiscoveryNameChange.dll
   NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
Running test: Test with fixed name - always runs

NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 1 Tests (1 Passed, 0 Failed, 0 Skipped) run in 300 ms ==========

Run it again and, depending on your luck, you might have the same output or "Test with random zero or one" may also run:

========== Starting test run ==========
NUnit Adapter 4.2.0.0: Test execution started
Running selected tests in C:\W ork\Bugs\NUnitDiscoveryNameChange\bin\Debug\netcoreapp3.1\NUnitDiscoveryNameChange.dll
   NUnit3TestExecutor discovered 2 of 2 NUnit test cases using Current Discovery mode, Non-Explicit run
Running test: Test with fixed name - always runs

Running test: Test with random zero or one - sometimes runs: 0

NUnit Adapter 4.2.0.0: Test execution complete
========== Test run finished: 2 Tests (2 Passed, 0 Failed, 0 Skipped) run in 301 ms ==========

[OsirisTerje]

@loop-evgeny Do you have the VS real time discovery option turned on ?

[loop-evgeny]

@OsirisTerje "Discover tests in real time from C# and Visual Basic .NET source files" was checked, yes. I tried unchecking it, but that did not change the behavior described in this issue (even after restarting VS). Tested on VS 17.4.5 now.

[OsirisTerje]

But running with dotnet test, they all run, right ?

[loop-evgeny]

Right.

[OsirisTerje]

The problem here is that the Test Explorer runs both a discovery and an execution. The dotnet test only runs execution. The discovery is so that the tree can be displayed after a build. The discovery phase will reflect over the types involved to detect their FQN names. Those names are the primary key for the display in the Test Explorer.

When that name is changing, the Test Explorer is not able to match what it found during discovery with what it found during execution.

This is the way that the Test Explorer have been working from day one. I am not sure if it would accept the names to be delivered only during execution. I have noticed that Rider does allow this, and trying your code on Rider, all three tests are run as they should.

It could be worth an experiment.