Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VSTest retry functionality failes when having spaces or ampersands (&) in TestCase parameter #807

Open
SabotageAndi opened this issue Nov 24, 2020 · 8 comments
Labels

Comments

@SabotageAndi
Copy link

When reporting a bug, please provide the following information to speed up triage:

  • NUnit and NUnit3TestAdapter versions:
    NUnit: 3.12
    NUnit3TestAdapter: 3.15.1
  • Visual Studio edition and full version number (see Help About)
    VS 2019 Enterprise Version 16.8.1
  • A short repro, preferably attached or pointing to a git repo or gist
    https://github.com/SabotageAndi/NUnitRetry2213
  • What .net platform and version is being targeted
    .NET Core 3.1
  • If TFS/VSTS issue, what version, hosted, on-premises, and what build task you see this in
    Azure DevOps/ VSTest Task V2

The Azure DevOps VSTest task has the functionality to rerun failed tests. It looks like it doesn't work when you are using TestCaseAttributes and have spaces or ampersands (&) in the parameters.

Error:

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=NUnitRetry2213.Features.CalculatorFeature.AddTwoNumbers(\"Spa ces\"\,null)"
"D:\a\1\s\NUnitRetry2213\bin\Release\netcoreapp3.1\NUnitRetry2213.dll"
/Settings:"D:\a\_temp\0lawbeaogzy.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"D:\a\1\s"
##[error]Unhandled Exception: System.ArgumentException: Illegal characters in path.
##[error]   at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)
##[error]   at System.IO.Path.IsPathRooted(String path)
##[error]   at vstest.console.Internal.FilePatternParser.GetMatchingFiles(String filePattern)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.CommandLineOptions.AddSource(String source)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.TestSourceArgumentExecutor.Initialize(String argument)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
##[error]   at System.Lazy`1.CreateValue()
##[error]   at System.Lazy`1.LazyInitValue()
##[error]   at System.Lazy`1.get_Value()
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.ArgumentProcessorFactory.<>c__DisplayClass20_0.<WrapLazyProcessorToInitializeOnInstantiation>b__0()
##[error]   at System.Lazy`1.CreateValue()
##[error]   at System.Lazy`1.LazyInitValue()
##[error]   at System.Lazy`1.get_Value()
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.GetArgumentProcessors(String[] args, List`1& processors)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)
##[error]   at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)

Code:

[NUnit.Framework.TestAttribute()]
[NUnit.Framework.DescriptionAttribute("Add two numbers")]
[NUnit.Framework.CategoryAttribute("mytag")]
[NUnit.Framework.TestCaseAttribute("Spa ces", null)]
public virtual void AddTwoNumbers(string column, string[] exampleTags)

The code is generated by SpecFlow, but I think we are generating the code correctly. We could escape the string in SpecFlow when we create the code, but there would nowhere be a normalization of it.

Repro project: https://github.com/SabotageAndi/NUnitRetry2213
GitHub Issue at SpecFlow: SpecFlowOSS/SpecFlow#2213

For me, it looks like there is some escaping missing in the test adapter full qualified name generation for spaces.

@OsirisTerje
Copy link
Member

@SabotageAndi There are known issues around FullyQualifiedNames so this can be another one of those. But, I don't see the adaptercode in the stack trace there, so wonder a bit if this is something in the retry code. A bit strange also that it only happens on retry. If it was the adapter it should have triggered also on the initial run. Can you check more?. E.g. turn on verbosity in the runsettings file, and see if you get anything from the adapter at all.

@OsirisTerje OsirisTerje added Investigate confirm We need a separate confirmation of this issue labels Nov 24, 2020
@SabotageAndi
Copy link
Author

The retry functionality in the tasks works, that it is starting a new vstest.console.exe call with a TestCaseFilter parameter.

This is the second call.

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=NUnitRetry2213.Features.CalculatorFeature.AddTwoNumbers(\"Spa ces\"\,null)"
"D:\a\1\s\NUnitRetry2213\bin\Release\netcoreapp3.1\NUnitRetry2213.dll"
/Settings:"D:\a\_temp\0lawbeaogzy.tmp.runsettings"
/Logger:"trx"
/TestAdapterPath:"D:\a\1\s"

This is the generated filter parameter:

/TestCaseFilter:"FullyQualifiedName=NUnitRetry2213.Features.CalculatorFeature.AddTwoNumbers("Spa ces",null)"

And this is where the FQN generation comes into place. This is the FQN from the first run which gets passed as a parameter to the second run.

@OsirisTerje
Copy link
Member

Thanks! This would certainly trigger the FQN issue.

@fernandozpiccin
Copy link

Is there any fix/workaround for this yet?

@rsuk-mb
Copy link

rsuk-mb commented May 25, 2022

Thanks! This would certainly trigger the FQN issue.

@OsirisTerje Do you think the FQN issue is likely to be actioned at this point? Trying to identify the prospect of a workaround or looming solution as this is preventing us using the retry functionality in DevOps currently as many of our data driven test cases contain problematic characters (e.g. #).

@OsirisTerje
Copy link
Member

@rsuk-mb We're still waiting .... I hope it will be within this year though.

FWIW: I would sanitize the test cases - it should not be too much extra work. You can derive a class from TestCaseData and use that.

@akshaygit09
Copy link

Is there any fix/workaround for this yet?

@OsirisTerje
Copy link
Member

@akshaygit09 Unfortunately no, we haven't heard anything about this for a long time.

@OsirisTerje OsirisTerje added VS Issue and removed confirm We need a separate confirmation of this issue Investigate labels Mar 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants