Performing a whole solution analysis in VS2019

[pavanupb]

I am developing an analyzer where I need to report diagnostics from a list after compiling the entire solution in VS 2019. I registered a CompilationEndAction in my CompilationStartAction but the breakpoint at my CompilationEndAction is not getting hit. I have checked the "Enable full solution analysis" options in VS 2019. Is there a different way to enable full solution analysis?. I am trying to perform a taint analysis and adding tainted values to a list. I would like to report diagnostics for all the tainted values in the list after compiling the entire solution. Is there any other way apart from using the CompilationEndAction?

[mavasani]

"Enable full solution analysis" options in VS 2019

Just to confirm - you have selected "Entire solution" for the radio buttons here: https://docs.microsoft.com/visualstudio/code-quality/configure-live-code-analysis-scope-managed-code?view=vs-2019#custom-analysis-scope

[pavanupb]

I am using VS 2019 community edition. I do not see the radio buttons as mentioned in : https://docs.microsoft.com/visualstudio/code-quality/configure-live-code-analysis-scope-managed-code?view=vs-2019#custom-analysis-scope.
I am able to see an option as shown below in my VS 2019

[mavasani]

Which version of VS2019 are you using? Can you update to latest release or prerelease as per https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes or https://docs.microsoft.com/en-us/visualstudio/releases/2019/release-notes-preview?

[pavanupb]

I am currently using version 16.4.3

[pavanupb]

I installed the latest version. Now I am able to see the radio buttons. I activated it and I am still facing issues while reporting diagnostics when rebuilding the entire solution.
For instance, I have a program

void abc()
{
     String test1 = "Hard Code String 1";
     String test2 = "Hard-Coded String 2";

     foo(test1, test2)
}

void foo(String value1, value2)
{
   String string1 = value1;
   String string2 = value2;
}

I am performing a taint analysis along with alias analysis, where my analyzer reports diagnostics for all tainted values. Method abc() is defined in one project and foo() is defined in another project. I am using the analysis data collected while analyzing abc() in method foo(). When I open each document individually I see the diagnostics being reported correctly. But when I rebuild the solution with the "EntireSolution" radio button selected, I do not see all the diagnostics being reported.

[mavasani]

I am performing a taint analysis along with alias analysis, where my analyzer reports diagnostics for all tainted values.

Just to confirm, you are reporting diagnostics with compilation end action correct?

But when I rebuild the solution with the "EntireSolution" radio button selected, I do not see all the diagnostics being reported.

Are you suggesting that the command line build output does not report these diagnostics? Can you please provide details on each of the below actions:

1. Invoke a command line build outside of VS
2. Invoke a build command in VS and look at output window text
3. Invoke a build command in VS and look at error list

[mavasani]

Method abc() is defined in one project and foo() is defined in another project. I am using the analysis data collected while analyzing abc() in method foo().

So, are you doing cross project/assembly analysis? Seems like your analyzer is crossing compilation/project boundaries, which means it cannot work reliably on build as each project is compiled separately on command line build - there is no concept of solution on command line, each project is compiled separately, with analyzer instances not being shared.

[pavanupb]

I am performing a taint analysis along with alias analysis, where my analyzer reports diagnostics for all tainted values.

Just to confirm, you are reporting diagnostics with compilation end action correct?

But when I rebuild the solution with the "EntireSolution" radio button selected, I do not see all the diagnostics being reported.

Are you suggesting that the command line build output does not report these diagnostics? Can you please provide details on each of the below actions:

1. Invoke a command line build outside of VS
2. Invoke a build command in VS and look at output window text
3. Invoke a build command in VS and look at error list

Right now I am not using the compilation end action as it is not getting executed. So I am reporting the diagnostics inline while analyzing the code. i.e, I am reporting the diagnostics in the registered syntax node actions. Since I am using a dictionary to store the context information of other code blocks, would that create a problem while performing analysis for the whole solution?.

Regarding points 2 and 3, I am able to see the same behavior. However, when I build the project several times, I see at some point all the diagnostics being reported.

Regarding point 1, I haven't tried out since I am using the VSIX installer and did not try installing the nuget package for the project.

[pavanupb]

Method abc() is defined in one project and foo() is defined in another project. I am using the analysis data collected while analyzing abc() in method foo().

So, are you doing cross project/assembly analysis? Seems like your analyzer is crossing compilation/project boundaries, which means it cannot work reliably on build as each project is compiled separately on command line build - there is no concept of solution on command line, each project is compiled separately, with analyzer instances not being shared.

Okay. I also tried within the same project. When I build my project in VS2019 with the analyze the whole solution option enabled, I see that some of the diagnostics are not getting reported. But when I open a particular document I am able to see the diagnostics getting reported correctly with the red/green squiggles.

[pavanupb]

I created a nuget package of my analyzer and tried to build it from the command line. I am getting the error : CSC : warning CS8032: An instance of the analyzer cannot be created. Exception has been thrown by the target of an invocation. I am using "Microsoft.CodeAnalysis.Analyzers" v2.6.2 and "Microsoft.CodeAnalysis.CSharp.Workspaces" v2.10. I went through several posts raised on the same issue, but could not solve the warning. Do you have any suggestions on the same?
My analyzer contains the following packages:

<ItemGroup>
    <PackageReference Include="Microsoft.Bcl.AsyncInterfaces" Version="1.1.0" />
    <PackageReference Include="Microsoft.CodeAnalysis.Analyzers" Version="2.6.2" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="2.10" PrivateAssets="all" />
    <PackageReference Include="Microsoft.CodeAnalysis.FlowAnalysis.Utilities" Version="2.9.6" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="3.1.0" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="3.1.0" />    
  </ItemGroup>

When performing a whole solution analysis, how is Roslyn considering the start point?. Does Roslyn start the analysis from the file containing the main() method?

[pavanupb]

I am performing a taint analysis along with alias analysis, where my analyzer reports diagnostics for all tainted values.

Just to confirm, you are reporting diagnostics with compilation end action correct?

But when I rebuild the solution with the "EntireSolution" radio button selected, I do not see all the diagnostics being reported.

Are you suggesting that the command line build output does not report these diagnostics? Can you please provide details on each of the below actions:

1. Invoke a command line build outside of VS
2. Invoke a build command in VS and look at output window text
3. Invoke a build command in VS and look at error list

Right now I am not using the compilation end action as it is not getting executed. So I am reporting the diagnostics inline while analyzing the code. i.e, I am reporting the diagnostics in the registered syntax node actions. Since I am using a dictionary to store the context information of other code blocks, would that create a problem while performing analysis for the whole solution?.

Regarding points 2 and 3, I am able to see the same behavior. However, when I build the project several times, I see at some point all the diagnostics being reported.

Regarding point 1, I haven't tried out since I am using the VSIX installer and did not try installing the nuget package for the project.

@mavasani Tried with CompilationEndAction. But still, I am observing the same behavior. When I go to individual files I see all the diagnostics being reported correctly but when I build the project I don't see all the diagnostics being reported.

[mavasani]

@mavasani Tried with CompilationEndAction. But still, I am observing the same behavior. When I go to individual files I see all the diagnostics being reported correctly but when I build the project I don't see all the diagnostics being reported.

@pavanupb Can you confirm if your analysis is not relying on any analysis results from outside the current compilation/project?

[mavasani]

when I build the project I don't see all the diagnostics being reported.

Can you clarify if you mean build the project in VS or command line? If the former, are you looking at diagnostics in error list or output window?

[pavanupb]

when I build the project I don't see all the diagnostics being reported.

Can you clarify if you mean build the project in VS or command line? If the former, are you looking at diagnostics in error list or output window?

I am building the project in VS and I am looking at the error list window

[pavanupb]

@mavasani Tried with CompilationEndAction. But still, I am observing the same behavior. When I go to individual files I see all the diagnostics being reported correctly but when I build the project I don't see all the diagnostics being reported.

@pavanupb Can you confirm if your analysis is not relying on any analysis results from outside the current compilation/project?

I am not relying on any analysis result outside the current project/current compilation.

[pavanupb]

Created a simple example for a demonstration. The variables returned from the callee is used in the caller at line https://github.com/pavanupb/SecureCodeAnalyzer/blob/0fe7ae7440cc69d6a1294c787bd6aff021f11da3/Example-Code-3/AliasAnalysisTest/Program.cs#L13. The points-to analysis returns only for AliasAnalysisTest and for the parameter args at line https://github.com/pavanupb/SecureCodeAnalyzer/blob/0fe7ae7440cc69d6a1294c787bd6aff021f11da3/Example-Code-3/AliasAnalysisTest/Program.cs#L7.
Also performed a copy analysis. The copy analysis contains the parameter args at line https://github.com/pavanupb/SecureCodeAnalyzer/blob/0fe7ae7440cc69d6a1294c787bd6aff021f11da3/Example-Code-3/AliasAnalysisTest/Program.cs#L7 with a kind "KnowReferenceCopy". Even though the parameter args is not being used why is it present as a "KnownReferenceCopy"

[mavasani]

You will need to be more clear on what you are expecting in the points to set. Strings are tracked as value copies, so will not be part of points-to-set. Where is GetValues defined? If it is outside the current project/assembly boundary, then no interprocedural analysis is performed as analyzers can only look at sources inside the current compilation.

[pavanupb]

You will need to be more clear on what you are expecting in the points to set. Strings are tracked as value copies, so will not be part of points-to-set. Where is GetValues defined? If it is outside the current project/assembly boundary, then no interprocedural analysis is performed as analyzers can only look at sources inside the current compilation.

The GetValues is defined within the same project.

[mavasani]

The GetValues is defined within the same project.

Would you be able to create a small unit test to demonstrate the issue? For example, see some tests at https://github.com/dotnet/roslyn-analyzers/blob/master/src/Utilities.UnitTests/FlowAnalysis/Analysis/PropertySetAnalysis/PropertySetAnalysisTests.cs. If you can create a unit test with your expected points-to-set for your example, and file an issue on roslyn-analyzers repo, it will be much more actionable and easier to investigate.

[pavanupb]

The GetValues is defined within the same project.

Would you be able to create a small unit test to demonstrate the issue? For example, see some tests at https://github.com/dotnet/roslyn-analyzers/blob/master/src/Utilities.UnitTests/FlowAnalysis/Analysis/PropertySetAnalysis/PropertySetAnalysisTests.cs. If you can create a unit test with your expected points-to-set for your example, and file an issue on roslyn-analyzers repo, it will be much more actionable and easier to investigate.

Sure will create a unit test with my expected points-to set.

[pavanupb]

Does Roslyn perform a points-to analysis for a method present in a different assembly? For instance at line:https://github.com/pavanupb/SecureCodeAnalyzer/blob/5547ca268e8c4569226362ab2183481d7ad0ac76/Example-Code-2/AesEncOnly/AesEncryption.cs#L27

[mavasani]

No, there is no interprocedural analysis outside the current source assembly.

[pavanupb]

Is the interprocedural analysis conducted something similar to a call strings approach(value-based approach) where each callee is analyzed as a separate context limiting this context length to 3?

[mavasani]

where each callee is analyzed as a separate context

Each callee is analyzed with context from the call site. There is a default length of method call that gets analyzed for performance reasons, but this is configurable both by the end users (see here) and from API users (see here and here).

[pavanupb]

Okay. Does the points-to analysis results dictionary mean that a particular allocation site is represented as key and the variables that points-to the allocation site are represented as values?

[mavasani]

Key: https://github.com/dotnet/roslyn-analyzers/blob/master/src/Utilities/FlowAnalysis/FlowAnalysis/Framework/DataFlow/AnalysisEntity.cs#L15-L33

Value: https://github.com/dotnet/roslyn-analyzers/blob/master/src/Utilities/FlowAnalysis/FlowAnalysis/Analysis/PointsToAnalysis/PointsToAbstractValue.cs#L11-L15

[pavanupb]

Does the copy-analysis tell which are the variables that alias with each other? Does copy-analysis as well works only with object properties.?. For instance I have a couple of statements

byte[] secret = { 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16 };
DemoClass demoClass = new DemoClass();
demoClass.ByteValue = secret;  

The Copy-analysis is returning only demoClass.ByteValue as "knowReferenceCopy". I was expecting both demoClass.ByteValue and secret to be of type "KnownReferenceCopy"

[mavasani]

NOTE: Copy analysis is still in experimental phase, hence disabled by default for all analyses/analyzers. I wouldn't be surprised if it was not as precise as your expectations.

[pavanupb]

Ah okay. Thanks

[pavanupb]

@mavasani Is Roslyn using an interprocedural-CFG while performing a points-to analysis?.