Add support for BenchmarkRunner.RunSource on .NET 5

[jonsequitur]

Currently, RunSource is only supported on .NET Framework. It would be very useful to add support for this for .NET 5.

using BenchmarkDotNet.Running;

var summary =  BenchmarkRunner.RunSource("System.Console.WriteLine(\"hi!\");");

Error: System.NotSupportedException: Supported only on Full .NET Framework
at BenchmarkDotNet.Running.BenchmarkRunner.RunSourceWithDirtyAssemblyResolveHelper(String source, IConfig config)
at BenchmarkDotNet.Running.BenchmarkRunner.<>c__DisplayClass7_0.<RunSource>b__0()
at BenchmarkDotNet.Running.BenchmarkRunner.RunWithExceptionHandling(Func`1 run)
at BenchmarkDotNet.Running.BenchmarkRunner.RunSource(String source, IConfig config)
at Submission#5.<<Initialize>>d__0.MoveNext()
--- End of stack trace from previous location ---
at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray`1 precedingExecutors, Func`2 currentExecutor, StrongBox`1 exceptionHolderOpt, Func`2 catchExceptionOpt, CancellationToken cancellationToken)

[YegorStepanov]

@jonsequitur
I've added support for .NET Core (hasn't sent a PR yet), but the user cannot add the library/package.

BenchmarkRunner.RunSource doesn't treat input string as .csx, which means it doesn't support #r "path-to-lib.dll".

Is #r support required for you?
How would the BDN ideally work in dotnet-interactive and what needs to be done on the BDN side?

P.S BDN print a lot of logs at the moment. I can send you a workaround on how to display only validation errors and a table.
Although it is better to wait until the #190 is closed.

[jonsequitur]

#r support isn't needed, and I would expect that benchmarking C# script code would add noise to the benchmarks anyway.

Is there an API for adding assembly references that the benchmarked code can depend on? If so, then we can use .NET Interactive to acquire the dependencies (including from NuGet) and provide the paths to the needed assemblies.

[YegorStepanov]

#r support isn't needed, and I would expect that benchmarking C# script code would add noise to the benchmarks anyway.

This is good because it would require a dependency (maintainers don't like to add dependencies)

The generated project from RunSource compiles with Release, but you want to pass Debug assemblies, don't you?
I expect code from RunSource will be optimized, but code from external assemblies that you import will not optimized (I could be wrong)

Is there an API for adding assembly references that the benchmarked code can depend on?

We can add optional parameter:

1. IEnumerable<MetadataReference> additionalReferences:
   We don't have it in .NET Framework (the old framework expects the paths to assemblies).
2. IEnumerable<Assembly> additionalAssemblies
   As I understand, you are working with in-memory assemblies, so we can't create MetadataReference like:

MetadataReference.CreateFromFile(assembly.Location); //assembly.Location is null

Possible workaround: https://stackoverflow.com/a/54371960

#!release

Is it hard to add #!release to dotnet-repl?

#benchmark

Do you want to add #benchmark smth command?

[jonsequitur]

The generated project from RunSource compiles with Release, but you want to pass Debug assemblies, don't you?

I'm not interested in benchmarking debug assemblies. I'm thinking of the notebook or REPL as a way to provide a UX for writing code to be benchmarked, but not as the compilation environment for the benchmark.

Is it hard to add #!release to dotnet-repl?

It's not something we've looked into much and not currently a goal.

Do you want to add #benchmark smth command?

The idea I had in mind was to provide a magic command that benchmarks the code within a cell rather than directly running it. But also allowing people to benchmark entirely external code (e.g. in a project) might also be nice because of the notebook's usefulness in visualizing and sharing results.

[YegorStepanov]

I don't think anything needs to be done on the BDN side as it's hard to get this working in .NET Core.

Step 1:

Add package that scans bin folder:

<PackageReference Include="Microsoft.Extensions.DependencyModel" Version="7.0.0" />

Step 2:

Copy the reference assemblies to bin/Release/net7.0/ref folder (adds 5mb for empty BDN project)

<PreserveCompilationContext>true</PreserveCompilationContext>

Now, you can run it as following:

var assemblyPaths = DependencyContext.Default.CompileLibraries
    .Where(l => l.Assemblies.Count != 0)
    .SelectMany(l => l.ResolveReferencePaths()).ToArray();

var references = assemblyPaths
    .Select(path => MetadataReference.CreateFromFile(path))
    .Cast<MetadataReference>()
    .ToArray();

string source = ...;
var executingAssembly = Assembly.GetExecutingAssembly();
BenchmarkRunner.Run(source, executingAssembly, references);

Not very convenient API.

It would be easier if you implemented it completely on your side.

Full code:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Environments;
using BenchmarkDotNet.Loggers;
using BenchmarkDotNet.Running;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.Extensions.DependencyModel;
using TypeInfo = System.Reflection.TypeInfo;

public class Program
{
    public static void Main(string[] args)
    {
        // uses <PackageReference Include="MathNet.Numerics" Version="5.0.0" />
        string source = @"
using System;
using System.Threading.Tasks;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Columns;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Reports;
using BenchmarkDotNet.Running;

[ShortRunJob(RuntimeMoniker.Net60)]
[MemoryDiagnoser]
public class MyBenchmarks
{
    [Benchmark] public double B1() => MathNet.Numerics.SpecialFunctions.Beta(1, 2);
}
";
        var assemblyPaths = DependencyContext.Default.CompileLibraries
            .Where(l => l.Assemblies.Count != 0)
            .SelectMany(l => l.ResolveReferencePaths()).ToArray();

        var references = assemblyPaths
            .Select(path => MetadataReference.CreateFromFile(path))
            .Cast<MetadataReference>()
            .ToArray();

        // it should be Assembly.GetEntryAssembly(), but it returns null when:
        // 1) calling from unmanaged code
        // 2) calling from unit test framework? (https://stackoverflow.com/a/53228317)
        // 3) debugging?
        var executingAssembly = Assembly.GetExecutingAssembly();
        var assemblyName = AssemblyName.GetAssemblyName(executingAssembly.Location).Name + Postfix; //ThisProject__GeneratedProject
        var runInfos = GetBenchmarksFromSource(source, assemblyName, references);

        //todo: copy ThisProject.csproj to ThisProject__GeneratedProject.csproj

        BenchmarkRunner.Run(runInfos);
    }

    private const string Postfix = "__GeneratedProject";

    public static BenchmarkRunInfo[] GetBenchmarksFromSource(string source, string assemblyName, IReadOnlyCollection<MetadataReference> assemblyReferences, IConfig config = null)
    {
        if (source == null) throw new ArgumentNullException(nameof(source));

        var logger = HostEnvironmentInfo.FallbackLogger;
        var compiledAssembly = CompileAssembly(source, assemblyName, assemblyReferences, logger);

        if (compiledAssembly == null)
            return Array.Empty<BenchmarkRunInfo>();

        return compiledAssembly.GetTypes()
            .Where(type => BenchmarkHelper.ContainsRunnableBenchmarks(type))
            .Select(type => BenchmarkConverter.TypeToBenchmarks(type, config))
            .Select(runInfo => new BenchmarkRunInfo(
                runInfo.BenchmarksCases.Select(b => BenchmarkWithSource(b, source)).ToArray(),
                runInfo.Type,
                runInfo.Config))
            .ToArray();

        //// Version 2: simpler solution
        // 
        // return compiledAssembly.GetTypes()
        //     .Select(type =>
        //     {
        //         try
        //         {
        //             return BenchmarkConverter.TypeToBenchmarks(type, config);
        //         }
        //         catch
        //         {
        //             return null;
        //         }
        //     })
        //     .Where(runInfo => runInfo != null && runInfo.BenchmarksCases.Length != 0)
        //     .ToArray();

        static BenchmarkCase BenchmarkWithSource(BenchmarkCase b, string additionalLogic)
        {
            return BenchmarkCase.Create(
                new Descriptor(
                    b.Descriptor.Type,
                    b.Descriptor.WorkloadMethod,
                    b.Descriptor.GlobalSetupMethod,
                    b.Descriptor.GlobalCleanupMethod,
                    b.Descriptor.IterationSetupMethod,
                    b.Descriptor.IterationCleanupMethod,
                    b.Descriptor.WorkloadMethodDisplayInfo,
                    additionalLogic,
                    b.Descriptor.Baseline,
                    b.Descriptor.Categories,
                    b.Descriptor.OperationsPerInvoke),
                b.Job,
                b.Parameters,
                b.Config);
        }
    }

    private static Assembly CompileAssembly(string benchmarkContent, string assemblyName, IReadOnlyCollection<MetadataReference> references, ILogger logger)
    {
        var compilationOptions = new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary)
            .WithOptimizationLevel(OptimizationLevel.Release)
            .WithAllowUnsafe(true);

        var parseOptions = new CSharpParseOptions(LanguageVersion.Preview, DocumentationMode.Parse, SourceCodeKind.Regular);
        var syntaxTree = CSharpSyntaxTree.ParseText(benchmarkContent, parseOptions);

        var compilation = CSharpCompilation.Create(assemblyName, new[] { syntaxTree }, references, compilationOptions);

        string directoryName = Path.GetDirectoryName(typeof(BenchmarkCase).Assembly.Location)
                               ?? throw new DirectoryNotFoundException(typeof(BenchmarkCase).Assembly.Location);

        var outputPath = Path.Combine(directoryName, $"{assemblyName}.dll");

        var compilationResult = compilation.Emit(outputPath);
        if (!compilationResult.Success)
        {
            logger.WriteLine($"Compilation done with errors:");
            foreach (var diagnostic in compilationResult.Diagnostics.Where(diagnostic => diagnostic.Severity == DiagnosticSeverity.Error))
            {
                logger.WriteError($"{diagnostic.Location}{diagnostic.Id}: {diagnostic.GetMessage()}");
            }

            return null;
        }

        return Assembly.LoadFrom(outputPath);
    }
}

public static class BenchmarkHelper
{
    // from ReflectionExtensions.cs
    internal static bool ContainsRunnableBenchmarks(Type type)
    {
        var typeInfo = type.GetTypeInfo();

        if (typeInfo.IsAbstract
            || typeInfo.IsSealed
            || typeInfo.IsNotPublic
            || typeInfo.IsGenericType && !IsRunnableGenericType(typeInfo))
            return false;

        return GetBenchmarks(typeInfo).Any();
    }

    private static bool IsRunnableGenericType(TypeInfo typeInfo)
        => // if it is an open generic - there must be GenericBenchmark attributes
            (!typeInfo.IsGenericTypeDefinition || typeInfo.GenericTypeArguments.Any() || typeInfo.GetCustomAttributes(true).OfType<GenericTypeArgumentsAttribute>().Any())
            && typeInfo.DeclaredConstructors.Any(ctor => ctor.IsPublic && ctor.GetParameters().Length == 0); // we need public parameterless ctor to create it

    private static MethodInfo[] GetBenchmarks(TypeInfo typeInfo)
        => typeInfo
            .GetMethods(BindingFlags.Instance | BindingFlags.Public | BindingFlags.Static) // we allow for Static now to produce a nice Validator warning later
            .Where(method => method.GetCustomAttributes(true).OfType<BenchmarkAttribute>().Any())
            .ToArray();
}

You need to copy .csproj because of this:

BenchmarkDotNet/src/BenchmarkDotNet/Toolchains/CsProj/CsProjGenerator.cs

Lines 154 to 161 in 28a8e78

	// important assumption! project's file name === output dll name
	string projectName = benchmarkTarget.GetTypeInfo().Assembly.GetName().Name;
	var possibleNames = new HashSet<string> { $"{projectName}.csproj", $"{projectName}.fsproj", $"{projectName}.vbproj" };
	var projectFiles = rootDirectory
	.EnumerateFiles("*proj", SearchOption.AllDirectories)
	.Where(file => possibleNames.Contains(file.Name))
	.ToArray();

[YegorStepanov]

I suggest deprecating or removing the UrlToBenchmarks and SourceToBenchmarks methods:

1. working only on .NET Framework/Windows
2. It doesn't support external references (only default .NET Framework libraries)
3. It uses C# 7.3
4. It's not popular/convenient
5. It's very hard to get .NET Core version (see comment above)

[AndreyAkinshin]

I suggest deprecating or removing the UrlToBenchmarks and SourceToBenchmarks methods:

@YegorStepanov I agree. These were experimental methods that were added in January 2016, but it seems that nowadays they are not useful or popular. Also, they have a limited compatibility scope and it's hard to maintain them. I think we should deprecate them in the next release and remove them in one of the future releases.

[adamsitnik]

I think we should deprecate them in the next release and remove them in one of the future releases.

👍 💯 (I really don't like these methods ;) )

[YegorStepanov]

I shared the code for runtime/interactive.

This issue can be closed.