[Build] Dotnet packages on nuget are not built with Release optimizations

[sandrohanea]

Describe the issue

I want to benchmark some application that is using OnnxRuntime in dotnet using BenchmarkDotNet but I receive the following error:

// Validating benchmarks:
//    * Assembly MLBenchmarks which defines benchmarks references non-optimized Microsoft.ML.OnnxRuntime
        If you own this dependency, please, build it in RELEASE.
        If you don't, you can disable this policy by using 'config.WithOptions(ConfigOptions.DisableOptimizationsValidator)'.

I see that you're using RelWithDebInfo:

onnxruntime/.pipelines/windowsai-steps.yml

Line 99 in 6d9636f

configuration: RelWithDebInfo

Any reason for having this non-standard behavior for the package in nuget.org?

Same issue is for all version (1.20.1, 1.20.0. etc)

Urgency

No blockers, also workaround exists.

Target platform

win-x64 (possibly others as well)

Build script

Just create a benchmark using Microsoft.ML.OnnxRuntime and the validator will stop the execution of the benchmark if not disabled.

Error / output

Validating benchmarks:
* Assembly MLBenchmarks which defines benchmarks references non-optimized Microsoft.ML.OnnxRuntime
If you own this dependency, please, build it in RELEASE.
If you don't, you can disable this policy by using 'config.WithOptions(ConfigOptions.DisableOptimizationsValidator)'.

Visual Studio Version

No response

GCC / Compiler Version

No response

[skottmckay]

Microsoft.ML.OnnxRuntime contains native libraries (built from C and C++ code) that are built using the cmake 'RelWithDebInfo' config. That's a release build, but it doesn't throw away the symbols info to make debugging issues easier. AFAIK that does not prevent any optimizations from running.

How is MLBenchmarks determining it's 'non-optimized' given the nuget package contains native libraries?

[sandrohanea]

Hello @skottmckay,

MLBenchmarks is not determining if some dependency is "non-optimized" but BenchmarkDotNet.

MlBenchmarks is just a name of a sample app, but here is a simple repro:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using Microsoft.ML.OnnxRuntime;

BenchmarkRunner.Run<MyBenchmarkOnnx>();

public class MyBenchmarkOnnx
{
    private InferenceSession inferenceSession;

    public MyBenchmarkOnnx()
    {
        this.inferenceSession = new InferenceSession("model.onnx");
    }

    [Benchmark]
    public async void Test()
    {
        await Task.Delay(100);
    }
}

It seems this is what it checks:
https://github.com/dotnet/BenchmarkDotNet/blob/6367ad84177e51a11e93f93237ab1e13b673f55f/src/BenchmarkDotNet/Validators/JitOptimizationsValidator.cs#L39

It's not about the native library but about the dotnet assembly (just the wrapper).

This is what it checks on the assembly:
https://github.com/dotnet/BenchmarkDotNet/blob/6367ad84177e51a11e93f93237ab1e13b673f55f/src/BenchmarkDotNet/Extensions/AssemblyExtensions.cs#L30C104-L30C126

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[sandrohanea]

Hello @skottmckay,

MLBenchmarks is not determining if some dependency is "non-optimized" but BenchmarkDotNet.

MlBenchmarks is just a name of a sample app, but here is a simple repro:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using Microsoft.ML.OnnxRuntime;

BenchmarkRunner.Run<MyBenchmarkOnnx>();

public class MyBenchmarkOnnx
{
    private InferenceSession inferenceSession;

    public MyBenchmarkOnnx()
    {
        this.inferenceSession = new InferenceSession("model.onnx");
    }

    [Benchmark]
    public async void Test()
    {
        await Task.Delay(100);
    }
}

It seems this is what it checks: https://github.com/dotnet/BenchmarkDotNet/blob/6367ad84177e51a11e93f93237ab1e13b673f55f/src/BenchmarkDotNet/Validators/JitOptimizationsValidator.cs#L39

It's not about the native library but about the dotnet assembly (just the wrapper).

This is what it checks on the assembly: https://github.com/dotnet/BenchmarkDotNet/blob/6367ad84177e51a11e93f93237ab1e13b673f55f/src/BenchmarkDotNet/Extensions/AssemblyExtensions.cs#L30C104-L30C126

Commenting as this was marked as stale:
Details were provided above, if there is any other details needed, please, ping me.