Setting Openvino EP to run on one core with one thread

[Darshvino]

Hi ONNXRT team,

How to set Openvino EP to run on one core with one thread?

[mayavijx]

Hi @Darshvino,
You can build onnxruntime from source after adding below lines after https://github.com/microsoft/onnxruntime/blob/master/onnxruntime/core/providers/openvino/backends/basic_backend.cc#L221

  if (global_context_.device_type.find("CPU") != std::string::npos) {
    config["CPU_THREADS_NUM"] = "1";
  }

This will set no. of cpu threads to 1.
Please refer below link for build instruction:
https://onnxruntime.ai/docs/build/eps.html#openvino

[ozancaglayan]

isnt this configurable through provider opts or an env var?

[zhuker]

even why set

    config["CPU_THREADS_NUM"] = "1";

openvino ep still uses 8 threads

My session options config

    options.SetGraphOptimizationLevel(ORT_DISABLE_ALL);
    OrtOpenVINOProviderOptions opts;
    opts.device_type = "CPU_FP32";
    opts.enable_vpu_fast_compile = 0;
    opts.device_id = "";
    opts.num_of_threads = 1;
    opts.enable_opencl_throttling = false;
    options.AppendExecutionProvider_OpenVINO(opts);
    options.SetExecutionMode(ExecutionMode::ORT_SEQUENTIAL);
    options.SetInterOpNumThreads(1);
    options.SetIntraOpNumThreads(1);
    options.EnableCpuMemArena();

tested on rel-1.14.0 branch
openvino 2022.3

[DoubleEspresso]

I am seeing the same behavior. Recompiling from source @mayavijx suggests did not work for me as well.

My session options,

    options.enable_opencl_throttling = 1;
    options.num_of_threads = 1;
    options.device_type = "CPU_FP32";
    sessionOptions.AppendExecutionProvider_OpenVINO(options);

tested on rel-1.15.1 branch
openvino 2022.3

[sspintel]

We've now mapped the num_of_threads provider option to the INFERENCE_NUM_THREADS property of OpenVINO, and by setting it to 1 at runtime you should be able to achieve single thread execution.

The PR: #18596

[weimeng23]

We've now mapped the num_of_threads provider option to the INFERENCE_NUM_THREADS property of OpenVINO, and by setting it to 1 at runtime you should be able to achieve single thread execution.

The PR: #18596

The PR hasn't been released yet?

[sfatimar]

#18596 PR merged