Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] Expose use_per_session_threads options in the C# layer #19703

Closed
jw444 opened this issue Feb 28, 2024 · 1 comment
Closed
Assignees
Labels
api:CSharp issues related to the C# API feature request request for unsupported feature or enhancement

Comments

@jw444
Copy link

jw444 commented Feb 28, 2024

Describe the feature request

We host a model inferencing service and we run ONNX model inferences with multiple models (>100) simultaneously. Since by default the runtime uses per session (model) thread pool, this generates many threads and we run into contention issue leading to long inferencing latency.
One potential solution is to use the global thread pool instead of using per session thread pools so that models can share thread management. However, the option, use_per_session_threads, is not available in the C# layer and it's true by default. Could you please add this option in the C# runtime so we can configure it? Thanks.

Describe scenario use case

We run multiple (> 100) ONNX model inferences simultaneously in a web service authored with C# OnnxRuntime.

@jw444 jw444 added the feature request request for unsupported feature or enhancement label Feb 28, 2024
@yuslepukhin yuslepukhin self-assigned this Feb 28, 2024
@pranavsharma pranavsharma added the api:CSharp issues related to the C# API label Feb 28, 2024
@yuslepukhin
Copy link
Member

#19730

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api:CSharp issues related to the C# API feature request request for unsupported feature or enhancement
Projects
None yet
Development

No branches or pull requests

3 participants