ScatterND multithreaded inference

[ghost]

Describe the issue

Hi,

Using a model consisting of a single ScatterND node, and pre-generated input NumPy arrays, we noticed that performing several consecutive inferences (with the same inputs), results in different outputs.

We then found that this does not occur if the InferenceSession runs on a single thread, so this looks like a multithreading issue.

Seems like I cannot share ONNX or NumPy files: the model and data for network inputs will be in the following archive: files.zip.

I you need any additional information, feel free to ask.

Thanks and have a good day

To reproduce

Once again, cannot share Python files, so please find the code in the snippet below.

This showcases the issue: if anything in unclear, once again go ahead and notify me.

import os

import onnxruntime
import numpy as np

def build_input_feed(sess: onnxruntime.InferenceSession) -> dict[str, np.ndarray]:
    input_feed = {
        inp.name: np.load(f"{inp.name}.npy")
        for inp in sess.get_inputs()
    }
    return input_feed


def run_twice_and_compare(sess_options: onnxruntime.SessionOptions = None) -> None:

    sess = onnxruntime.InferenceSession("model.onnx", sess_options=sess_options)

    input_feed = build_input_feed(sess)

    print("Performing 2 successive runs...")
    first_run = sess.run(None, input_feed)
    second_run = sess.run(None, input_feed)

    if not np.array_equal(first_run, second_run):
        print("Arrays differ.")


if __name__ == "__main__":

    print("Running multi threaded...")

    run_twice_and_compare()

    print("Running single threaded...")

    sess_options = onnxruntime.SessionOptions()
    sess_options.intra_op_num_threads = 1
    sess_options.execution_mode = onnxruntime.ExecutionMode.ORT_SEQUENTIAL
    sess_options.add_session_config_entry("session.intra_op.allow_spinning", "1")

    run_twice_and_compare(sess_options)

Urgency

No response

Platform

Linux

OS Version

Fedora 38 (also observed on Windows 10)

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.16.1

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[xadupre]

With multitheading, the order of the operations is not guaranteed. Is it possible input_1 (indices) have duplicates? In that case, the monothreaded version always keeps the last one, the multithreaded version keeps one keeps one of them.

[ghost]

Hi,

Thanks for your answer ! input_1 has duplicates.

We were wondering about duplicate indices, at some point, but ONNX's operators documentation seems to indicate that the duplicate indices should not be an issue when reduction attribute's value is "none" (it is "add", in the shared model).

Admittedly, I don't see why it would make a difference, but it lead us to assume there may have been an issue.