Segmentation fault during inference

[bwtlt]

Describe the issue

Hello,

I am getting a segmentation fault (SIGSEGV) after calling Ort::detail::SessionImpl::Run with two different programs and two different stack traces. The API calls are made in a wrapper, so the code is identical for the 2 programs.

Backtrace for the first program (the largest):

#0  0x00007f78187723e8 in std::char_traits<char>::compare (__s1=0x42765090 "x.3", __s2=0x7f780e7038f8 "x.3", __n=3) at /usr/include/c++/8/bits/char_traits.h:312
No locals.
#1  0x00007f78187764f6 in std::basic_string_view<char, std::char_traits<char> >::compare (this=0x7f780e702080, __str="x.3") at /usr/include/c++/8/string_view:256
        __rlen = 3
        __ret = 32632
#2  0x00007f7818786a61 in std::operator==<char, std::char_traits<char> > (__x="x.3", __y="x.3") at /usr/include/c++/8/string_view:438
No locals.
#3  0x00007f781877396f in absl::lts_20240116::container_internal::StringEq::operator() (this=0x4231b918, lhs="x.3", rhs="x.3")
    at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/hash_function_defaults.h:88
No locals.
#4  0x00007f78188abad8 in absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::EqualElement<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator()<std::basic_string_view<char, std::char_traits<char> >, std::piecewise_construct_t const&, std::tuple<std::basic_string_view<char, std::char_traits<char> > const&>, std::tuple<onnxruntime::InferenceSession::InputOutputDefMetaData const&> > (this=0x7f780e702270, lhs="x.3") at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/raw_hash_set.h:2814
No locals.
#5  0x00007f78188a9538 in absl::lts_20240116::container_internal::memory_internal::DecomposePairImpl<absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::EqualElement<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::basic_string_view<char, std::char_traits<char> > const&, std::tuple<onnxruntime::InferenceSession::InputOutputDefMetaData const&> > (f=..., p={...})
    at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/container_memory.h:140
        key = "x.3"
#6  0x00007f78188a632a in absl::lts_20240116::container_internal::DecomposePair<absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::EqualElement<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData>&> (f=..., args#0={...})
    at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/container_memory.h:207
No locals.
#7  0x00007f78188a2839 in absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>::apply<absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::EqualElement<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData>&> (f=..., args#0={...}) at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/flat_hash_map.h:591
No locals.
#8  0x00007f781889c83d in absl::lts_20240116::container_internal::hash_policy_traits<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, void>::apply<absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::EqualElement<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData>&, absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData> > (f=..., ts#0={...})
    at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/hash_policy_traits.h:134
No locals.
#9  0x00007f7818892d03 in absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::find<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (this=0x4231b918, key="x.3", 
    hash=5296137216916740558) at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/raw_hash_set.h:2695
        i = 1
        __for_range = @0x7f780e70223e: {<absl::lts_20240116::container_internal::NonIterableBitMask<unsigned short, 16, 0>> = {mask_ = 2}, <No data fields>}
        __for_begin = {<absl::lts_20240116::container_internal::NonIterableBitMask<unsigned short, 16, 0>> = {mask_ = 2}, <No data fields>}
        __for_end = {<absl::lts_20240116::container_internal::NonIterableBitMask<unsigned short, 16, 0>> = {mask_ = 0}, <No data fields>}
        g = {static kWidth = 16, ctrl = {-9187201950435750145, -9187201950435737472}}
        __PRETTY_FUNCTION__ = "absl::lts_20240116::container_internal::raw_hash_set<Policy, Hash, Eq, Alloc>::iterator absl::lts_20240116::container_internal::raw_hash_set<Policy, Hash, Eq, Alloc>::find(absl::lts_20240116::containe"...
        seq = {mask_ = 1, offset_ = 1, index_ = 0}
        slot_ptr = 0x424d0e00
        ctrl = 0x424d0de8
#10 0x00007f7818886c12 in absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::find<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (this=0x4231b918, key="x.3", hash=5296137216916740558) at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/raw_hash_set.h:2714
No locals.
#11 0x00007f781887a3e3 in absl::lts_20240116::container_internal::raw_hash_set<absl::lts_20240116::container_internal::FlatHashMapPolicy<std::basic_string_view<char, std::char_traits<char> >, onnxruntime::InferenceSession::InputOutputDefMetaData>, absl::lts_20240116::container_internal::StringHash, absl::lts_20240116::container_internal::StringEq, std::allocator<std::pair<std::basic_string_view<char, std::char_traits<char> > const, onnxruntime::InferenceSession::InputOutputDefMetaData> > >::find<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (this=0x4231b918, key="x.3") at /home/keranova/libraries/onnxruntime/build/Linux/Debug/_deps/abseil_cpp-src/absl/container/internal/raw_hash_set.h:2720
No locals.
#12 0x00007f781886a846 in onnxruntime::InferenceSession::ValidateInputsOutputs (this=0x4231b090, names=..., feeds_fetches=..., input_output_meta_map=..., arg_type=onnxruntime::ArgType::kInput) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/inference_session.cc:2165
        name = "x.3"
        iter = {inner_ = {<absl::lts_20240116::container_internal::HashSetIteratorGenerationInfoDisabled> = {<No data fields>}, ctrl_ = 0x0, {slot_ = 0x0}}}
        input_output_ml_value = <error reading variable>
        expected_type = 0x7f7818893440 <absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >, unsigned long>::CompressedTuple<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >, unsigned int, true>(std::allocator<std::shared_ptr<onnxruntime::IAllocator> >&&, unsigned int&&)+68>
        i = 0
        __PRETTY_FUNCTION__ = "onnxruntime::common::Status onnxruntime::InferenceSession::ValidateInputsOutputs(gsl::span<const std::__cxx11::basic_string<char> >, gsl::span<const OrtValue>, const InputOutputDefMetaMap&, onnxruntim"...
        is_inputs = true
        input_output_moniker = 0x7f78199df094 "input"
        feed_fetches_moniker = 0x7f78199df0a1 "feed"
        is_sparse_initializer = {__this = 0x4231b090}
        __FUNCTION__ = "ValidateInputsOutputs"
#13 0x00007f781886b8ef in onnxruntime::InferenceSession::ValidateInputs (this=0x4231b090, feed_names=..., feeds=...) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/inference_session.cc:2277
No locals.
#14 0x00007f781886c57e in onnxruntime::InferenceSession::Run (this=0x4231b090, run_options=..., feed_names=..., feeds=..., output_names=..., p_fetches=0x7f780e703850, p_fetches_device_info=0x0) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/inference_session.cc:2459
        _status = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        shrink_memory_arenas = <error reading variable: Cannot access memory at address 0x9>
        info = {feed_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 31 times>}}}}, output_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 31 times>}}}}, feeds_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 43 times>}}}}, fetches_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 43 times>}}}}}
        owned_run_logger = std::unique_ptr<onnxruntime::logging::Logger> = {get() = 0x0}
        run_logger = @0x7f780e7039c0: {logging_manager_ = 0x1, id_ = "\370\070p\016x\177\000\000\003\000\000\000\000\000\000\000x.3\000v\177\000\000\000\211;B\000\000\000\000\004\000\000\000\000\000\000\000p-\000`v\177\000\000@qU\032x\177", '\000' <repeats 18 times>, "\200\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\a\000\000\000\000\000\000\000\220\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\b\t\000`v\177\000\000\177\000\000\000\000\000\000\000\260\000\000\000\000\000\000\000\260\373\377\377\377\377\377\377\002\000\000\000\000\000\000\000\t\000\000\000\062", '\000' <repeats 11 times>, "[\000\000\000w", '\000' <repeats 11 times>, "\001\000\000\000\000\000\000\000x8p\016x\177\000\000"..., min_severity_ = 404965733, filter_user_data_ = 120, max_vlog_level_ = 0}
        sequential_run_lock = std::optional<std::lock_guard<onnxruntime::OrtMutex>> [no contained value]
        device_stream_collection_holder = {session_state_ = 0x0, p_ = std::unique_ptr<onnxruntime::DeviceStreamCollection> = {get() = 0x0}}
        device_stream_collection = 0x7f7660002f40
        feeds_fetches_manager = {device_copy_checks_ = {status = onnxruntime::DeviceCopyCheck::Unknown, input_copy_needed = onnxruntime::DeviceCopyCheck::Unknown, output_copy_needed = onnxruntime::DeviceCopyCheck::Unknown}, feeds_fetches_info_ = {feed_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 31 times>}}}}, output_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 31 times>}}}}, feeds_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 43 times>}}}}, fetches_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 43 times>}}}}}, feeds_device_copy_info_ = std::vector of length 0, capacity 0, fetches_device_copy_info_ = std::vector of length 0, capacity 0}
        exec_providers_to_stop = {storage_ = {static kOptimalInlinedSize = 6, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<onnxruntime::IExecutionProvider*>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<onnxruntime::IExecutionProvider*>, 0, true>> = {<std::allocator<onnxruntime::IExecutionProvider*>> = {<__gnu_cxx::new_allocator<onnxruntime::IExecutionProvider*>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 47 times>}}}}
        arenas_to_shrink = {storage_ = {static kOptimalInlinedSize = 3, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >, 0, true>> = {<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<onnxruntime::IAllocator> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 47 times>}}}}
        tp = {__d = {__r = 0}}
        retval = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        env = @0x7f781a558100: {_vptr.Env = 0x7f781a54e8e0 <vtable for onnxruntime::(anonymous namespace)::PosixEnv+16>, env_time_ = 0x7f781a558140 <onnxruntime::EnvTime::Default()::default_env_time>}
        graph_annotation_id = 0
        graph_annotation_str = ""
        control_spinning = false
        intra_tp = 0x0
        inter_tp = 0x0
        runs_refcounter_and_tp_spin_control = {intra_tp_ = 0x0, inter_tp_ = 0x0, concurrent_num_runs_ = @0x4231b970}
        __FUNCTION__ = "Run"
#15 0x00007f781886de1a in onnxruntime::InferenceSession::Run (this=0x4231b090, run_options=..., feed_names=..., feeds=..., fetch_names=..., fetches=...) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/inference_session.cc:2659
        num_feeds = 1
        num_fetches = 1
        feed_name_vec = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f780e7038f8, allocated_capacity = 3}, inlined = {inlined_data = "\370\070p\016x\177\000\000\003\000\000\000\000\000\000\000x.3\000v\177\000\000\000\211;B\000\000\000"}}}}
        feed_vec = {storage_ = {static kOptimalInlinedSize = 2, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<OrtValue>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<OrtValue>, 0, true>> = {<std::allocator<OrtValue>> = {<__gnu_cxx::new_allocator<OrtValue>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f76600018b0, allocated_capacity = 140146393492896}, inlined = {inlined_data = "\260\030\000`v\177\000\000\240-\000`v\177\000\000@WU\032x\177\000\000\340\070p\016x\177\000\000\"Xw\030x\177\000\000\220;p\016x\177\000"}}}}
        fetch_name_vec = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f780e703888, allocated_capacity = 3}, inlined = {inlined_data = "\210\070p\016x\177\000\000\003\000\000\000\000\000\000\000\065\065\062\000x\177\000\000\300\070p\016x\177\000"}}}}
        fetch_vec = std::vector of length 1, capacity 1 = {{data_ = std::shared_ptr<void> (use count 2, weak count 0) = {get() = 0x7f7660001940}, type_ = 0x7f781a555740 <onnxruntime::TensorTypeBase::Type()::tensor_base>}}
        status = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        fetch_unique_ptrs = {storage_ = {static kOptimalInlinedSize = 6, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >, 0, true>> = {<std::allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >> = {<__gnu_cxx::new_allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 140153615038512}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f78187a9f5c <std::_Tuple_impl<0ul, long*, std::default_delete<long []> >::_M_head(std::_Tuple_impl<0ul, long*, std::default_delete<long []> >&)+24>, allocated_capacity = 140153615038528}, inlined = {inlined_data = "\\\237z\030x\177\000\000@8p\016x\177\000\000\330:p\016x\177\000\000P8p\016x\177\000\000\262\063z\030x\177\000\000p8p\016x\177\000"}}}}
        __PRETTY_FUNCTION__ = "onnxruntime::common::Status onnxruntime::InferenceSession::Run(const RunOptions&, gsl::span<const char* const>, gsl::span<const OrtValue* const>, gsl::span<const char* const>, gsl::span<OrtValue*>)"
#16 0x00007f781877d6c0 in OrtApis::Run (sess=0x4231b090, run_options=0x4231b020, input_names=0x7f780e703d20, input=0x7f780e703c60, input_len=1, output_names=0x7f780e703d28, output_names_len=1, output=0x7f780e703c68) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/onnxruntime_c_api.cc:831
        input_names_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f780e703d20}}
        output_name_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f780e703d28}}
        status = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        session = 0x4231b090
        input_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f780e703c60}}
        output_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f780e703c68}}
#17 0x000000000074a5f0 in Ort::detail::SessionImpl<OrtSession>::Run (output_count=1, output_values=0x7f780e703c68, output_names=0x7f780e703d28, input_count=1, input_values=0x7f780e703c60, input_names=0x7f780e703d20, run_options=..., this=0x41ed5340) at /home/keranova/libraries/onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_api.h:573
        ort_input_values = 0x7f780e703c60
        ort_output_values = 0x7f780e703c68
        ort_input_values = <optimized out>
        ort_output_values = <optimized out>

Backtrace for the second program

#0  0x00007f8a7d61c51e in onnxruntime::ExecuteThePlan(onnxruntime::SessionState const&, gsl::span<int const, 18446744073709551615ul>, gsl::span<OrtValue const, 18446744073709551615ul>, gsl::span<int const, 18446744073709551615ul>, std::vector<OrtValue, std::allocator<OrtValue> >&, std::unordered_map<unsigned long, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtDevice const&, OrtValue&, bool&)>, std::hash<unsigned long>, std::equal_to<unsigned long>, std::allocator<std::pair<unsigned long const, std::function<onnxruntime::common::Status (onnxruntime::TensorShape const&, OrtDevice const&, OrtValue&, bool&)> > > > const&, onnxruntime::logging::Logger const&, onnxruntime::DeviceStreamCollection const*, bool const&, bool, bool) (
    session_state=<error reading variable: Cannot access memory at address 0x7f8a789d3e80>, feed_mlvalue_idxs=<error reading variable: Cannot access memory at address 0x7f8a789d3e70>, 
    feeds=<error reading variable: Cannot access memory at address 0x7f8a789d3e60>, fetch_mlvalue_idxs=..., fetches=std::vector of length 1, capacity 1 = {...}, 
    fetch_allocators=std::unordered_map with 0 elements, logger=..., device_streams=0x0, terminate_flag=@0x4d64b278: false, 
    only_execute_path_to_fetches=<error reading variable: Cannot access memory at address 0x7f8a789d3e5c>, single_thread_mode=<error reading variable: Cannot access memory at address 0x7f8a789d3e58>)
    at /home/keranova/libraries/onnxruntime/onnxruntime/core/framework/sequential_executor.cc:534
        execution_plan = 0x7f8a7ca5bc3d
     <gsl::span<int const, 18446744073709551615ul>::storage_type<gsl::details::extent_type<18446744073709551615ul> >::storage_type<unsigned long>(int const*, unsigned long)+63>
        __FUNCTION__ = "ExecuteThePlan"
        valid_streams = 32650
        ctx = <error reading variable ctx (Cannot access memory at address 0x7f8a789d3f30)>
        session_scope = <error reading variable session_scope (Cannot access memory at address 0x7f8a789d3f20)>
        tp = 0x7f8a7c9c7d52 <absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>::get<1>() const &+24>
#1  0x00007f8a7d5bbf37 in onnxruntime::utils::ExecuteGraphImpl(const onnxruntime::SessionState &, const onnxruntime::FeedsFetchesManager &, gsl::span<OrtValue const, 18446744073709551615>, std::vector<OrtValue, std::allocator<OrtValue> > &, const std::unordered_map<long unsigned int, std::function<onnxruntime::common::Status(const onnxruntime::TensorShape&, const OrtDevice&, OrtValue&, bool&)>, std::hash<long unsigned int>, std::equal_to<long unsigned int>, std::allocator<std::pair<long unsigned int const, std::function<onnxruntime::common::Status(const onnxruntime::TensorShape&, const OrtDevice&, OrtValue&, bool&)> > > > &, ExecutionMode, const bool &, const onnxruntime::logging::Logger &, onnxruntime::DeviceStreamCollection *, bool, onnxruntime::Stream *) (session_state=..., feeds_fetches_manager=..., feeds=..., 
    fetches=std::vector of length 1, capacity 1 = {...}, fetch_allocators=std::unordered_map with 0 elements, execution_mode=ORT_SEQUENTIAL, terminate_flag=@0x4d64b278: false, logger=..., 
    device_stream_collection=0x0, only_execute_path_to_fetches=false, parent_stream=0x0) at /home/keranova/libraries/onnxruntime/onnxruntime/core/framework/utils.cc:658
        status = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x7f8a789d47a0}}
        feeds_fetches_info = @0x7f8a789d4a40: {feed_names = {storage_ = {static kOptimalInlinedSize = 1, 
              metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {
                    value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d4a58, allocated_capacity = 3}, inlined = {
                  inlined_data = "XJ\235x\212\177\000\000\003\000\000\000\000\000\000\000x.3", '\000' <repeats 12 times>}}}}, output_names = {storage_ = {static kOptimalInlinedSize = 1, 
              metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {
                    value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d4a80, allocated_capacity = 3}, inlined = {
                  inlined_data = "\200J\235x\212\177\000\000\003\000\000\000\000\000\000\000\065\065\062", '\000' <repeats 12 times>}}}}, feeds_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, 
              metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {
                  inlined_data = '\000' <repeats 43 times>}}}}, fetches_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, 
              metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x23f, allocated_capacity = 0}, inlined = {
                  inlined_data = "?\002", '\000' <repeats 41 times>}}}}}
        device_copy_checks = @0x7f8a789d4a30: {status = onnxruntime::DeviceCopyCheck::NoCopy, input_copy_needed = onnxruntime::DeviceCopyCheck::NoCopy, 
          output_copy_needed = onnxruntime::DeviceCopyCheck::NoCopy}
        execution_plan = 0x4d7b3440
        is_subgraph = false
        single_thread_mode = true
        __FUNCTION__ = "ExecuteGraphImpl"
#2  0x00007f8a7d5bc9c8 in onnxruntime::utils::ExecuteGraph (session_state=..., feeds_fetches_manager=..., feeds=..., fetches=std::vector of length 1, capacity 1 = {...}, execution_mode=ORT_SEQUENTIAL, terminate_flag=@0x4d64b278: false, logger=..., device_stream_collection_holder=..., only_execute_path_to_fetches=false, parent_stream=0x0) at /home/keranova/libraries/onnxruntime/onnxruntime/core/framework/utils.cc:755
        __FUNCTION__ = "ExecuteGraph"
        device_stream_collection = 0x0
        retval = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x7f8a789d4c00}}
#3  0x00007f8a7d5bcaa2 in onnxruntime::utils::ExecuteGraph (session_state=..., feeds_fetches_manager=..., feeds=..., fetches=std::vector of length 1, capacity 1 = {...}, execution_mode=ORT_SEQUENTIAL, run_options=..., device_stream_collection_holder=..., logger=...) at /home/keranova/libraries/onnxruntime/onnxruntime/core/framework/utils.cc:782
No locals.
#4  0x00007f8a7c995c33 in onnxruntime::InferenceSession::Run (this=0x4d64b2c0, run_options=..., feed_names=..., feeds=..., output_names=..., p_fetches=0x7f8a789d5950, p_fetches_device_info=0x0) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/inference_session.cc:2531
        shrink_memory_arenas = ""
        info = {feed_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d4988, allocated_capacity = 0}, inlined = {inlined_data = "\210I\235x\212\177", '\000' <repeats 11 times>, ".3", '\000' <repeats 12 times>}}}}, output_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d49b0, allocated_capacity = 0}, inlined = {inlined_data = "\260I\235x\212\177", '\000' <repeats 11 times>, "\065\062", '\000' <repeats 12 times>}}}}, feeds_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 43 times>}}}}, fetches_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x23f, allocated_capacity = 0}, inlined = {inlined_data = "?\002", '\000' <repeats 41 times>}}}}}
        owned_run_logger = std::unique_ptr<onnxruntime::logging::Logger> = {get() = 0x7f89d81e6ba0}
        run_logger = @0x7f89d81e6ba0: {logging_manager_ = 0x4d2608c0, id_ = "", min_severity_ = onnxruntime::logging::Severity::kWARNING, filter_user_data_ = false, max_vlog_level_ = 0}
        sequential_run_lock = std::optional<std::lock_guard<onnxruntime::OrtMutex>> [no contained value]
        device_stream_collection_holder = {session_state_ = 0x4d7b30d0, p_ = std::unique_ptr<onnxruntime::DeviceStreamCollection> = {get() = 0x0}}
        device_stream_collection = 0x7f89d85b2d70
        feeds_fetches_manager = {device_copy_checks_ = {status = onnxruntime::DeviceCopyCheck::NoCopy, input_copy_needed = onnxruntime::DeviceCopyCheck::NoCopy, output_copy_needed = onnxruntime::DeviceCopyCheck::NoCopy}, feeds_fetches_info_ = {feed_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d4a58, allocated_capacity = 3}, inlined = {inlined_data = "XJ\235x\212\177\000\000\003\000\000\000\000\000\000\000x.3", '\000' <repeats 12 times>}}}}, output_names = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d4a80, allocated_capacity = 3}, inlined = {inlined_data = "\200J\235x\212\177\000\000\003\000\000\000\000\000\000\000\065\065\062", '\000' <repeats 12 times>}}}}, feeds_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 43 times>}}}}, fetches_mlvalue_idxs = {storage_ = {static kOptimalInlinedSize = 11, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<int>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<int>, 0, true>> = {<std::allocator<int>> = {<__gnu_cxx::new_allocator<int>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x23f, allocated_capacity = 0}, inlined = {inlined_data = "?\002", '\000' <repeats 41 times>}}}}}, feeds_device_copy_info_ = std::vector of length 1, capacity 1 = {{source_device = {static CPU = 0 '\000', static GPU = 1 '\001', static FPGA = 2 '\002', static NPU = 3 '\003', device_type = 0, memory_type = 0, device_id = 0}, target_device = {static CPU = 0 '\000', static GPU = 1 '\001', static FPGA = 2 '\002', static NPU = 3 '\003', device_type = 0, memory_type = 0, device_id = 0}, unique_stream_index_consumes_it = -1}}, fetches_device_copy_info_ = std::vector of length 1, capacity 1 = {{source_device = {static CPU = 0 '\000', static GPU = 1 '\001', static FPGA = 2 '\002', static NPU = 3 '\003', device_type = 0, memory_type = 0, device_id = 0}, target_device = {static CPU = 0 '\000', static GPU = 1 '\001', static FPGA = 2 '\002', static NPU = 3 '\003', device_type = 0, memory_type = 0, device_id = 0}, unique_stream_index_consumes_it = -1}}}
        exec_providers_to_stop = {storage_ = {static kOptimalInlinedSize = 6, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<onnxruntime::IExecutionProvider*>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<onnxruntime::IExecutionProvider*>, 0, true>> = {<std::allocator<onnxruntime::IExecutionProvider*>> = {<__gnu_cxx::new_allocator<onnxruntime::IExecutionProvider*>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x4d64aeb0, allocated_capacity = 0}, inlined = {inlined_data = "\260\256dM", '\000' <repeats 43 times>}}}}
        arenas_to_shrink = {storage_ = {static kOptimalInlinedSize = 3, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >, 0, true>> = {<std::allocator<std::shared_ptr<onnxruntime::IAllocator> >> = {<__gnu_cxx::new_allocator<std::shared_ptr<onnxruntime::IAllocator> >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 0}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x0, allocated_capacity = 0}, inlined = {inlined_data = '\000' <repeats 47 times>}}}}
        tp = {__d = {__r = 0}}
        retval = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        env = @0x7f8a7e681100: {_vptr.Env = 0x7f8a7e6778e0 <vtable for onnxruntime::(anonymous namespace)::PosixEnv+16>, env_time_ = 0x7f8a7e681140 <onnxruntime::EnvTime::Default()::default_env_time>}
        graph_annotation_id = 0
        graph_annotation_str = ""
        control_spinning = false
        intra_tp = 0x0
        inter_tp = 0x0
        runs_refcounter_and_tp_spin_control = {intra_tp_ = 0x0, inter_tp_ = 0x0, concurrent_num_runs_ = @0x4d64bba0}
        __FUNCTION__ = "Run"
#5  0x00007f8a7c996e1a in onnxruntime::InferenceSession::Run (this=0x4d64b2c0, run_options=..., feed_names=..., feeds=..., fetch_names=..., fetches=...) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/inference_session.cc:2659
        num_feeds = 1
        num_fetches = 1
        feed_name_vec = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d59f8, allocated_capacity = 3}, inlined = {inlined_data = "\370Y\235x\212\177\000\000\003\000\000\000\000\000\000\000x.3\000\211\177\000\000P\363\326M\000\000\000"}}}}
        feed_vec = {storage_ = {static kOptimalInlinedSize = 2, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<OrtValue>, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<OrtValue>, 0, true>> = {<std::allocator<OrtValue>> = {<__gnu_cxx::new_allocator<OrtValue>> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f89d81e6c70, allocated_capacity = 140230013119744}, inlined = {inlined_data = "pl\036؉\177\000\000\000m\036؉\177\000\000@\347g~\212\177\000\000\340Y\235x\212\177\000\000\"\350\211|\212\177\000\000\220\\\235x\212\177\000"}}}}
        fetch_name_vec = {storage_ = {static kOptimalInlinedSize = 1, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0, true>> = {<std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<__gnu_cxx::new_allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 2}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a789d5988, allocated_capacity = 3}, inlined = {inlined_data = "\210Y\235x\212\177\000\000\003\000\000\000\000\000\000\000\065\065\062\000\212\177\000\000\300Y\235x\212\177\000"}}}}
        fetch_vec = std::vector of length 1, capacity 1 = {{data_ = std::shared_ptr<void> (use count 2, weak count 0) = {get() = 0x7f89d85b2d90}, type_ = 0x7f8a7e67e740 <onnxruntime::TensorTypeBase::Type()::tensor_base>}}
        status = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        fetch_unique_ptrs = {storage_ = {static kOptimalInlinedSize = 6, metadata_ = {<absl::lts_20240116::container_internal::internal_compressed_tuple::CompressedTupleImpl<absl::lts_20240116::container_internal::CompressedTuple<std::allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >, unsigned long>, std::integer_sequence<unsigned long, 0, 1>, true>> = {<absl::lts_20240116::container_internal::internal_compressed_tuple::uses_inheritance> = {<No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<std::allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >, 0, true>> = {<std::allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >> = {<__gnu_cxx::new_allocator<std::unique_ptr<OrtValue, std::default_delete<OrtValue> > >> = {<No data fields>}, <No data fields>}, <No data fields>}, <absl::lts_20240116::container_internal::internal_compressed_tuple::Storage<unsigned long, 1, false>> = {value = 140232705792304}, <No data fields>}, <No data fields>}, data_ = {allocated = {allocated_data = 0x7f8a7c8d2f5c <std::_Tuple_impl<0ul, long*, std::default_delete<long []> >::_M_head(std::_Tuple_impl<0ul, long*, std::default_delete<long []> >&)+24>, allocated_capacity = 140232705792320}, inlined = {inlined_data = "\\/\215|\212\177\000\000@Y\235x\212\177\000\000\330[\235x\212\177\000\000PY\235x\212\177\000\000\262Ì|\212\177\000\000pY\235x\212\177\000"}}}}
        __PRETTY_FUNCTION__ = "onnxruntime::common::Status onnxruntime::InferenceSession::Run(const RunOptions&, gsl::span<const char* const>, gsl::span<const OrtValue* const>, gsl::span<const char* const>, gsl::span<OrtValue*>)"
#6  0x00007f8a7c8a66c0 in OrtApis::Run (sess=0x4d64b2c0, run_options=0x4d64b250, input_names=0x7f8a789d5e20, input=0x7f8a789d5d60, input_len=1, output_names=0x7f8a789d5e28, output_names_len=1, output=0x7f8a789d5d68) at /home/keranova/libraries/onnxruntime/onnxruntime/core/session/onnxruntime_c_api.cc:831
        input_names_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f8a789d5e20}}
        output_name_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f8a789d5e28}}
        status = {state_ = std::unique_ptr<onnxruntime::common::Status::State> = {get() = 0x0}}
        session = 0x4d64b2c0
        input_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f8a789d5d60}}
        output_span = {static extent = <optimized out>, storage_ = {<gsl::details::extent_type<18446744073709551615>> = {size_ = 1}, data_ = 0x7f8a789d5d68}}
#7  0x000000000049eda0 in Ort::detail::SessionImpl<OrtSession>::Run (output_count=1, output_values=0x7f8a789d5d68, output_names=0x7f8a789d5e28, input_count=1, input_values=0x7f8a789d5d60, input_names=0x7f8a789d5e20, run_options=..., this=0x7f8a7c254030) at /home/keranova/libraries/onnxruntime/include/onnxruntime/core/session/onnxruntime_cxx_api.h:573
        ort_input_values = 0x7f8a789d5d60
        ort_output_values = 0x7f8a789d5d68
        ort_input_values = <optimized out>
        ort_output_values = <optimized out>

I have checked the model. My wrapper works when I use it in a smaller program.

Thank you for your help.

To reproduce

Here is the code for my wrapper, containing the API calls.
OnnxModelWrapper.txt

I manage to make it work with a smaller program, I can't seem to reproduce it in a simple example.

Urgency

No response

Platform

Linux

OS Version

Debian 10.13

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

v1.18.0

ONNX Runtime API

C++

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[snnn]

You may try to build the code with address sanitizer, or run the binary with valgrind.
Our build script has a "--enable_address_sanitizer" flag.

[yuslepukhin]

Any chance you can give us your model OR any model where it repros? Most of the segfaults in GH issues occur when the input/output shape does not match the buffer, so ORT reads or writes beyond the actual allocation.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.