Realm: Too many simultaneous local client threads

[mariodirenzo]

After updating legion from 7990f5e9e to 65d1a3b7d, I non-deterministically get the following error during the initialization phase of the HTR solver: *** FATAL ERROR (proc 1): in gasneti_fatal_threadoverflow() at bed-gasnet/source/extended-ref/gasnet_extended_common.c:63: GASNet Extended API: Too many simultaneous local client threads (limit=256). To raise this limit, configure GASNet using --with-max-pthreads-per- node=N.

The backtrace of the error is

  [0] = /lib64/libc.so.6(gsignal+0x10f) [0x15124c044a9f]
  [1] = /lib64/libc.so.6(abort+0x126) [0x15124c017e04]
  [2] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(gasneti_error_abort+0x4e) [0x1575694]
  [3] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(_gasneti_fatalerror+0xb2) [0x1575747]
  [4] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(gasneti_fatal_threadoverflow+0x4e) [0x43d5cce]
  [5] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(gasnete_new_threaddata+0x219) [0x43d5fa9]
  [6] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(gasnetc_AM_PrepareRequestMedium+0x2e6) [0x43c5036]
  [7] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x43b3468]
  [8] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x43a12ee]
  [9] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x43a13af]
  [10] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3ded732]
  [11] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3de6bec]
  [12] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Realm::Processor::spawn(unsigned int, void const*, unsigned long, Realm::ProfilingRequestSet const&, Realm::Event, int) const+0x85) [0x3de3755]
  [13] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::VirtualChannel::send_message(bool, Legion::Internal::Runtime*, Realm::Processor, Legion::Internal::MessageKind, bool, bool, Legion::Internal::RtEvent)+0x2b5) [0x359db95]
  [14] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::VirtualChannel::package_message(Legion::Serializer&, Legion::Internal::MessageKind, bool, Legion::Internal::RtEvent, Legion::Internal::Runtime*, Realm::Processor, bool, bool)+0x1fe) [0x359de3e]
  [15] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::Runtime::send_index_partition_child_response(unsigned int, Legion::Serializer&)+0x53) [0x35b0073]
  [16] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::IndexPartNode::handle_node_child_request(Legion::Internal::RegionTreeForest*, Legion::Deserializer&, unsigned int)+0x1fb) [0x352d29b]
  [17] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::VirtualChannel::handle_messages(unsigned int, Legion::Internal::Runtime*, unsigned int, char const*, unsigned long) const+0x353) [0x35f9543]
  [18] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::VirtualChannel::process_message(void const*, unsigned long, Legion::Internal::Runtime*, unsigned int)+0x16b) [0x35faaeb]
  [19] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec(Legion::Internal::Runtime::legion_runtime_task(void const*, unsigned long, void const*, unsigned long, Realm::Processor)+0xc7) [0x35fb0f7]
  [20] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3e35960]
  [21] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3e359d5]
  [22] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3e33fa8]
  [23] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3e37df1]
  [24] = /leonardo/home/userexternal/mdirenzo/htr/bin/prometeo.exec() [0x3e3b0ae]
  [25] = /lib64/libpthread.so.0(+0x81ce) [0x151260dd21ce]
  [26] = /lib64/libc.so.6(clone+0x42) [0x15124c02fdd2]
  [27] = [0xffffffffffffffff]

I haven't been able to reproduce the issue in debug mode and I am only able to reproduce the issue on a system where it is not possible to connect to the node to attach a debugger so I can't produce (for now) more detailed backtraces.

@elliottslaughter, can you please add this issue to #1032 ?

[lightsighter]

Are you running with -ll:force_kthreads?

[mariodirenzo]

Yes

[lightsighter]

And if you remove the -ll:force_kthreads does the error go away?

[streichler]

See #1508. If you need -ll:force_kthreads, it should also be workaround-able with -gex:immediate 0.

[mariodirenzo]

Both including -gex:immediate 0 or removing -ll:force_kthreads solve the problem.
Thanks for suggesting the workaround.
I'll close this issue as #1508 already keeps track of this problem