Assertion `!subscription_event.exists()' failed.

[mariodirenzo]

I'm randomly hitting this assertion at the end of some multinode runs. The backtrace is

#0  0x00007fb7ed2ce23f in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fb4dc582920, rem=rem@entry=0x7fb4dc582920) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78
#1  0x00007fb7ed2d3ec7 in __GI___nanosleep (requested_time=requested_time@entry=0x7fb4dc582920, remaining=remaining@entry=0x7fb4dc582920) at nanosleep.c:27
#2  0x00007fb7ed2d3dfe in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#3  0x00007fb7f08bd0fd in Realm::realm_freeze (signal=6) at /home/mariodr/legion/runtime/realm/runtime_impl.cc:187
#4  <signal handler called>
#5  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#6  0x00007fb7ed213859 in __GI_abort () at abort.c:79
#7  0x00007fb7ed213729 in __assert_fail_base (fmt=0x7fb7ed3a9588 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7fb7f14a75d4 "!subscription_event.exists()", file=0x7fb7f14a6c60 "/home/mariodr/legion/runtime/legion/runtime.cc", line=909,
    function=<optimized out>) at assert.c:92
#8  0x00007fb7ed224fd6 in __GI___assert_fail (assertion=0x7fb7f14a75d4 "!subscription_event.exists()", file=0x7fb7f14a6c60 "/home/mariodr/legion/runtime/legion/runtime.cc", line=909, function=0x7fb7f14a75a0 "virtual Legion::Internal::FutureImpl::~FutureImpl()")
    at assert.c:101
#9  0x00007fb7f00d14a0 in Legion::Internal::FutureImpl::~FutureImpl (this=0x7fb5d6a4aeb0, __in_chrg=<optimized out>) at /home/mariodr/legion/runtime/legion/runtime.cc:909
#10 0x00007fb7f00d1788 in Legion::Internal::FutureImpl::~FutureImpl (this=0x7fb5d6a4aeb0, __in_chrg=<optimized out>) at /home/mariodr/legion/runtime/legion/runtime.cc:929
#11 0x00007fb7f02b456c in Legion::Internal::DistributedCollectable::handle_downgrade_success (runtime=0x55c5691b92b0, derez=...) at /home/mariodr/legion/runtime/legion/garbage_collection.cc:1278
#12 0x00007fb7f0135f47 in Legion::Internal::Runtime::handle_did_downgrade_success (this=0x55c5691b92b0, derez=...) at /home/mariodr/legion/runtime/legion/runtime.cc:24566
#13 0x00007fb7f01037c6 in Legion::Internal::VirtualChannel::handle_messages (this=0x7fb5cc000bd0, num_messages=1, runtime=0x55c5691b92b0, remote_address_space=1, args=0x7fb48f5cdf20 "Y\267\n", arglen=12) at /home/mariodr/legion/runtime/legion/runtime.cc:12423
#14 0x00007fb7f0102947 in Legion::Internal::VirtualChannel::process_message (this=0x7fb5cc000bd0, args=0x7fb48f5cdf04, arglen=32, runtime=0x55c5691b92b0, remote_address_space=1) at /home/mariodr/legion/runtime/legion/runtime.cc:11894
#15 0x00007fb7f0104f09 in Legion::Internal::MessageManager::receive_message (this=0x7fb5cc000ba0, args=0x7fb48f5cdf00, arglen=40) at /home/mariodr/legion/runtime/legion/runtime.cc:13573
#16 0x00007fb7f013a922 in Legion::Internal::Runtime::process_message_task (this=0x55c5691b92b0, args=0x7fb48f5cdefc, arglen=44) at /home/mariodr/legion/runtime/legion/runtime.cc:26327
#17 0x00007fb7f015224c in Legion::Internal::Runtime::legion_runtime_task (args=0x7fb48f5cdef0, arglen=48, userdata=0x55c5691b35d0, userlen=8, p=...) at /home/mariodr/legion/runtime/legion/runtime.cc:32058
#18 0x00007fb7f0c3369e in Realm::LocalTaskProcessor::execute_task (this=0x55c567b8a8f0, func_id=4, task_args=...) at /home/mariodr/legion/runtime/realm/proc_impl.cc:1147
#19 0x00007fb7f0a3977f in Realm::Task::execute_on_processor (this=0x7fb48f5cdd70, p=...) at /home/mariodr/legion/runtime/realm/tasks.cc:303
#20 0x00007fb7f0a3d9c8 in Realm::KernelThreadTaskScheduler::execute_task (this=0x55c567cb9580, task=0x7fb48f5cdd70) at /home/mariodr/legion/runtime/realm/tasks.cc:1367
#21 0x00007fb7f0a3c712 in Realm::ThreadedTaskScheduler::scheduler_loop (this=0x55c567cb9580) at /home/mariodr/legion/runtime/realm/tasks.cc:1106
#22 0x00007fb7f0a3cd63 in Realm::ThreadedTaskScheduler::scheduler_loop_wlock (this=0x55c567cb9580) at /home/mariodr/legion/runtime/realm/tasks.cc:1218
#23 0x00007fb7f0a4544c in Realm::Thread::thread_entry_wrapper<Realm::ThreadedTaskScheduler, &Realm::ThreadedTaskScheduler::scheduler_loop_wlock> (obj=0x55c567cb9580) at /home/mariodr/legion/runtime/realm/threads.inl:97
#24 0x00007fb7f0a11152 in Realm::KernelThread::pthread_entry (data=0x7fb4dc38cef0) at /home/mariodr/legion/runtime/realm/threads.cc:781
#25 0x00007fb7ed1cc609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#26 0x00007fb7ed310133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

PS: @elliottslaughter could you please add this issue to #1032?

[elliottslaughter]

Added.

[lightsighter]

This is a Legion Spy only bug. Pull and try again.

[mariodirenzo]

I believe that this has been fixed by a5f5d3436a22afa7fc21152b65d34a3420290e8d.
Thanks for your help