flecsi-tutorial/00-driver stopped working with legion runtime

[korobkin]

After commit bfd2828 the flecsi-tutorial/00-driver won't work with legion on Wolf:
Steps to reproduce:

1. module load cmake/3.9.0 python/2.7-anaconda-4.1.1 gcc/6.4.0 boost/1.61 mpich/3.2.1
2. checkout flecsi-third-party commit bfd2828
3. checkout flecsi, current master branch (or commit laristra/flecsi@ecd3c76)
4. configure and build flecsi-third-party (complete config: CMakeCache-ftp-wolf.txt):

  cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/tmp/issue03 \
         -DENABLE_HPX=OFF -DENABLE_LEGION=ON -DLEGION_USE_OPENMP=OFF 
  make -j8

5. configure, make and install flecsi (full config file: CMakeCache-flecsi-wolf.txt)

  cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/tmp/issue03 \
         -DENABLE_HPX=OFF -DFLECSI_RUNTIME_MODEL=legion
  make -j8
  make install

6. Setup tutorial environment and run the test:

  source $CMAKE_INSTALL_PREFIX/bin/flecsi-tutorial.sh
  cd flecsi/flecsi-tutorial/00-driver
  flecsit compile driver.cc
  ./driver

• Expected: "Hello world!" output
• Actual:

*** Caught a fatal signal: SIGSEGV(11) on node 0/1
NOTICE: Before reporting bugs, run with GASNET_BACKTRACE=1 
in the environment to generate a backtrace. 
Segmentation fault (core dumped)

Full error output with the flag GASNET_BACKTRACE=1: error_message.txt

[korobkin]

Note: with the previous commit of flecsi-third-party (1e8dc30), or with -DFLECSI_RUNTIME_MODEL=mpi the example works fine.

[ipdemes]

@korobkin : there was a bug in flecsi-tutorial that made this test fail for us. I have a fix here laristra/flecsi@839edaf, and I should merge it into master sometime today. Could you, please, check if the test still fail for you after the fix is merge into the master?

[korobkin]

@ipdemes : unfortunately still fails, with the same error (segfault)

[charest]

Was that version of mpich built with "--enable-threads=multiple"?

[korobkin]

@charest : good call! but yes, I built it with that option:

   ./configure --prefix=$PROJECT/share --enable-threads=multiple --enable-fortran=yes

[ipdemes]

@korobkin : have you build+used your own mpich or do you use one from the module "mpich/3.2.1" ?

[korobkin]

@ipdemes : I have built it on my own, because turquoise does not have mpich/3.2.1. Maybe there is something wrong with my build? I made it into a module, so that must have been confusing.

[korobkin]

@ipdemes : I will try other versions of MPI.
On the other hand, it was working before: there is no error if I run with runtime = legion but previous version of flecsi-third-party, before bfd2828. Or if I run with runtime = mpi.

[lightsighter]

If this is a single node run, your should pull the latest version of the Legion master branch as I believe the issue you are seeing has been fixed.

[korobkin]

@lightsighter : Great! I checked out the master branch of Legion + tried CMAKE_BUILD_TYPE = Debug, and the segfault disappeared.
However, without Debug option (default) the segfault still persists.

[ipdemes]

@korobkin: that explains why I don’t see the issue: I always build in Debug mode

[lightsighter]

@korobkin How many nodes is this with? Is it the same backtrace as before? Is there any chance you can compile with optimizations on but with the "-g" flag also so we can get line numbers for the backtrace?

[korobkin]

@lightsighter @ipdemes : OK just found out that an old trick of simply increasing the stack limit makes segfault disappear, both for Debug and Release compilation options: ulimit -s unlimited. Shall we close the issue?

[ipdemes]

@lightsighter : I think I found the issue: the legion_defines.h file has DEBUG options wnabled for both Debug and Release builds:
#ifndef DEBUG_REALM
#define DEBUG_REALM
#endif
#ifndef DEBUG_LEGION
#define DEBUG_LEGION
#endif
#ifndef PRIVILEGE_CHECKS
#define PRIVILEGE_CHECKS
#endif
#ifndef BOUNDS_CHECKS
#define BOUNDS_CHECKS
#endif

[ipdemes]

@korobkin : I don't think so. I think we should fix an issue with "legion_defines.h" first

[korobkin]

@ipdemes : do you mean this file? legion/cmake/legion_defines.h.in
Can you reproduce the problem? If not, what is your suggestion to fix it, so I can try it out.

@lightsighter : yes, this is a single-node run. If I try to run it in gdb after recompiling with '-g' option, the problem disappears. If I increase the stack, it also disappears. There is a file above (error_messages.txt) which hints at approximate location of where the problem might have occured, but without exact line numbers.

[lightsighter]

@ipdemes Let's fix the issue that you identified and then see where we are.

[ipdemes]

@lightsighter , @korobkin : sorry for the confusion: it seems like the issue is already fixed in the most recent legion's master branch.

[lightsighter]

@ipdemes Yes, that would make sense to me given the backtrace that was reported earlier. @korobkin can you still create the segfault case on the most recent master branch of Legion?

[ipdemes]

@korobkin : can we close the issue?