MS-MPI hangs in MPI_Finalize() inside DLL.

[gkarpa]

Hi all.

I have a case where I'm using MPI inside a DLL file. It serves like an API which means that MPI_Finalize() must be called only once, from inside the DLL, in the end of its caller program. To accomplish this I've put MPI_Finalize() in a function that is marked with __attribute__((destructor)), so its code will execute upon unloading of the DLL.

However, the program hangs there, and MPI_Finalize() never returns. My setup is (although the issue was reproducible in other PC as well):
Windows 10 Pro Version 21H1 (OS Build 19043.1237).
MS-MPI Version 10.1.12498.18.
Build with gcc 10.1.0 using mingw64 from MSYS2 with target x86_64-w64-mingw32.

I have managed to reproduce the issue in a minimal example:

1. myapi.cpp (The DLL code)

#include <stdio.h>
#include <stdint.h>
#include <mpi.h>

void bin_destructor() __attribute__((destructor));

void bin_destructor(){
  int initialized;
  MPI_Initialized(&initialized);
  int finalized;
  MPI_Finalized(&finalized);

  printf("*** %s. Line %d ***\n", __FILE__, __LINE__); fflush(stdout);
  if(initialized && (!finalized)){
    printf("*** %s. Line %d, Finalizing MPI...", __FILE__, __LINE__); fflush(stdout);
    MPI_Finalize();  // <-- It hangs here
    printf("Done. ***\n"); fflush(stdout);
  }
  printf("*** %s. Line %d ***\n", __FILE__, __LINE__); fflush(stdout);
}

extern "C" __stdcall __declspec(dllexport) int MyAPI(const int n){
  int initialized;
  MPI_Initialized(&initialized);
  if( !initialized ) MPI_Init(NULL, NULL);  // Initialize only once.
  printf("*** %s. Line %d n=%d ***\n", __FILE__, __LINE__, n); fflush(stdout);

  return 0;
}

compiling with

g++ -shared -o libmyapi.dll -Wall -fPIC -IC:\SourceCode\MPI\Include myapi.cpp C:\Windows\System32\msmpi.dll

2. main.cpp (Program that uses the DLL)

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>

extern "C" int MyAPI(const int n);

int main(int argc, char** argv){
  printf("*** %s. Line %d ***\n", __FILE__, __LINE__); fflush(stdout);
  MyAPI(1);
  printf("*** %s. Line %d ***\n", __FILE__, __LINE__); fflush(stdout);
  MyAPI(2);
  printf("*** %s. Line %d ***\n", __FILE__, __LINE__); fflush(stdout);

  return 0;
}

compiling with

g++ -o main.exe -Wall main.cpp -L. -llibmyapi 

When I run the program serially, e.g. $>main.exe , the output is:

*** main.cpp. Line 8 ***
*** myapi.cpp. Line 26 n=1 ***
*** main.cpp. Line 10 ***
*** myapi.cpp. Line 26 n=2 ***
*** main.cpp. Line 12 ***
*** myapi.cpp. Line 13 ***
*** myapi.cpp. Line 15, Finalizing MPI...Done. ***
*** myapi.cpp. Line 19 ***

When I run the program in parallel, even with only 1 process, e.g. $>mpiexec -np 1 main.exe , the output is:

*** main.cpp. Line 8 ***
*** myapi.cpp. Line 26 n=1 ***
*** main.cpp. Line 10 ***
*** myapi.cpp. Line 26 n=2 ***
*** main.cpp. Line 12 ***
*** myapi.cpp. Line 13 ***
*** myapi.cpp. Line 15, Finalizing MPI...

..and it hangs there.

Any help so I can explain & resolve this behavior is greatly appreciated.
Best regards,
George