Add perpendicular-flap case solid-solids4foam and solid-openfoam

[solids4foam]

(description by @MakisH)

This adds two solid participant cases in the perpendicular flap tutorial:

• solid-openfoam (using the standard OpenFOAM solver solidDisplacementFoam)
• solid-solids4foam (using the solids4Foam solver of solids4foam)

Note that this depends on precice/openfoam-adapter#236 and in the nextRelease development line of solids4foam.

[MakisH]

Thanks a lot for adding a solid-openfoam case as well! I had some trouble getting the case to run, even though it compiled successfully with v2012 and v2206. Looks also like you accidentally pushed some build artifacts.

[MakisH]

I currently have a very strange issue running the solid-solids4foam case, which I am not sure where it originates from. However, it looks related to the fact that both preCICE and solids4foam use Eigen.

Logs: https://gist.github.com/MakisH/26342b40980ea0c3260ef90110a15295

This run was with:

• Ubuntu 22.04, which includes Eigen 3.4.0.
• preCICE v2.4.0, built from source in Debug mode
• OpenFOAM v2012
• The OpenFOAM adapter state from Adds functionality to allow FSI to work with solids4foam as the solid solver openfoam-adapter#236 (commit a591b)
• solids4foam nextRelease branch (commit 50881), built with its own copy of Eigen 3.3.7.

Other tutorial cases work normally (e.g., the solid-openfoam).

I know that one can ask solids4foam to use the global Eigen installation. I did so by setting EIGEN_DIR="/usr/include/eigen3" and rebuilding solids4foam (after executing Allwclean). Looks like Allwmake discovered my installation and just created a symlink to that, so both preCICE and solids4foam should be using the same version. But even with that, I get the same error.

@solids4foam did you have any such issues?

@fsimonis any clue what could be wrong here, if any, in preCICE?

[fsimonis]

@MakisH This definitely looks like conflicting versions. The callback of the tag uses an instance of Eigen::MatrixXd, so this all makes sense.

Can you somehow see which version of Eigen OpenFOAM is actually using (build logs, compiler invocations or similar maybe?)

Alternatively, you could build preCICE with the Eigen version used by OpenFOAM. This sounds like a shorter compilation time. 🥵

[solids4foam]

@MakisH This definitely looks like conflicting versions. The callback of the tag uses an instance of Eigen::MatrixXd, so this all makes sense.

Can you somehow see which version of Eigen OpenFOAM is actually using (build logs, compiler invocations or similar maybe?)

Alternatively, you could build preCICE with the Eigen version used by OpenFOAM. This sounds like a shorter compilation time. 🥵

As Eigen is a header-only library, I am surprised that it is causing a conflict since it is only looked up during compilation, not at run-time (e.g., as like a dynamic library).
Can you track down the actual lines in the Adapter and/or preCICE that cause the problem? Maybe, the -DFULLDEBUG -g -O0 compilation flags would help for the adapter.

By the way, can you check if solids4foam works on its own, e.g. run ./Allrun in solids4foam-release/tutorials/solids/linearElasticity/plateHole/plateHoleTotalDisp, just to rule out solids4foam-specific issues.

[solids4foam]

@MakisH This definitely looks like conflicting versions. The callback of the tag uses an instance of Eigen::MatrixXd, so this all makes sense.
Can you somehow see which version of Eigen OpenFOAM is actually using (build logs, compiler invocations or similar maybe?)
Alternatively, you could build preCICE with the Eigen version used by OpenFOAM. This sounds like a shorter compilation time. 🥵

As Eigen is a header-only library, I am surprised that it is causing a conflict since it is only looked up during compilation, not at run-time (e.g., as like a dynamic library). Can you track down the actual lines in the Adapter and/or preCICE that cause the problem? Maybe, the -DFULLDEBUG -g -O0 compilation flags would help for the adapter.

By the way, can you check if solids4foam works on its own, e.g. run ./Allrun in solids4foam-release/tutorials/solids/linearElasticity/plateHole/plateHoleTotalDisp, just to rule out solids4foam-specific issues.

FYI, I have tried the following, and it works for me:

• Ubuntu 20.04.4 LTS
• OpenFOAM-v2012 from apt-get
• preCICE binary libprecice2_2.4.0_focal.deb
• solids4foam, nextRelease branch (commit 4e3f64b0): I did not set EIGEN_DIR and so solids4foam downloaded its own copy of Eigen (eigen-3.3.7)

I have also tried (and it works):

• macOS 11.4
• OpenFOAM-v2012 built from source using the patches from https://github.com/mrklein/openfoam-os-x and the clang compiler (Apple clang version 12.0.5)
• preCICE binaries from homebrew (thank you, very convenient!)
• solids4foam, nextRelease branch (commit 50188d0e - the more recent): once again, I did not set EIGEN_DIR and so solids4foam downloaded its own copy of Eigen (eigen-3.3.7)

I wonder if the debug mode of preCICE catches something that gets missed in the optimised version.

[MakisH]

The Debug vs Release mode sounds indeed like something relevant. Building in Release mode, preCICE is initialized normally, but then breaks the next time it uses Eigen. This output is without PETSc, as the error handling seems to obfuscate issues, which also makes the RBF mapping use Eigen. But even building in Release mode with PETSc gives again a segfault.

Same situation when solids4foam is built with its own Eigen copy. solids4foam by itself runs fine.

FYI, I have tried the following, and it works for me:

I will try to reproduce the first environment here, but to me this sounds like an actual issue that others will eventually also face.

[MakisH]

Update: the case does run in the preCICE Demo VM, with OpenFOAM v2012. It completes without any issues and I get reasonable results.

I would suggest that:

• I open an issue summarizing our discussion
• Maybe, for visibility, @solids4foam creates a similar issue at the solids4foam repository (I this the most probably is that the issue originates from there, more in the issue I will open)
• We add a link to issue to the documentation (this PR)
• We merge this.

[fsimonis]

It is almost definitely an ODR issue with various versions of Eigen.

As Eigen is a header-only library, I am surprised that it is causing a conflict since it is only looked up during compilation, not at run-time (e.g., as like a dynamic library).

The preCICE shared library exports all symbols, which is bad. We are working to change in the future. This means that the linker will treat both versions of Eigen as equal. This consequently leads to the observable issues with the destructor.

OpenFOAM downloads Eigen 3.3.7, while preCICE uses the system installation of Eigen.
Ubuntu 20.04 ships Eigen 3.3.7 and Ubuntu 22.04 ships Eigen 3.4.0.
Hence, Ubuntu 20.04 works as both versions are identical.
The versions differ on Ubuntu 22.04, explaining the violation of the ODR and the crash.

Note that we most likely violate the ODR even with identical versions of Eigen, as we conditionally enable EIGEN_INITIALIZE_MATRICES_BY_NAN in Debug builds. Which of the compiled versions the linker finally picks, most likely doesn't dramatically influence the behaviour of Eigen, hence no crash.

[MakisH]

Finally, I don't have any more comments to add, rather than that this was a really nice contribution and review! 😄 Thanks a lot!

[raynoldtan]

We have recently tried to run the perpendicular flap case using percice 2.4 with solids4Foam NextRelease branch and the develop branch of the openFoam Adapter. We have encounter similar issue. We have used gdb to get the call stack. Attached is the screenshot of the error message from solid side.

For the fluid side:

please advise how to fix.

[fsimonis]

@raynoldtan Same issue. You need to tell solid4foam to use the system/spack eigen. https://solids4foam.github.io/installation/installFromSource.html#eigen

This needs to point to the same version used by spack. Use spack find --paths eigen to see the path of the used version.

[raynoldtan]

Hi fsimonis,
@fsimonis @MakisH

we have already did this before the compilation of solids4Foam. We have used

export EIGEN_DIR=/home/raynoldtan/spack/opt/spack/linux-ubuntu20.04-cascadelake/gcc-9.4.0/eigen-3.4.0-4k2lkyslxqecxkljs4yh6t7rcbrsvh7d

which points the EIGEN_DIR to where the eigen3.4.0 was installed with spack. This is the same version of eigen that was used to install precice 2.4.0. After the compilation of solids4Foam, we verified that by looking into:

/home/raynoldtan/solids4foam-release/ThirdParty/eigen3/share/pkgconfig/eigen3.pc. It shows the path of the eigen library being used.

prefix=/home/raynoldtan/spack/opt/spack/linux-ubuntu20.04-cascadelake/gcc-9.4.0/eigen-3.4.0-4k2lkyslxqecxkljs4yh6t7rcbrsvh7d
exec_prefix=${prefix}

Name: Eigen3
Description: A C++ template library for linear algebra: vectors, matrices, and related algorithms
Requires:
Version: 3.4.0
Libs:
Cflags: -I${prefix}/include/eigen3

But the issue still remains.

[fsimonis]

@raynoldtan The EIGEN_DIR is expected to point to the include directory of Eigen. The Eigen repo follows an unusual directory structure. 😉

So you need to append /include to the installation prefix.

export EIGEN_DIR=/home/raynoldtan/spack/opt/spack/linux-ubuntu20.04-cascadelake/gcc-9.4.0/eigen-3.4.0-4k2lkyslxqecxkljs4yh6t7rcbrsvh7d/include

[raynoldtan]

@fsimonis

Thanks for letting me know. I changed the export command to now append the /include into the path.

export EIGEN_DIR=/home/raynoldtan/spack/opt/spack/linux-ubuntu20.04-cascadelake/gcc-9.4.0/eigen-3.4.0-4k2lkyslxqecxkljs4yh6t7rcbrsvh7d/include

And then I recompiled the solids4Foam. After that I go to ThirdParty folder to check,

It seems correct now. Then I tried to rerun the perpendicular flap case again. But it still give the same issue as:

[0]PETSC ERROR: ------------------------------------------------------------------------
[0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[0]PETSC ERROR: or see https://petsc.org/release/faq/#valgrind
[0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple MacOS to find memory corruption errors
[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run 
[0]PETSC ERROR: to get more information on the crash.
[0]PETSC ERROR: User provided function() at unknown file:0  
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 59.

I not sure if the version of precice used affect this? or the eigen version used will affect this?

Raynold

[fsimonis]

@raynoldtan The path is still incorrect.

It has to look like this ThirdParty/eigen3, so only one eigen3.

The following file must exist: ThirdParty/eigen3/Eigen/Core

I would manually remove the eigen3 directory in ThirdParty and recompile from scratch.

[raynoldtan]

@fsimonis

I have removed the eigen3 folder. Then, I set the env variable using the following path:

export EIGEN_DIR=/home/raynoldtan/spack/opt/spack/linux-ubuntu20.04-cascadelake/gcc-9.4.0/eigen-3.4.0-4k2lkyslxqecxkljs4yh6t7rcbrsvh7d/include/eigen3

After that, I recompiled solids4Foam. Then I get the same directory path for eigen 3 as described by you above:

~/solids4foam-release/ThirdParty/eigen3/Eigen

where the file Core is located.

Then I tried to rerun the case again. But I still get the same error as above.

[fsimonis]

Ok, so there is something else going on.
Could you please open a new issue in this repository and describe your setup in additional detail?

[raynoldtan]

Ok thanks for your help anyway. I have included the link below.

#292