Using solids4foam with preCICE leads to Eigen-related runtime errors

[MakisH]

(extracted from precice/tutorials#286 (comment), more details there)

On my system, running the perpendicular-flap tutorial with solid-solids4foam and fluid-openfoam leads to some problems.
These logs come from Ubuntu 22.04, OpenFOAM v2012, preCICE v2.4.0 (built from source in Debug and Release modes), and OpenFOAM-preCICE v1.1.0.

@solids4foam could not reproduce the issue, while I could make it work on Ubuntu 20.04 with preCICE v2.3.0 (preCICE Demo VM + OpenFOAM v2012).

This can either be:

1. A system-specific issue
2. A platform-specific issue
3. An issue with the preCICE library (specifically related to the Eigen symbols export)
4. An issue with solids4foam (specifically related to the Eigen symbols export)

While I have currently only encountered it on my system, I don't think this is a system-specific issue. It could be, though, platform-specific (I am using GCC 11.2.0, the default in Ubuntu 22.04, while it works with GCC 9.4.0 on Ubuntu 20.04).

I don't think that this originates from preCICE, since @fsimonis has taken special care to hide symbols as part of making preCICE xSDK-compatible. solids4foam itself works on my system, but the problem seems to be specifically in the collaboration between solids4foam and preCICE.

Please comment if you face any similar issues, as well as if this works on your system.

[fsimonis]

Some clarification here precice/tutorials#286 (comment).

I don't think that this originates from preCICE, since @fsimonis has taken special care to hide symbols as part of making preCICE xSDK-compatible. solids4foam itself works on my system, but the problem seems to be specifically in the collaboration between solids4foam and preCICE.

Note that this is not the case, resulting in the issue at hand.
Even if we finally implement precice/precice#623, then there will still be ODR issues.
The only solution is to make all dependencies configurable.

[fsimonis]

The solution in this case is to configure preCICE to use the solid4foam Eigen version.
Additionally, I would expect solid4foam to use the Eigen version of the system by default and maybe download/extract another version as a backup.

preCICE is definitely guilty of this too, relying on checked-in versions of fmtlib and nlohmann/json. We will run into issues sooner or later.

[MakisH]

Additional context:

• solids4foam now has documentation regarding Eigen
• @philipcardiff / @solids4foam opened an issue in the solids4foam BitBucket
• @fsimonis opened an issue in solids4foam GitHub (context: solids4foam is currently gradually transferring from BitBucket to GitHub).
• @raynoldtan also faced the same issue on a cluster

[raynoldtan]

Hi all,

@makis @fsimonis

Recently, I did try to recompile solids4Foam with the updated version of nextRelease branch without the eigen library. (export S4F_NO_USE_EIGEN=1 && ./Allwmake) Subsequently, I try to run the perpendicular flap case again and got the same error despite the fact that solids4Foam was compiled without eigen.

The second thing I did was to reinstall precice using spack but making sure that the packages.yaml uses the preinstalled eigen3 from /usr/include/eigen3 (system) and then recompile solids4Foam with eigen from /usr/include/eigen3/. But again, I got the same error message.

I noticed in the above discussion that the precice was installed using binary packages instead of spack? Subsequently I installed precice using binary focal Debian file instead. I am using ubuntu 20.04. And the simulation case was able to run. I am not sure why but this is what I have noticed.

[MakisH]

I noticed in the above discussion that the precice was installed using binary packages instead of spack? Subsequently I installed precice using binary focal Debian file instead. I am using ubuntu 20.04. And the simulation case was able to run. I am not sure why but this is what I have noticed.

I also noticed the same, and the only difference was the Eigen version (and the compiler).

[fsimonis]

@solids4foam @philipcardiff Do you have any further ideas?

[philipcardiff]

@solids4foam @philipcardiff Do you have any further ideas?

It is strange that there are issues when solids4foam is compiled without Eigen; @raynoldtan was the solids4foam installation cleaned (i.e. Allwclean) before re-compiling? Or, even downloading a fresh copy to be sure could be an option.

[raynoldtan]

Hi @philipcardiff

Yes, I am sure that I have performed Allwclean before recompiling solids4Foam.
I have even tried to delete the solids4Foam folder and then git clone a new one. Then I switch to the nextRelease branch before compiling. I presume there was no issue with solids4Foam bcos I was able to run the Alltest inside the tutorial folder.

[raynoldtan]

@MakisH

What is the eigen version and compiler used?

[MakisH]

What is the eigen version and compiler used?

In Ubuntu 20.04: Eigen 3.3.7, GCC 9.4.0.
In Ubuntu 22.04: Eigen 3.4.0, GCC 11.2.0.

[raynoldtan]

Hi @MakisH @fsimonis @philipcardiff

I was trying to run the fluid-open foam case and the solid-open foam or solid-solids4Foam case together. I was able to run the fluid-open foam case in parallel but the solid-open foam case cannot run in parallel despite the settings in decomposePar. Running both case in serial has no issues. The following scenarios were tested.

Fluid-openfoam | Solid-openfoam | working/not working
Serial run | Serial run | working
Parallel run | Serial run | working
Serial run | Parallel run | working
Parallel run | Parallel run | not working

The same scenario applies to the solid4Foam case as well. I suspect there is some issue with the coupling both solvers when running in parallel. Whether this is coming from adapter or precice itself, please advise. The error is the same PETSC error as above.

Raynold

[MakisH]

Hi @raynoldtan,

could you please provide more details for the non-working cases? Also how did you partition the domain?

The error you see as coming from PETSc is not really a PETSc error. It is just that every exception is caught and handled by the PETSc error handler, when preCICE is compiled with PETSc. You probably need to look more carefully further up in the logs for the actual error message.

[raynoldtan]

HI @MakisH

Sorry for the confusion. The scenario where the cases are run in parallel on the fluid and solid side is working now. :)
I not sure why there was error just now. But it is working now. I have tested it a few time.

Raynold

[MakisH]

An update on this: using solids4foam v2.0 compiled with OpenFOAM v2012 on Ubuntu 22.04, I do not get this issue anymore. It has been resolved on the solids4foam side: solids4foam/solids4foam#4