Running RapidCFD on openFoam9

[sachin1911]

Hi
I ran openFoam9 samples on CPU and everything is working fine. I want to run RapidCFD.
I am using ubuntu 18.04, Cuda 11.4 and openFoam9. Is it not possible to run RapidCFD with this specs or do I need to go with older versions on openFoam, Cuda and Ubuntu?
Please share the procedure to install rapidCFD with compatible versions! Also please share how to run RapidCFD and How to compile it!

Thanks in Advance!

[TonkomoLLC]

Hi,

Please see issue #92 . There's some discussion about RapidCFD and newer version of CUDA.

Please see issue #93 for more detailed information on installation.

Between these comments I hope you are able to get started with RapidCFD.

Good luck!

[sachin1911]

Hi,
Thank you so much for your reply, will try that and let you know!

[sachin1911]

Hi,
I referred both the issue that you mentioned and I was able to compile now, but the compilation gets terminated with the following error,

-ldriftFluxTransportModels -ldriftFluxRelativeVelocityModels -lfiniteVolume -lmeshTools -lsampling -lfvOptions -lincompressibleTransportModels -lturbulenceModels -lcompressibleTurbulenceModels -lOpenFOAM -ldl -lm -o /opt/RapidCFD-dev/platforms/linux64NvccDPOpt/bin/driftFluxFoam
nvcc warning : The 'compute_35', 'compute_37', 'compute_50', 'sm_35', 'sm_37' and 'sm_50' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::dictionary::writeOptionalEntries' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::mapDistribute::mapDistribute(int, Foam::Xfer<Foam::List<Foam::List > > const&, Foam::Xfer<Foam::List<Foam::List > > const&, bool, bool)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Vector<double>, double, (unsigned char)3>::min' /opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libsampling.so: undefined reference to Foam::mappedPatchBase::facePoint(Foam::polyMesh const&, int, Foam::polyMesh::cellRepresentation)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Vector<double>, double, (unsigned char)3>::componentNames' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Vector, double, (unsigned char)3>::one'
/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libsampling.so: undefined reference to Foam::meshSearch::meshSearch(Foam::polyMesh const&, Foam::polyMesh::cellRepresentation)' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::polyPatch::movePoints(Foam::PstreamBuffers&, Foam::Field<Foam::Vector > const&)'
/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libfiniteVolume.so: undefined reference to Foam::cyclicACMILduInterfaceField::transformCoupleField(Foam::gpuField<double>&, unsigned char) const' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::GAMGInterface::agglomerateCoeffs(Foam::Field const&) const'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::mapDistributeBase::checkReceivedSize(int, int, int)' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::Ostream& Foam::operator<< Foam::token(Foam::Ostream&, Foam::InfoProxyFoam::token const&)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Vector<double>, double, (unsigned char)3>::typeName' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::SymmTensor Foam::flipOp::operator()<Foam::SymmTensor >(Foam::SymmTensor const&) const'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Vector<double>, double, (unsigned char)3>::max' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::polyMesh::pointInCell(Foam::Vector const&, int, Foam::polyMesh::cellDecomposition) const'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Vector<double>, double, (unsigned char)3>::zero' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::facePointPatch::movePoints(Foam::PstreamBuffers&, Foam::Field<Foam::Vector > const&)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::polyMesh::findCell(Foam::Vector<double> const&, Foam::polyMesh::cellDecomposition) const' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::facePointPatch::initMovePoints(Foam::PstreamBuffers&, Foam::Field<Foam::Vector > const&)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to double Foam::flipOp::operator()<double>(double const&) const' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::UOPstream::write(long)'
/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libfiniteVolume.so: undefined reference to Foam::cyclicAMILduInterfaceField::transformCoupleField(Foam::gpuField<double>&, unsigned char) const' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Tensor, double, (unsigned char)9>::zero'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::operator<<(Foam::Ostream&, unsigned char)' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::Vector Foam::flipOp::operator()<Foam::Vector >(Foam::Vector const&) const'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::SphericalTensor<double> Foam::flipOp::operator()<Foam::SphericalTensor<double> >(Foam::SphericalTensor<double> const&) const' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::Tensor Foam::flipOp::operator()<Foam::Tensor >(Foam::Tensor const&) const'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to void Foam::mapDistribute::transform::operator()<bool>(Foam::vectorTensorTransform const&, bool, Foam::List<bool>&) const' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::UPstream::allToAll(Foam::UList const&, Foam::UList&, int)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::readInt32(Foam::Istream&)' /opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libfiniteVolume.so: undefined reference to Foam::twoDPointCorrector::correctDisplacement(Foam::gpuField<Foam::Vector > const&, Foam::gpuField<Foam::Vector >&) const'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::VectorSpace<Foam::Tensor<double>, double, (unsigned char)9>::one' /usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to Foam::treeDataCell::treeDataCell(bool, Foam::polyMesh const&, Foam::polyMesh::cellDecomposition)'
/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/libmeshTools.so: undefined reference to `Foam::mapDistributeBase::schedule() const'
collect2: error: ld returned 1 exit status
/opt/RapidCFD-dev/wmake/Makefile:149: recipe for target '/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/bin/driftFluxFoam' failed
make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/bin/driftFluxFoam] Error 1

Why am I facing this error? any possible solutions that I can try? Kindly Help me with this error!

Thanks In advance!

[TonkomoLLC]

Hello,
I have not come across this error before but I will try to help.

1. What version of CUDA are you using? It must be recent to have the warning about deprecated sm_35 and so on. Can you drop down to CUDA 11.1, which is the last version that worked perfectly, at least on my machine?
2. Did you set sm_## in the wmake rules before compiling?
3. driftFluxFoam is pretty far down the list of files to compiled. Is this the first error, or were there errors before this one?

Thanks for this info. Hopefully we can figure this out together.

[sachin1911]

Hi,

1. I am having both CUDA 11.1 and 11.4 and I have exported both paths to use all libraries in both versions. Can I do like that or should I use only CUDA-11.1?

2. yes, I have set the sm_35 on both c and c++ files in wmake rules! (3.5 is my GPU compatibility )

3. driftFluxFoam is the first error I am facing during the compilation, but I am often getting this warning which is,
   /usr/local/cuda-11/bin/../targets/x86_64-linux/include/cub/util_cpp_dialect.cuh:142:13: warning: CUB requires at least C++14. C++11 is deprecated but still supported. C++11 support will be removed in a future release. Define CUB_IGNORE_DEPRECATED_CPP_DIALECT to suppress this message.
   CUB_COMPILER_DEPRECATION_SOFT(C++14, C++11);

   So I guess warnings are fine right? Or do I have to do anything about this warning?

Thanks for helping me with this!

[TonkomoLLC]

Firstly, you can generally ignore RapidCFD compilation warnings as long as the compiled code works.

If the solver you want to use was already compiled and works, you'll have to decide how much effort you put into figuring out how to compile driftFluxFoam, particularly if you do not plan to use driftFluxFoam.

Hmm. I have used CUDA 11.4 and 11.5 and I found it possible to compile driftFluxFoam. There were other issues with CUDA > 11.1 that I described in Issue #92.

Just be sure that when you change between CUDA 11.1 and 11.4 that you recompile. On my laptop I have several versions of CUDA installed in /usr/local/cuda-11.1 ... /usr/local/cuda-11.2, and so on... And then I have multiple versions of RapidCFD that are compiled for each version of CUDA with entirely separate directories. But if you are working in just one directory you should completely recompile RapidCFD if you change the CUDA version you are working with.

So, in sum,

• Is the solver you want to use working? If so, maybe you have a good enough compilation
• If not, please ensure that when you switched between versions of CUDA that you completely recompiled RapidCFD from a clean copy or you worked in separate directories... that is, you do not have RapidCFD compiled partially with one version of CUDA and partially compiled with another version of CUDA.

Thanks for checking these points.

[sachin1911]

Hi,
Honestly speaking I don't mind If I am not able to compile driftFluxFoam. As you said driftFlux is very last to compile, then other previous solvers will be working right! because they got compiled without any error!
And also for using multiple CUDA version, I will take your point and check again once!

So, I have a small doubt in running the tutorial example now! now lets say I want to run icoFoam using rapidCFD solver! but rapidCFD-dev doesn't have the tutorial package in it right! so what are the ways I can run the icoFoam tutorial example using rapidCFD solver. (FYI: openFoam and rapidCFD-dev are both in /opt directory in my system). The question I asked may be dumb but I am new to this, so I am hoping that you could help!

Thanks in advance!

[TonkomoLLC]

Hi,
Glad that you may be OK with the compilation of RapidCFD.
I have posted some RapidCFD test cases here. You'll find a cavity example that can be run with icoFoam.

I hope this helps you get started!

[sachin1911]

Hi TonkomoLLC,

rapidcfd.log

listed above is the rapidCFD compilation log, actually after going through the compilation i found I am facing errors at many points...

The first error was missing of mpi.h which i resolved now!

But after going through the log you can find this error

/opt/RapidCFD-dev/src/OpenFOAM/lnInclude/PointHit.H(89): error: identifier "Foam::Vector ::zero" is undefined in device code
1 error detected in the compilation of "cellClassification/cellClassification.C".
cellClassification/cellClassification.dep:433: recipe for target 'Make/linux64NvccDPOpt/cellClassification.o' failed
make: *** [Make/linux64NvccDPOpt/cellClassification.o] Error 1

1. In this case I am not able to find the lninclude folder inside opt/RapidCFD-dev/src/OpenFOAM/ , the PointHit.H file is found inside other directory. I want to know when this lninclude file is created in rapidCFD.
2. Also there are no cellClassification.dep file inside the directory. So why this error occurs, any explanation for this?
3. Most of the errors are linker errors!
4. How can we solve this Foam::Vector::Zero is undefined in device code error! If we comment it its working but is the permanent solution?

Thanks for checking the rapidCFD log!

[TonkomoLLC]

Hello,

Please see #92.

Issues 1 & 4. PointHit.H

In PointHit.H, replace line 89:

            hitPoint_(vector::zero),

with

            hitPoint_(vector(0,0,0)),

Issues 2 & 3. Hopefully fixing PointHit.H as above will also fix cellClassification and the link errors

Hopefully this quick fix will solve your immediate problems.

Best regards,

Eric

[sachin1911]

Hi,

1. I have resolved the hitpoint_(vector::zero) error by replacing it with vector(0,0,0).
2. Next I was facing error in
   solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C: In constructor ‘Foam::multiSolidBodyMotionFvMesh::multiSolidBodyMotionFvMesh(const Foam::IOobject&)’:
   solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C:104:6: error: reference to ‘const_iterator’ is ambiguous
   forAllConstIter(dictionary, dynamicMeshCoeffs_, iter)

which I reffered issue #92 and checked the make file in src dynamicMesh and the "dynamicRefineFvMesh/dynamicRefineFvMesh.C" line was already commented. But also I was getting the above error. So, I commented other line in the file
/*
solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C
*/

so after This I was not facing the above error. Is this the right way to do it?

3. Then to check this I tried to compile dynamicFvMesh file alone separately by running this command on terminal "user@BLTSP00316:/opt/RapidCFD-dev/src$ ./Allwmake dynamicFvMesh "
   So I made wcleanALL before running this command and I was able to compile the dynamicFvmesh files alone. Is this the right procedure to compile files individually?

4. After compiling the dynamicFvMesh individually, I didn't get the error which I mentioned in statement 2. But I am facing some new errors like
   nvlink fatal : Could not open input file '/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libOSspecific.o'
   /opt/RapidCFD-dev/wmake/Makefile:172: recipe for target '/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libOpenFOAM.so' failed
   make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libOpenFOAM.so] Error 1

   collect2: error: ld returned 1 exit status
   /opt/RapidCFD-dev/wmake/Makefile:172: recipe for target '/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libfvOptions.so' failed
   make: *** [/opt/RapidCFD-dev/platforms/linux64NvccDPOpt/lib/libfvOptions.so] Error 1

and

 **pointPatchFields/derived/sixDoFRigidBodyDisplacement/sixDoFRigidBodyDisplacementPointPatchVectorField.C:33:10: fatal     error: forces.H: No such file or director
   #include "forces.H"
      ^~~~~~~~~~
    compilation terminated.
    pointPatchFields/derived/sixDoFRigidBodyDisplacement/sixDoFRigidBodyDisplacementPointPatchVectorField.dep:541: recipe     for target 'Make/linux64NvccDPOpt/sixDoFRigidBodyDisplacementPointPatchVectorField.o' failed
     make: *** [Make/linux64NvccDPOpt/sixDoFRigidBodyDisplacementPointPatchVectorField.o] Error 1**

This errors occures during the compilation of dynamicFvMesh file alone.
Have you faced this errors? What are the possible solutions?

Thanks for checking all points!!

[TonkomoLLC]

Oh you're right. #92 does not stop the error you found with multiSolidBodyMotionFvMesh.C

Instead of editing src/dynamicFvMesh/Make/files, please edit src/Allwmake:

# wmake $makeType dynamicFvMesh

#92 has been updated with this advice.

You will find that any library or solver that relies directly or indirectly on libdynamicMesh will fail.

To completely avoid these problems, you should downgrade to CUDA 11.1. If you can live without dynamicMesh then CUDA 11.5 works for me.

With respect to your libOpenFOAM.so and libfvOptions.so issues, please completely clean your RapidCFD installation (e.g., from the /opt/RapidCFD-dev directory, type wclean all. I don't have an explanation for your compilation errors with these libraries. Please also note I have not tried CUDA >= 11.6 so if you are using the most recent versions of CUDA there may be more problems lurking that i have not found.

I hope you find this advice helpful.

Thanks and best regards.

[sachin1911]

Hi TonKomoLLC,

I have made most of the changes that you suggested and I am now able to compile the RapidCFD-Dev completely without any errors. Thanks to you!
I am using CUDA 11.4 and I have to comment this below line in make file to run the compilation without any error! I know this is not a permanent fix but if you don't want dynamicfvmesh, then commenting this will help you compile without any errors.
/*
solidBodyMotionFvMesh/multiSolidBodyMotionFvMesh.C
*/

Now I am trying to run the RapidCFD-Tests and I am able to run cavity using icoFoam,

1. I want to know is there any performance comparison with RapidCFD vs the traditional openFoam. What parameters should I compare to know how the RapidCFD is faster then the normal openFoam. How can we visualize the performance variance?
2. And also how can I verify that the RapidCFD in running on GPU.

please clarify these points! Thanks in Advance!

[TonkomoLLC]

Hi,

Glad you found a workaround for compilation with CUDA 11.4, and that a solution is to compile with commenting out just the multiSolidBodyMotionFvMesh.C file in the dynamicFvMesh Make/files .

To your questions.

1. I reported in compile RapidCFD on GPU machine #75

I placed some test cases here. As you noted, one needs a sizable grid to see the speed up with RapidCFD. But the trick is, especially with older GPU's, is that larger grids require more GPU memory... and you can find yourself limited with one GPU card.

I generally see 1 K20 GPU = ~8-16 cores of E5-2670 CPU's. This technology is several years old; I do not have stats for newer technology.

Recently in #97, @jiaqiwang969 reported ~128x speedup over a single core with a more modern A100 GPU.

To reiterate what I quoted above, to see any significant speedup, you need to run a very large case, likely >> 1 million cells but your experience can vary.

2. Since RapidCFD has a lot of code that relies on GPU operation, if you can get any case to run you can be sure that it is running on the GPU. But if you want to verify, you can look at the GPU usage by typing nvidia-smi.

Hope this is helpful.

Good luck.

[sachin1911]

Hi TonKomo,

Thank you so much for your so fast and detailed reply all the time you gave.

1. I use Nvidia Geforce GT 730 GPU to run RapidCFD. I know its very old. I ran the cavity test and I was able to run in 2.14s. But in normal openFoam I ran the same cavity in 0.07s which is much faster than GPU performance.

2. So, as you said to see any significant speedup, I need to run a very large case.

3. In RapidCFD-Tests the cavity test has all the executable files, like polymesh, and icoFoam values, so i was able to run it without any error. But for other examples we don't have the executable data. So when I try to do 'blockMesh' for pitzdaily in RapidCFD-test. I am facing this error
   blockMesh: symbol lookup error: /usr/lib/libextrudeModel.so: undefined symbol: _ZN4Foam10dictionary20writeOptionalEntriesE

4. So to make 'simpleFoam' I tried to copy the data from pitzdaily of normal openfoam to the rapidCFD-test but it also throws error. I know this step is not possible but I still tried it.

5. Can I get the data for Dambreak and sloshinTank like you gave for cavity to run directly.

6. Also I have other system which uses TeslaM60 GPU and I wanted to know if I copy this compiled RapidCFD-dev from this system to the other system that uses TeslaM60 and run the examples directly there without compiling in that system. so, can I cross compile the binary and run it in the other system? Is that possible?

So, If I can get the Data file like cavity for both dambreak and sloshingTank it would very helpful!

Thanks in advance!

[TonkomoLLC]

Hi,

I don't have the current ability to send the files you need. I think you are asking for polyMesh files for other examples? However, I am not sure because you asked for the data file for sloshing tank but I think the case includes polyMesh? The polyMesh for dambreak is very large and I would prefer that you prepare the polymesh for this case rather than my storing it on GitHub

Please prepare meshes cases using blockMesh found in OpenFOAM for CPU, e.g., 2.3.x. You may not be able to use newer versions of OpenFOAM like v9 and newer because the file header format has changed.

To question 6 - If you have a new GPU that has a different compute capability then you need to recompile RapidCFD from scratch.

Sorry if I did not fully help you but I hope you can figure this out.
Best regards,
Eric

[sachin1911]

Hi,

1. I used Rapid-CFD to run Dambreak example using laminar method, but why my Rapi-cfd is not running as fast as my cpu? my normal openfoam can run the dambreak much faster than RapidCFD that runs on GPU (Nvidia geforce GT 730, 2GB memory)

2. I want to know what GPU configuration you used to get the result that you have mentioned in the README file of dambreak.

Thank you!

[TonkomoLLC]

Hello,

I used a GeForce GTX 980M GPU for the DamBreak test, as described in the README. It has been a long time since I ran this test, but according to my notes in the referenced README file, this high resolution damBreak case requires more than 5B of GPU RAM. I am not sure how you ran this particular damBreak case with a 2GB GPU? How many cells are in your damBreak case (e.g., as reported by checkMesh)?

If you are comparing a single GT 730 (from 2014) with a modern CPU with 8+ cores (e.g., from 2022) then it will be tough to see a speed-up effect, I think. That is, if you ran a case with a 2014-era single GPU vs. a 2022-era 8-core CPU (and the case is decomposed for 8 CPU cores), I would be surprised if you see a speed-up effect for the GPU. But since I did not run this test I need to be clear that this is only my speculation. Also, to be clear, when I am comparing speedup per core, not speedup per CPU (which can contain main cores).

I hope this helps and you find the speedup you are expecting. It is there, especially for large cases and it is more pronounced when you compare GPU's and CPU's from the same time period.

Best regards,
Eric