SU2 Nonequilibrium solver residual stalling

[susmitsj]

Hello,

I have been using SU2 NEMO to run cases for steady, inviscid, hypersonic flow around blunt objects. Previously I have been able to successfully run cases using SU2 version 7.1.1. However, I have been running into residual stalling with the newer versions (7.4.1 and 7.5.1). Recently, I tried two different meshes which worked for me before (TestCases/euler/bluntbody/blunt_91.su2 and a RAM C-II mesh I previously used). I also tried updating the confguration file to match selections in TestCases/nonequilibrium/invwedge_ausm.cfg but I still face this issue. I have attached the relevant files here. Please let me know if this is a solver issue or if I need to change any selections in the configuration file.
residual_stalling.zip

[CatarinaGarbacz]

Hello, can you send the .cfg and .su2 files of a case that previously worked well and now it doesn't? I can check it. Thanks

[susmitsj]

Hi Catarina. Thanks for your help. Since there's a attachment file type limitation, I am attaching the files in a zip folder here. In the folder, the configuration files marked as '_old' are the ones that worked well before. 'invbb' refers to the blunt body case and 'invram' refers to the RAM C-II case. The mesh files are the same as before (blunt body is in .su2 format and ram c-ii in .cgns).

residual_stalling.zip

[CatarinaGarbacz]

Hello,

I have been debugging the differences between the two versions. I found 5 differences that lead to slightly different results. Two of them have been verified to be an improvement of physical modelling. The other 3 I am still trying to figure out. Could you please run the same tests with MUSCL_FLOW=NO and see if you're still facing this issue?

I see that the MUSCL reconstruction is one of the 3 things that have been modified and not sure if could be the cause of the problem or not. I will keep working on understanding the impact of the other 2 things.

When you do this - could you please save the simulation output log of both versions and send those back to me too (in this way I'll have a better feeling of what is happening to the residuals throughout the simulation)? We can just focus on the invbb case to make it simpler.

[aditya12398]

Hi @CatarinaGarbacz

Sorry to piggyback on this, I hope it is okay. I think I have the same issue...

I have been trying to get a hyper-sonic flow around the cylinder to converge but unfortunately to no avail. I assumed my setup was incorrect so I tried to look for an already tested case to run. I tried the case with the files uploaded in #1625 but that seems to not converge for me either. I have attached the residual plot

I am willing to run a few cases on my workstation to help debug this. Kindly let me know which files to use and with which settings toggled/changed.

Please let me know if you need any more information from my side.

[CatarinaGarbacz]

Hi @aditya12398

That would be helpful as it has been a bit difficult to identify the problem between versions of the code that have a lot of differences.

As mentioned in my prev message, one of these differences is the MUSCL implementation and I am leaving this out for now. If you could ran the case attached in the following manner, would be useful:

• sending cfg files invbb_old_711.cfg and invbb_old_751.cfg (for versions 7.1.1 and 7.5.1 respectively) + mesh file blunt_91.su2
• could you run 2 simulations, one with 7.1.1 another with 7.5.1 with those cfg files pls?

MUSCL is disabled, so if we obtained very different results between those 2 simulations we have a good hint that the problem is not MUSCL. and we can then decide what to do next.

Please confirm if you can do this and send me updates once you have them - thanks! I keep working on this in different fronts, more hands-on debug.

[aditya12398]

Hi @CatarinaGarbacz

Sorry for such a late response, my machine is bricked so it took a lot of time to get this task done. Anyway, I finally have the results for the two cases. Here you go:

invbb_711.zip
invbb_751.zip

Please do let me know if I have run the case as you wanted me to.

[susmitsj]

Hello, sorry for being missing from action for the past few weeks. Thanks once again for looking into this and for the insights you provided @CatarinaGarbacz.

I see that @aditya12398 has already completed the MUSCL toggle check. Let me know if anything else needs to be done.

[susmitsj]

Hi @CatarinaGarbacz

The issue has carried over to SU2 v8.0.0 as well. Just checking in to see if you have any more insights or updates or a way to work around them. I also noticed a similar issue reported for bluff-body hypersonic simulations on CFD online forum posted May 2023. Here's the link: https://www.cfd-online.com/Forums/main/250097-su2-nemo-divergence-hypersonic-apollo.html. Let me know if you need my help on this.

Best,
Susmit

[bigfooted]

I tried one of the cases in the residual_stalling.zip file and you can converge it in a straightforward way:

I then tried the case invbb_711 and that also converged. The mesh is a bit coarse, and it shows in the solution:

[susmitsj]

Hi @bigfooted,

Thanks so much for looking into this. That's great news (although a bit surprising)! Just to make sure,

1. Did you make any changes to the config file or was it exactly the same config file as the one in the zip folder?
2. Can you please also tell the SU2 version?

In the meantime, I'll try running it on my end again.

Thanks.

[bigfooted]

The invbb_711 case I tried on master (v8.0) and current develop, it works without any modifications, although it converges faster if you use adaptive CFL.

[susmitsj]

Hi again @bigfooted , sorry for the long gap in getting back to this issue. I am currently co-ordinating with the cluster helpdesk at my university about this and most likely, it has something to do with how the code has been installed on the cluster that I have access to. Here are a few questions they asked-

1. Are you running the script directly using "mpirun -n # SU2_CFD config.cfg" or do you use a python wrapper for this?
2. Did you build the code from source or are you using precompiled binaries?
3. What kind of operating system and processors are you running the script on?

It would be great if you could recall these details so that I can pass them on. Thanks, again!

[bigfooted]

Hi, I compiled the code from source. I recommend this if you are running on a cluster. It does not need any python wrapper. I used mpirun and compiled with the gnu compiler and openmpi, but intel compiler and mpich will also work.

I use this on our cluster to compile:

./meson.py build --optimization=2 -Denable-mixedprec=true -Ddebug=false -Denable-pywrapper=false -Denable-tests=false -Denable-autodiff=false -Dwith-mpi=enabled -Denable-cgns=true -Denable-tecio=false --prefix=~/Codes/su2_github/su2