Skip to content

Basic_HRTF_NumCalc_Simulation

Jump to bottom
Sergejs edited this page Aug 27, 2022 · 2 revisions

The HRTF Simulation

Main simulation part of the Mesh2HRTF which is resource intensive and benefits from a computer with as large amount of memory (RAM) as possible.

------------------------------------------------------------ Part of the the Complete Beginner’s tutorial ------------------------------------------------------------
Previous <<< Project Export last updated in 2022-08-27 Next >>> Saving HRTF SOFA files

Video version of this tutorial:

Introduction to the NumCalc solver

NumCalc solver is the heart of Mesh2HRTF - it is the practical implementation of the algorithms that calculate sound propagation around the head and ears and find the numerical solutions that can be packaged into the HRTF .sofa files. NumCalc is a powerful audio simulation tool that can be used for more than just HRTFs.

Simulation pre-requisites

NOTE: skip this paragraph if you run all Mesh2HRTF steps on the same computer.

Everything for NumCalc is already installed as part of the Mesh2HRTF installation, but it is very common that the NumCalc simulations are executed on another computer (and another operating system) than the rest of Mesh2HRTF steps. Usual reasons are: because simulation can take several hours (or in extreme cases even days) it is nice to avoid blocking the primary computer with simulation loads (note, it is possible to limit simulation resource usage). Plus distributing the simulation over additional computers is a very effective way to speed up simulations. Therefore,

it is good to know which parts of the Mesh2HRTF installation are necessary for a computer that only executes NumCalc simulation:

  1. The most important part is to install NumCalc.

    • On Windows, it is usually done by just extracting the Mesh2HRTF tools. + moving the "manage_numcalc_script.py" to the same Mesh2HRTF tools folder where "NumCalc_WindowsExe" is located.

    • On Linux and Mac there is a very fast and simple compilation step using the source code from the Mesh2HRTF project. + "manage_numcalc_script.py"

  2. make sure you installed "mesh2hrtf" Python package - it is required to run "manage_numcalc_script.py".

  3. To actually run the NumCalc it is highly recommended to use manage_numcalc_script.py.

    • On Windows "manage_numcalc_script.py" and the folder "NumCalc_WindowsExe" usually must stay next to each other. Together this file and folder can be moved/copied anywhere, on a computer that has "mesh2hrtf" Python package installed.

    • On Linux/Mac you only need "manage_numcalc_script.py". The script will find where you installed NumCalc after the compilation.

Run NumCalc simulation

- Since 2022-08 "NumCalcManager.py" functionality is part of the main Mesh2HRTF project under a name "manage_numcalc_script.py"

The recommended approach is to use “manage_numcalc_script.py”. Otherwise, there are alternative ways to launch NumCalc, including the most primitive "just run the NumCalc.exe" approach. Advanced users are welcome to run NumCalc any way they need - "manage_numcalc_script.py" has no impact on the final "NumCalc" simulation results.

Typical steps to run the simulation:

  1. Following the Blender project export you should have 2 project folders (for left and right HRTF sides) which contain "parameters.json" files along with other files and folders. A good place to have these folders is in your mesh2hrtf-tools folder, because then all tutorial steps can be performed in the same place.

  2. Make sure “manage_numcalc_script.py” is located in a valid location:

    1. The typical "manage_numcalc_script.py" location is "Next to project folders" - manager script will then scan all project folders next to it and start to simulate each folder sequentially one after another. This approach allows to simulate many HRTF projects one after another without manual launching of every folder. Any present finished project folders take only a few extra seconds to confirm their status and the manager script will continue executing just the instances that are not yet simulated.

    2. The Alternative "manage_numcalc_script.py" location is "Inside one project folder" - this way only the selected project folder will be executed (not very useful, but supported mode). Note that the "NumCalc_WindowsExe" folder that is necessary on Windows (not used on Linux/Mac) should still be located one level above - next-to the project folder.

  3. - Launch the manage_numcalc_script.py.

    • On Windows, if Python was installed Directly then it should be possible to just Double-click the “manage_numcalc_script.py” to launch the simulation. (If double-click does not work, try using right-mouse-click on the “manage_numcalc_script.py” and use "Open with" ==> "Python 3.x".)

    • For Linux, Mac, Windows Anaconda installations or any other Python environment cases it is always possible to write python manage_numcalc_script.py(or possibly "python3 manage_numcalc_script.py") in the command line of the environment where "mesh2hrtf" Python package was installed.

    • Be careful on Windows: do NOT click inside the Python window. If you see a “white square cursor” – the “manage_numcalc_script.py” script will be paused and it will not do anything. To escape the problem: press “Esc” key until “white square cursor” disappears (you should then see some status messages appearing at regular intervals).

    • Antivirus in Windows – if your antivirus raises a concern, it is recommended to treat “manage_numcalc_script.py” as Installer or Updater to prevent disturbances in repeatedly launching NumCalc.exe files. (anyone can see exactly what "manage_numcalc_script.py" does, so there is zero safety concern to use the unmodified script).

Done. Now wait for the simulation to complete :)

Advanced NumCalc tips

Advanced knowledge around NumCalc (not necessary, but may be useful to know):

  1. It is recommended to check the first NC*.out files when the first simulation instances are completed.

    • Make sure that simulation printed out End time: line. In case of any issues, see Troubleshooting section.

    • Note: You can ignore “Warning: There are some frequencies higher than the highest allowable frequency!”. This warning is only a rough indication that is valid for meshes that use uniform triangle edge size for the complete mesh. Properly optimized (graded) meshes almost always trigger this warning.

  2. Two projects == two sources of errors. Note that any problem or crash in the simulation can happen independently in the LEFT or RIGHT ear projects – even if one ear is perfect, the simulation of the other side may still need checking.

  3. It can be useful to re-launch “manage_numcalc_script.py” the main simulation run is complete to make sure that everything went well and all instances completed their tasks. If the project is fully simulated, the “manage_numcalc_script.py” will quickly report that everything is OK, alternatively it will automatically launch any instances that failed to produce output.

  4. Memory usage (performance & system resources) – the main limitation to the number of instances you can run is system memory (RAM) usage. For reasonable computation time it is necessary to both use all available RAM & CPU resources and avoid RAM overload (which can absolutely slow down processing to a crawl). To manage the resources the simple solution is to use “manage_numcalc_script.py” which runs as many NumCalc instances as possible and automates the process within one computer.

    1. Multiply your simulation speed by running more NumCalc Instances. Note that each NumCalc instance can only utilize 1 CPU core (single threaded) and therefore maximum CPU usage of one NumCalc instance can stay well under 10% for your CPU. To utilize more cores, the only option is to run more NumCalc instances (which requires more free RAM).

    2. More 3D mesh triangles == needs more RAM – Meshes with less than 15 thousand triangles require modest amounts of RAM/instance and can be calculated in a few hours on a PC with 32 GB of RAM. Meanwhile a mesh with 80k triangles can use over 10 GB of RAM/instance and hours of computation for each step. Note that with 16 GB of RAM available, a computer can easily be RAM-limited to run only one instance at a time despite negligibly low CPU usage (one core). For some reference, in this paper Table 1 shows some estimates for computation time for a server with plenty of RAM / CPU-core (while most computers would be RAM-limited).

    3. Higher sound frequency == needs more RAM – The RAM usage is highly dependent on the frequency that is computed in a given NumCalc instance. If computing 500 Hz takes just a couple of GB of RAM, the same mesh can use over 20GB when calculating 24 kHz. The amounts are approximate and highly dependent on exact mesh polygon count. “manage_numcalc_script.py” uses special NumCalc RAM requirement estimation command at the start of the simulation to have a rough idea how much memory will be used by each instance.

  5. Use multiple computers – Each simulation can easily be split into at least 2 parts (left and right side simulation project folders) and each project part can be simulated on any combination of Windows, Linux or Mac computers. When all simulations are complete the results can easily be copied back to one computer before running post-processing.

  6. Warning – It is dangerous to run “NumCalc.exe” without correct input arguments. If NumCalc executable is launched directly in a folder that contains NC.inp file, NumCalc will delete all existing output data in that folder without warnings.

  7. Do not run multiple “manage_numcalc_script.py” scripts on one computer. They will work, but the computer resources will be chaotically managed and the performance will be quite unpredictable. (If you need to schedule additional projects for simulation, it is better to kill the running “manage_numcalc_script.py” and restart it - you will loose progress in the currently running instances, but “manage_numcalc_script.py” will correctly scan all new and changed projects and schedule them for simulation.)

  8. Interruptions – “manage_numcalc_script.py” guarantees that it is safe to kill any “NumCalc.exe” or “manage_numcalc_script.py” processes at any time and just re-start “manage_numcalc_script.py” to continue.

    • “manage_numcalc_script.py” never re-runs instances that have completed their simulation. It properly checks the log files to detect which instances are complete. It means that unlike running raw NumCalc.exe, it is safe to launch “manage_numcalc_script.py” even when the whole project has been simulated to the end.

    • It can happen that some NumCalc instances (out of the hundreds executed during the simulation) do not complete simulation because the computer was hibernated or any other process caused NumCalc instance to exit. Rerunning "manage_numcalc_script.py" will fix this.

  9. Non-convergence – If the mesh is not smooth or has other defects the simulation can take 2-3 times longer to run. Normally the solution is found in less than 100 iterations, but in worst Non-convergence cases the simulation may run for 250 iterations before giving up. These extra iterations will prolong the total simulation time.

Limit simulation resource usage

Ideally the NumCalc should run when the computer is unused (over night?). It is always possible to kill "manage_numcalc_script" to immediately free up resources when needed and then just restart "manage_numcalc_script" to resume simulation.

In case you have to multi-task - continue using the computer that is also running "manage_numcalc_script.py" simulations, it is possible to limit resource usage in the manager script

Check the code inside "manage_numcalc_script.py" or run python manage_numcalc_script.py -h for help on available commands. To limit resources there are arguments --max_ram_load, --max_instances as well as --max_cpu_load (not very reliable) and --ram_safety_factor (can be used creatively to keep some RAM free).

Troubleshooting

When using "manage_numcalc", each instance of NumCalc simulation will generate:

  1. NC*.out text log file (open it in any text editor). It contains very good readable printouts and statistics written by the NumCalc for diagnostics. Last lines may mention why and during which step NumCalc encountered problems.

  2. NC*_log.txt text command line printout file. This information may compliment information from NC*.out file, but it usually is not necessary.

These files are located in ../yourProjectFolder/NumCalc/source_1/ and there is one file for every NumCalc instance (normally as many instances as there are frequency steps in the project).

Healthy NumCalc simulation

A sign of a successfully simulated NumCalc instance is a NC*.out text log file that ends with a line End time:. If end time line is missing, then the simulation is either not yet complete, has been interrupted or stopped due to an error. Here are EXAMPLES of:

Note that there may be some non-fatal warnings among the printouts. One exception is non-convergence issue - it is critical, but will still produce end time line

In some cases it may also be useful to check the example of a heathy "info.txt" file (generated by the Blender project export).

Immediate crashes of NumCalc

In the most extreme cases, if the first launched NumCalc instance crashes directly, "manage_numcalc_script.py" will explicitly stop and warn the user. Such crashes are often due to major defects in the input mesh or if NumCalc executable is not working correctly.

  • Don't forget that sometimes it may be enough to just re-try the simulation (by starting "manage_numcalc_script" again).

  • To check that NumCalc executable itself is working, on Windows, check that /NumCalc_WindowsExe/folder is next to the "manage_numcalc_script.py" and then double-click the "- run_NumCalc_instance.bat" inside this folder. If everything is working correctly:

    • Among the printouts there will be No input file found! - it means that NumCalc works.
    • Also in the same folder NumCalc will create new folder "be.out", and file "NC1-1.out".
    • If this is not the case - NumCalc.exe itself is not working on a given computer.

  • If "NC .out" files do not give any hints it may be a bug that should be reported as an Issue.

“3D analysis” not passed

The simulation crashes almost immediately and the “NC*.out” file contains just a few lines after the line (see example file with this issue):

3D analysis

This means that input mesh is obviously not suitable for simulation. For example the mesh is not air-tight, has some loose vertices, overlapping or internal surfaces or is just corrupt due to file format errors - these problems should be quite easy to identify by examining 3D mesh in detail.

“Non-Convergence issue”

The simulation takes much longer time and the “NC*.out” file contains lines (see example file with this issue):

Warning: Maximum number of iterations is reached!
CGS solver: number of iterations =

This information effectively says that the simulation did not find a solution (did not converge) and the result of this instance, for this frequency step is corrupt (garbage).

  • <= 24000 Hz – If this issue happens at frequencies within useful range (under 24kHz) the complete simulation results will NOT be usable! (the remaining usable data will not allow to produce HRTF for the common 48kHz sampling rate).

  • >24000 Hz – If this issue is encountered at simulation frequencies above 24000 Hz, the HRTF can still be used (assuming the frequency step is 150 or 75 Hz) because finalize_HRTF_simulation.py script will automatically drop all simulation data above the problematic frequency and still produce HRTF for the common 44.1 and 48kHz sampling rates. Higher than 24kHz frequencies are not fully tested and may require finer 3D than suggested in the recommended “hrtf_mesh_grading” (to keep closer to the rule of edge length must be at least 6times smaller than the shortest wavelength [Source]).

This problem usually affects only the highest frequencies, but unfortunately it does not always show up in all frequencies (for example simulation may converge at 20000 and 19900Hz, but fail to converge at 19800Hz). Mesh2HRTF post-processing has built in detection, reporting and work-arounds of this issue, therefore if this issue happens at high-enough frequencies, the simulation results can still be usable. The possible workarounds are:

  • Make sure that the 3D meshes were processed using “hrtf_mesh_grading” – other methods for re-meshing may cause less optimal final meshes that can cause non-convergence.

  • Only in case of using Point source simulations: make sure that the point source is at least ~0.3 mm distance away from the closest tringle. If Point Source is too close, it may get treated as it is inside of the mesh.

  • The main cause of this issue is significant errors in the mesh. Check if you can find and fix issues in the 3D mesh, such as:

    • Overlapping or folded-over triangles

    • Unnaturally sharp edges or deep holes.

    • Very large individual triangles.

    • Some face normals do not point outwards (try to "recalculate normals").

    • Any other surface disturbance in areas that should have a smooth skin surface.

  • Last resort: try this workaround: you need to "manually influence the cluster size" (the grouping of elements for the ML-FMM method). It is done by changing a value inside NC.inp file (located in the ../your_project/NumCalc/source_1/ folder). On the line 2. Main Parameters II, set the fourth number to 0.05 (everything else stays as it is):” for example like this: “ 0 1 0 0.05 0 0 0 ”. Then try to continue simulation using manage_numcalc_script.

“Gaussean points limit issue”

If simulation crashes and the “NC*.out” file contains lines (see example file with this issue):

Too many integral points in the theta-direction!
Number of integration points =

This issue usually only appears on complex (or broken) meshes at high frequencies. The possible workarounds are:

  • This issue has not been observed when 3D mesh is processed using “hrtf_mesh_grading” – this is the main fix (assuming the 3D mesh itself is good).

  • Check if you can find issues in the 3D mesh:

    • Overlapping or folded-over triangles

    • Unnaturally sharp edges or deep holes.

    • Very large individual triangles.

    • Some face normals do not point outwards (try to "recalculate normals").

    • Any other surface disturbance in areas that should have a smooth skin surface.

“MSBE issue”

If all simulation instances crash and the “NC*.out” file ends with line "nsbe must be < MSBE(= 110)" (see example file with this issue):

coordinates of the source point: 
nsbe must be < MSBE(= 110)
number of the element =

According to Dr. Wolfgang Kreuzer this issue is usually caused by:

  1. there is an overlapping element - a tringle that overlaps with another triangle.
  2. the Point Source is too close to the surface of the mesh - repositioning the Point Source at least 0.3 mm away from the surface should fix this case.

Overlapping elements can happen in rare cases even when using “hrtf_mesh_grading”. To detect and fix this issue:

  • Open for inspection the mesh that is causing the problem. (It can be the graded mesh saved by hrtf_mesh_grading or the same mesh exported out of Blender file from which the Mesh2HRTF Project was exported.)

  • If the graded mesh has a problem, it will show up in the "Air-tight mesh check" as described in How to make sure that 3D mesh is ready.

    • The most elegant fix for overlapping triangle issue is using Meshlab command: “Filters ==> Cleaning and Repairing ==> Remove Duplicate Faces”. If the command prints out "Successfully deleted 1 duplicated faces", then the problem is most likely resolved. The mesh will need to be re-imported into Blender and project re-exported.

Next tutorial step >>> Saving HRTF SOFA files

Clone this wiki locally