Question: Hypre GPU support

[gzagaris]

Hello,

I am exploring using Hypre in my application and I am particularly interested in its GPU capabilities:

• Which of the solvers and preconditioners in Hypre are available on the GPU? I am particularly interested in using GMRES with the BoomerAMG preconditioner
• Are there any GPU examples that illustrate filling the CSR matrix (using the IJ interface) within a GPU kernel and handing the pointers to Hypre?

Thank you for all your help.

[victorapm]

@gzagaris, thank you for your interest in hypre. Answering your questions:

1. Mostly all solvers/preconditioners are GPU-enabled with a few exceptions in terms of configuration options, e.g., HMIS coarsening for BoomerAMG does not run on GPUs. You can find a complete list here : https://github.com/hypre-space/hypre/wiki/GPUs#gpu-support-for-solvers-and-preconditioners
2. Assuming unified memory and matrices constructed from the host, you can follow the example https://github.com/hypre-space/hypre/blob/master/src/examples/ex5.c However, there aren't specific examples yet for matrices constructed directly on the GPU. The main difference here is that the HYPRE APIs such as HYPRE_IJMatrixSetValues cannot be called from the device kernel. Instead the workflow is to allocate "large" work arrays on the GPUs to hold row and column coordinates as well as nonzero coefficients and fill them in the device kernels. Next, those arrays will be passed to HYPRE_IJMatrixSetValues to construct the hypre matrix and will be subsequently freed by the user

Hope this helps,
Victor

[gzagaris]

Excellent! Thank you for the quick response @victorapm

Is calling HYPRE_IJMatrixSetValues the best way to update the matrix at each physics iteration, or is there a more efficient method for updating the non-zero values in the matrix after initial construction?

I am using Umpire and plan on leveraging the memory pools. However, I would like to avoid any unnecessary tear-down/setup and MPI communication for assembling the matrix as much as possible in subsequent iterations.

Thank you again for all your help.

[victorapm]

@gzagaris, calling HYPRE_IJMatrixSetValues2 (for entries in the current MPI rank) or HYPRE_IJMatrixAddToValues2 (for entries belonging to another rank) is currently the best way to update matrices during a simulation. I agree that in situations where the sparsity pattern doesn't change, we could think of a simpler approach for users and I will bring this up to the team.

[gzagaris]

Thank you for the clarification, @victorapm. I’ll be on the lookout for any updates or improvements on this front. I really appreciate you bringing this up to the Hypre team and for your consideration.

[victorapm]

@gzagaris, would you have an estimate of the costs (percentage of overall runtime) for assembling your linear systems?

[gzagaris]

@gzagaris, would you have an estimate of the costs (percentage of overall runtime) for assembling your linear systems?

Hi @victorapm, thank you for following up on this. Yes, I can provide this information. I am spread a bit thin this month, but I want to get back to this. Please, give me some time and I'll follow up on this thread.

[victorapm]

Thank you, @gzagaris!