Panzer: DofManager working with device mesh data structures

[maartenarnst]

@trilinos/panzer

The Panzer DOFManager and ConnManager are great tools for numbering degrees of freedom in finite element models.

However, as mesh data structures move to device, challenges arise:

• the DOFManager carries out key computational steps on host, in serial, using STL data structures (std::set ,...)
• the ConnManager hardcodes the use of std::vector as a type to provide mesh connectivity data

As a result, when the mesh data structure is on device, the computation does not exploit the parallelism of the device, and data transfers to the host are needed.

It seems that addressing this issue is of current interest at Sandia. E.g., we have seen recent work in Albany related to making the DOFManager and ConnManager work with the Omega_h device data structures.

On our side, we use the Panzer DOFManager and ConnManager in an in-house FE code that we develop in our group. We have experimented a bit with device implementations (use of Kokkos StdAlgorithms and Views) and we have seen significant speedups (factor of more than 20 times faster).

The goal of this issue is therefore to ask what's the panzer team's view on this? Would it be of interest to have a discussion with users/developers of the Panzer DOFManager to try to align ideas?

Just as a suggestion, starting points for a discussion might be:

• introduce a PR to declare member functions of DOFManager virtual, so that users can experiment with device implementations.
• think of ways in which ConnManager could support types such as Kokkos::View in place of std::vector; an issue may be how to maintain backward compatibility.

@rppawlo @romintomasetti @CamelliaDPG @mperego @bartgol @cwsmith.

[rppawlo]

We definitely would like to move all computations possible to device. Backwards compatibility is important. Will any of you be attending the TUG meeting in Albuquerque at the end of this month? Would could have a side meeting there. Otherwise I can set up a virtual meeting over teams.

[maartenarnst]

Hi @rppawlo. I won't be attending the TUG. I'd be happy to meet over teams.

[maartenarnst]

Hi @rppawlo. I've gone ahead and made a first PR that declares some of the member function of the dof manager as virtual:

• Make functions virtual in Panzer_DOFManager. #12432

This way, we can get the ball rolling. And this first step may not be the step where significant discussion is needed.

A follow-up could be to think about how to generalise the conn manager, so as to allow the data to arrive in other types than std::vector. I have the impression that discussion would be needed there to find the best way forward.

Would you have a moment to take a look at #12432? Thanks in advance!

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.