int64 for large meshes

[jcwright77]

what is the advisable way to use int64 for large meshes in mfem. If you use that option in mumps mfem complains that full 64 bit is not supported in mumps yet. if you just add it to parmetis, hypre and superlu complain

linalg/superlu.cpp:23:2: error: #error "Mismatch between HYPRE (32bit) and SuperLU (64bit) integer types"

I using the spack package in spack 0.22 release with:
specs:

• mfem@develop+examples+strumpack+sundials+superlu-dist+mumps ^mumps+metis+parmetis+ptscotch ^hypre
• parmetis +int64

[v-dobrev]

Hi @jcwright77, in the first Spack spec, can try using ^hypre+mixedint or ^hypre+bigint? You will also want to enable 64bit ints in SuperLU_DIST with ^superlu-dist+int64 (and you probably need ^parmetis+int64 for that). For STRUMPACK, I don't think you need to add anything -- the MFEM wrapper objects for it will use the index size from hypre.

As you noticed, mumps+int64 is not yet supported in MFEM though it's probably not too hard to add -- @psocratis knows best.

[jcwright77]

this worked for mfem and examples; ==> 2 root specs [+] ***@***.*** +examples+mumps+strumpack+sundials [+] openmpi ~atomics+cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java+legacylaunchers~lustre~memchecker+pmi+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools cuda_arch=none fabrics=ucx schedulers=slurm though it introduces and error in a miniapp fromHypre headers: schwarz.cpp: In constructor 'VertexPatchInfo::VertexPatchInfo(mfem::ParMesh*, int)': schwarz.cpp:303:26: error: cannot convert 'int*' to 'HYPRE_BigInt*&' {aka 'long long int*&'} 303 | B->GetOffd(H1pr_offd, cmap); | ^~~~ | | | int* In file included from ../../linalg/complex_operator.hpp:18, from ../../linalg/linalg.hpp:21, from ../../mfem.hpp:44, from schwarz.hpp:2, from schwarz.cpp:1: ../../linalg/hypre.hpp:592:52: note: initializing argument 2 of 'void mfem::HypreParMatrix::GetOffd(mfem::SparseMatrix&, HYPRE_BigInt*&) const' 592 | void GetOffd(SparseMatrix &offd, HYPRE_BigInt* &cmap) const; | ~~~~~~~~~~~~~~~^~~~

…

On 2024-06-06 18:59, Veselin Dobrev wrote: Hi @jcwright77 [1], in the first Spack spec, can try using ^hypre+mixedint or ^hypre+bigint? You will also want to enable 64bit ints in SuperLU_DIST with ^superlu-dist+int64 (and you probably need ^parmetis+nt64 for that). For STRUMPACK, I don't think you need to add anything -- the MFEM wrapper objects for it will use the index size from hypre. As you noticed, mumps+int64 is not yet supported in MFEM though it's probably not too hard to add -- @psocratis [2] knows best. -- Reply to this email directly, view it on GitHub [3], or unsubscribe [4]. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- -john Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright Links: ------ [1] https://github.com/jcwright77 [2] https://github.com/psocratis [3] #4345 (comment) [4] https://github.com/notifications/unsubscribe-auth/AB7SLTOHG3H35YGK67W7KCTZGDSVNAVCNFSM6AAAAABI5E43O6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMOJUGY3TQ

[v-dobrev]

Hi @jcwright77, can you clarify: which branch is this schwarz.cpp code from? I don't see it in master or a release.

[jcwright77]

plasma-devUsed in transport2dBut can just run make in that miniapp/plasma directory I suspect it’s an issue of creating the right variable integer to take the assignment. There is possibly in need for some flags and hyper that aren’t necessarily in the spack package.Works without int64-johnOn Jun 8, 2024, at 8:21 PM, Veselin Dobrev ***@***.***> wrote: Hi @jcwright77, can you clarify: which branch is this schwarz.cpp code from? I don't see it in master or a release. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>

[v-dobrev]

Okay, I see. It looks like @dylan-copeland wrote that code originally, so it's probably easiest for him to make the changes needed for 64bit ints -- some int/Array<int> data needs to be replaced by HYPRE_BigInt/Array<HYPRE_BigInt>. However, I'm not sure how deep into the code this may need to propagate.

[jcwright77]

On 2024-06-08 22:29, Veselin Dobrev wrote: Okay, I see. It looks like @dylan-copeland [1] wrote that code originally, so it's probably easiest for him to make the changes needed for 64bit ints -- some int/Array<int> data needs to be replaced by HYPRE_BigInt/Array<HYPRE_BigInt>. However, I'm not sure how deep into the code this may need to propagate. -- Reply to this email directly, view it on GitHub [2], or unsubscribe [3]. You are receiving this because you were mentioned.Message ID: ***@***.***>

thanks, I figure it was something like that. I saw some examples where Hypre with bigint worked but they didnt declare HYPRE_BigInt/Array<HYPRE_BigInt>.

…

-- -john Principal Research Scientist John Wright Office 617-253-9612 zoom: https://mit.zoom.us/my/jcwright Links: ------ [1] https://github.com/dylan-copeland [2] #4345 (reply in thread) [3] https://github.com/notifications/unsubscribe-auth/AB7SLTMSKZ6LDUDXOQCKKTTZGO4XFAVCNFSM6AAAAABI5E43O6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TOMJXGA3TO

[dylan-copeland]

I committed an initial attempt at generalizing for HYPRE_BigInt in branch plasma-dev-bigint:
10e5732
It is not tested yet, so it doesn't work yet. It has been a long time since I worked in this branch, and I'm not sure what tests actually work now. The "benchmark" test in the comments in transport2d.cpp blows up with a seg-fault, even with a normal build of the plasma-dev branch. @jcwright77 please share a test command you would like to run, if it still fails with this branch. If it works already with this commit, we can go ahead and merge this branch into plasma-dev.