Inconsistent behavior of hypre_GetPointerLocation

[ictmay]

The function hypre_GetPointerLocation in src/utilities/memory.c behaves differently on CUDA and HIP builds.

The functions cudaPointerGetAttributes and hipPointerGetAttributes are unreliable at identifying host pointers that weren't obtained from the CUDA/HIP host allocators respectively. For CUDA builds there are these lines in the above mentioned function:

if (attr.type == cudaMemoryTypeUnregistered)
{
   *memory_location = hypre_MEMORY_HOST;
}

A similar set of lines should be present for HIP builds, but are not.

Both CUDA and HIP builds will return hypre_MEMORY_HOST in the event of an error. However, given an arbitrary host pointer hipPointerGetAttributes will typically not error. Instead it will return a location of unregistered, and then hypre_GetPointerLocation will return hypre_MEMORY_UNDEFINED instead of hypre_MEMORY_HOST.

On debug builds with HIP this can result in a failed assertion in hypre_CheckMemoryLocation. I observed the following sequence of events recently:

1. Called HYPRE_IJVectorAssemble
2. That called hypre_IJVectorAssembleParDevice
3. That called hypre_AuxParVectorDestroy(aux_vector)
4. That deallocates the data fields inside the aux_vector struct correctly by specifying HYPRE_MEMORY_DEVICE
5. Deallocating the struct itself fails since it is of course on host, and the above assertion fails when hypre_GetPointerLocation can not figure out that the struct is indeed on host.