Pure SoA Particle: Separate Array for IdCPU

[ax3l]

Summary

This addresses a regression we see when moving to pure SoA particles:

• slightly slower read/write to Ids when needed, e.g., for sorting
• issues going up to the full 64bit range

Additional background

Once finished, this will close #3569.

Checklist

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[AlexanderSinn]

It would be nice if there was some sort of option to disable the 64-bit ID to save memory in the cases where a unique id for each particle is not required (i.e., 1 bit per particle in some other component is sufficient to track valid/invalid). It might be possible to “hack” this outside of amrex by setting the allocator of the ID PODVector to a null allocator.

[ax3l]

Hi @AlexanderSinn, thank you for the comment.

On first thought, I am not sure this small optimization is worth the additional compile time that this will cause in pyAMReX, ABLASTR, AMReX and HiPACE++ for differentially specialized PC template classes.

What performance benefits do you see from this? Are the species that never need tracking very large and will significantly save memory without it?

[AlexanderSinn]

For the plasma in HiPACE++ there is no per-particle diagnostics as it would be too much data, so there is no possibility to use the id for particle tracking. Besides the id the plasma has two other integer components so we could put those inside a 64 bit id.

The beam in HiPACE++ does have per particle diagnostics and can be interesting for particle tracking, however only if the per particle diagnostics is actually used. It is possible to have multiple beams that are each a few terabytes in size of which the 64bit id would be 12.5%. This is too much data for diagnostics/tracking, but the id still has to be communicated. Technically because the communication in HiPACE++ is custom, we don’t have to communicate every component that is in the particle container if we add a special case.

[ax3l]

Thank you for the details. I am not yet fully convinced if this is not a premature optimization, because for background plasmas (in WarpX), filtering diagnostics can be essential to statistically analyze representative particles. The same can be true in the future for HiPACE++ for specific research tasks. But let's chat more tomorrow in the dev meeting about it.

In general, if we can easily make this a runtime decision, then I am all for it. @atmyers takes the lead on this, what do you think?

I just want to reduce the compile-time variations for particle containers at all cost, instead of driving them up further. For instance, the allocator being a compile-time option in particles (vs. a runtime one in MultiFabs - which is better) is another design (as in: maintainability, compile-time, link-time, extensibility vs. performance benefit) issue we should address another time (off-topic) #3643

[AlexanderSinn]

Using amrex::PolymorphicArenaAllocator the allocator can be changed at runtime. That way only some components could have a different allocator.

[ax3l]

Thank you Andrew for your help!

I pushed a minor cleanup and inline comment and added a quick question on the new template argument in StructOfArrays.