Questions on having a separate set of grids for particles

[n8xm]

The Particles documentation states that the "set of grids used to define the ParticleContainer doesn’t have to be the same set used to define the simulation’s mesh data."

Futhermore, according to the section on interacting with mesh data, if the particles are not "defined on the same grid as the... field MultiFabs":

...then an additional copy will need to be performed. However, if the particles are distributed in an extremely uneven fashion, it is possible that the load balancing improvements associated with the two-grid approach are worth the cost of the extra copy."

I have two questions about the above.

1. Is the “extra copy” from the grids used for the field, to the set of grids that are used to define the ParticleContainer?
2. Why exactly does the “two-grid approach” have better load balancing?
   a. At first, I was thinking that if the particle distribution is uneven, there could be many boxes that do not contain particles, thus necessitating a different set of boxes, However, I am not sure that is necessary because the ParIter already skips "grids or tiles that don’t have any particles". Furthermore, if we have the "two hierarchies track each other", e.g. with the help of the AmrParticleContainer class, then I'm not sure there is any difference between the spatial distribution of boxes for particles and boxes for the mesh. So where does the load balancing advantage come from?

[hengjiew]

Hello Nathan,

1. The copy can go both ways, i.e., fluid grids <-> particle grids.

2. It is easier to show this with an example. Assume we are solving two boxes of equal size with 2 processes, i.e., 1 box per process. The fluid computation is balanced. Now if box 1 has much more particles than box 2, the particle computation will be very imbalanced. The dual-grid approach allows you to, for instance, cut box1-2 to four smaller boxes as particle grids and assign one small box to process 1 and the others to process 2. In this way, it can better balance particle computation at the expense of transferring data between two sets of grids (particle and fluid). As the paragraph you cited says, this will improve the overall performance if the reduction in the particle solve time overweighs the data transfer cost.
   I am not exactly sure what you mean by "spatial distribution of boxes". The particle and fluid grids do cover the same geometric domain. But the load balance here is more about the workload distribution over the processes. In this sense, dual-grid allows users to choose a separate distribution (over processes) for particle grids, which can improve load balance in certain cases.

Hengjie

[n8xm]

Thank you for responding so quickly to my post! 👍 The thing that I did not understand previously was that you can cut boxes for the particle grid, therefore allowing the distribution of particles per process to be more balanced. Somehow, I had thought that the particle grid had to be the same as the grid for the mesh data.