Re: [h5md-user] Variable-size particle groups

[Peter Colberg]

On Tue, May 29, 2012 at 05:03:24PM +0200, Olaf Lenz wrote:
> Hi!
> 
> On 05/29/2012 12:48 PM, Peter Colberg wrote:
> >> In general, standardizing such a subgroup might also be useful in
> >>  simulations with fixed particle numbers, e.g. for parallel IO, 
> >> where the particles are not always stored on the same CPU.
> > 
> > Am I correct that one would *always* want to write the particle 
> > identities with parallel IO? Since particles move between 
> > processors, a fixed order (e.g. the initial order) would require
> > each processor to perform a scattered write to the “value” dataset,
> > which is probably quite slow with HDF5. By storing particles
> > according to their current order in memory, each processor could
> > write to a linear region in the dataset.
> 
> That is exactly what I had in mind. A scattered write is not real
> parallel IO, therefore the ids might be of interest.
> 
> Note, however, that this is still not the whole solution, as the number
> of particles per task might vary from timestep to timestep. Parallel IO
> is only efficient when each task knows exactly where in the file to
> write, and to do that, the write size of each task has to be known
> beforehand. Therefore it would be necessary to leave "holes" in the
> trajectory data, as for example undefined positions.
> 
> This probably also explains why the "range" dataset causes trouble for
> parallel IO, as in that case, the chunk to be written or read by each of
> the tasks is not known beforehand.

It must be possible without leaving holes.

Again, this is merely an idea, I have yet to implement it:

At any given timestep, each processor owns a defined set of particles
of varying number, which shall be written to an HDF5 dataset. The task
that needs to be solved is to calculate for each process the offset
from the beginning of the “range” for the current time-step.

This could be done in parallel using a prefix sum [1].

[1] http://en.wikipedia.org/wiki/Prefix_sum

When the calculation of the prefix sum is finished, each process has
its own offset in the dataset, and then writes into a linear region
starting at that offset.

This results in a continuous dataset :-).

I have implemented prefix summing on the GPU. Would it also work with MPI?

Peter