Re: [h5md-user] Variable-size particle groups

[Olaf Lenz]

Hi!

On 05/29/2012 05:43 PM, Peter Colberg wrote:
> > The idea of parallel IO (with a distributed file system) is that a
> >  single file is sliced up over several hard disks, so that each
> > task just has to write to its local disk. Otherwise, parallel IO
> > wouldn't really help, as the bottleneck is the IO speed of the
> > hard disks.
>
> Oh, now I see what you mean with parallel I/O.
>
> I was assuming a different definition of parallel I/O. At SciNet in
> Toronto, they provide a parallel filesystem (GPFS), which allows
> writing data in parallel from thousands of processes to a *single*
> file. I believe this is the scenario that parallel HDF5 implements.

Yes, I also mean writing in parallel to a single file on a parallel file
system. To be able to do that, the single file has to be somehow
distributed over several disks, so that it is actually possible to write
to multiple disks at once. And this is really effective only when the
file has regular sized chunks on each disk.

I must admit, however, that I do not really have much practical
experience with parallel IO, mostly because so far the people I talked
to told me that it is only worth it when the file is split up regularly.
As soon as you start writing the file in a less regular fashion, the
advantage of parallel IO is mostly gone.

Maybe this is not true for modern PFS anymore. If that is so, I would be
happy to learn about it.

Olaf
--
Dr. rer. nat. Olaf Lenz
Institut für Computerphysik, Pfaffenwaldring 27, D-70569 Stuttgart
Phone: +49-711-685-63607

olenz.vcf
Description: Vcard