R, HDF5, Swarm output serialization

[Norberto Eiji Nawa]

Hello Marcus and all:

My Swarm application outputs 10-15 floating point numbers at each
iteration, one run consisting of more or less 1000-3000
iterations. What I am doing now is storing the data in a simple
structure with one field for each of the data streams, indexing it
with the iteration number and calling [outputDataHDF5Archiver
putShallow: object:];

When the run is completed, I have an HDF5 file of about
1.5MB. I can usually retrieve the data within R, and do all the
processing I want. The problem is when I want to manipulate many of
these HDF5 files in R to compare different runs, or do some
statistics.

I am using R-1.2.2 (which performs garbage collection, dynamically
allocating and releasing memory). But if I try to load say 50 of these
1.5 MB files, by the file # 15 my Linux box runs out of memory (to the
point of hanging up). That happens even if I don't keep the loaded
objects in R (load the file, clean up the memory, load the next, clean
up the memory, etc.).

I thought another alternative would be to make a single HDF5 data blob
with the data from all the run, indexing the data structures with runs
and the iterations. In that case, the output of 50 runs makes a +-50MB
file, which also eats up the whole memory I have when I try to load it
(256MB RAM + 256 swap).

Are there any other alternatives to that? Or this is a HDF5 + R
limitation? Also, I would like to hear from other people using Swarm +
HDF5 + R, and people not using R but another free equivalent
software. One of the reasons I moved to R (from Matlab) is because it
is will integrated with HDF5 (Thanks, Marcus!) and therefore Swarm,
but if I can't do the above I will look for an alternative. 

I left 50 HDF5 samples (HDF5samples.tar.gz), the single big data file
(ArchiverHDF5.hdf.gz) and the trivial R script to load in a loop the
files (testHDF5load.R) under http://www.isd.atr.co.jp/~eiji/swarm for
those who are interested.

Thanks a lot in advance,

Eiji

                  ==================================
   Swarm-Support is for discussion of the technical details of the day
   to day usage of Swarm.  For list administration needs (esp.
   [un]subscribing), please send a message to <address@hidden>
   with "help" in the body of the message.