[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
R, HDF5, Swarm output serialization - Part II
From: |
Norberto Eiji Nawa |
Subject: |
R, HDF5, Swarm output serialization - Part II |
Date: |
Mon, 09 Apr 2001 22:14:56 +0900 |
Hello:
A while ago (*), I reported on problems I am having to load several HDF5
files in R, to process data generated in my models. I am not able to
load more than 10 (of 50) 1.5-2MB files without almost hanging the
computer.
I wrote another small program to check whether this was a R-dependent
problem; it is a self-sufficient piece of code that directly calls the
HDF5 library functions to load the 50 files in a row and store the
data in some internal variables. Again, I got the same symptoms; the
program is unable to load more than 6 or 7 files without it starts
swapping badly.
For those who are interested, the files (app, Makefile, sample data)
can be found at http://www.isd.atr.co.jp/~eiji/swarm/loadingHDF5/
Please, let me know if you get the same problems or not.
Since the very beginning, I suspected that the cause of the problem is
actually the way I am serializing data from my model. Each time the
model outputs data, it builds a "compound object", with one field to
each variable that is saved, and pushes it into an HDF5 archiver. For
2000 iterations, there are 2000 objects, each one of them containing 7
to 10 float variables. This was actually a suggestion from Marcus,
which I properly spoiled (**) and made it into a problem to myself :-)
My guess is that it should be working anyway, and now I decided to
blame the computer.
If anyone happens to run these simple test programs fine, please, let
me know, so I can look for the problem in my platform.
Thanks a lot!
Eiji
(*) http://www.santafe.edu/projects/swarm//archive/list-archive.0103/0068.html
(**) http://www.santafe.edu/projects/swarm/archive/list-archive.0009/0142.html
==================================
Swarm-Support is for discussion of the technical details of the day
to day usage of Swarm. For list administration needs (esp.
[un]subscribing), please send a message to <address@hidden>
with "help" in the body of the message.
- R, HDF5, Swarm output serialization - Part II,
Norberto Eiji Nawa <=
- Re: R, HDF5, Swarm output serialization - Part II, Marcus G. Daniels, 2001/04/09
- Re: R, HDF5, Swarm output serialization - Part II, Marcus G. Daniels, 2001/04/09
- Message not available
- Re: R, HDF5, Swarm output serialization - Part II, Marcus G. Daniels, 2001/04/10