Getting data out and saving states!

[Sven Thommesen]

To Barry, Glen, Manor et al:

As a side comment to our discussion on Random.h, here are some thoughts
on this issue of saving states and -printSelf. They are meant as input
to your internal discussion on how to implement object saving, as well
as info on what one user is doing ... 

(To the rest of the list: Please discard if not of interest!)

I've felt a need to have a general way of getting data out of my
simulation, so I have implemented my own protocols in this regard.
(Still experimental, of course -- more after I get unstuck from my
mysterious core dumps. Any thoughts on the gdb trace I posted, Glen?)

Basically, for getting data (results, state variables) out there's 3
questions to be answered for every object:

WHAT data should be supplied, HOW should it be supplied, and WHERE
should it end up?

a) the HOW: 

I can imagine several formats: 

(i) the multiline text format now supplied by ObjectSaver (
<variable>=<value> );

(ii) a format where sprintf is used to create a single text line;

(iii) and finally some kind of binary format. 

I've implemented the first two myself for now; will do binary if the
need arises.

b) the WHERE: 

(i) each object supplies output to a disk file whose name is of the form
"DATA.objectName.Run#.Period#. (I.e. one file per object per time
period.) How often is governed by a global parameter
'dataCollectionFrequency'.

(ii) each object supplies output to disk as above, but to a disk file
which the object keeps open for the duration of the run:
"DATA.objectName.Run#.LOG".

(iii) each object supplies a string of data each period to a centralized
DataLoggerAgent, who writes all data to one or more disk files of its
own. The message for every swarm, space and agent to do this is put into
a Schedule by the BatchSwarm or ObserverSwarm at start.

(iv) as above, but instead of the objects pushing data on the
DataLogger, the DataLogger solicits data from each object in turn;

(v) each object writes stuff to stdout (-printSelf).

Not knowing which method will be most useful and least taxing on
resources, I've implemented all ...

c) the WHAT: ah, there's the rub. For purposes of outputting results, of
course, we just supply either data from averagers, or the raw data from
every agent and space. A global parameter 'amountOfDataOuput' governs
how much is supplied.

Now, for purposes of debugging and saving the full object state, I've
identified the following classes of data (perhaps someone can think of
more -- please comment), with interspersed comments on the need for
saving:

(i) parameter variables set on startup (via Drone, from the command
line, etc.):
   a) unique to the agent;
   b) common to his species;
   c) global to the simulation;

[ i(b) and i(c) need not be saved by the agent; they can be more
economically saved by the higher-up swarm to which they belong. i(a)
must be saved by the agent, as those parameters are unique to the given
run! ]

(ii) fixed working variables (state variables) set in method -initState
that are derived from the parameters in a repeatable way;

[no need to save these; they can be re-generated when needed]

(iii) fixed working variables (state variables) set by the agent in the
-initState method, based on hardwired constants;

[again, no need to save these]

(iv) other state variables; presumably this is the set from which 'data
output' is derived;

[must be saved]

(v) pointers to static and global objects such as the World grid, the
Heatspace etc.

[no need to save these; they must be regenerated as the simulation is
reloaded]

(vi) dynamic or local pointers: pointers to transient objects, or
pointers to other agents signifying transient relationships;

[these must be saved -- but memory pointers are useless. My preliminary
move is to give every agent an id tag (agentNumber or agentName) which
can be saved instead.]

I expect what can get tricky is the order in which objects are reloaded
from disk .. 
 
For now, I have implemented the following global parameters:

-dataSamplingFrequency governs how often data are output;
[note: this can be different from displayFrequency, which makes the use
of EZGraph impossible ...]

-levelOfDataDetail: amount of data supplied ((iv) above);

-levelOfStateDetail: amount of state data supplied ( (i) through (vi)
above);

-modeOfDataOutput: governs the HOW and WHERE above (implemented as
different methods).

[And also a -haltingFrequency, which governs how often the display stops
and beeps; easy to do and very useful.]

Comments welcome!

Sven Thommesen
School of Human Sciences
Auburn University