Re: [igraph] Saving Graphs

[Matthew Walker]

Marco,

If you are concerned about the performance of re-reading the data, then yes, I agree, a compressed XML file might not be your best solution.

Have you looked at SQLite [1]?  It is a library that allows you to read and write information to/from a database file using SQL.  This would have no problem with such quantities of data, and the read access time would be very fast.  I do not know the best way to store your data in such a database, but at worst you could store the XML representation. 

There exist Python libraries that interface with SQLite.

[1] http://www.sqlite.org/

Good luck!

Matthew

Marco wrote:

Hi Matthew,

thank you for your input!

I forgot to mention a little problem that makes me wonder if this
approach (which I thought of) is practicable.

The files (of the graphs) I write (in the number of thousands, as I
said, and totaling various gigabytes for each "species") have then to
be reread and analyzed several times (by another piece of code).

This makes me wonder if the overhead of compressing/decompressing so
much data each time is something I want to deal with: I'd say, having tried
just once, that the amount of time needed to do such
compression/decompression is longer than the actual time for the
analysis.

Being this true "always", would  it be worth the effort?

Thank you again!

marco

--
restituiremo il riso ai salici



On Thu, Dec 3, 2009 at 4:10 PM, Matthew Walker
<address@hidden> wrote:
  

Hi Marco,

I am no igraph nor python expert, but maybe you could try writing your
graphs into one large XML file.  You can easily compress such a file once
you've finished adding graphs.  I would even suspect that python would have
the ability to compress the file on the fly.

I think GraphML (an XML format) would be ideal for this.  You could write
out a graph to a temporary file (using igraph_write_graph_graphml(), or its
python equivalent) and then merge the temporary file into a main multi-graph
file (also in GraphML).  Just how to best do this, I am not sure.  I imagine
python already has strong abilities to manipulate XML files.  If that's not
for you, then there is a useful command-line utility called xmlstar [1] that
could help you do such a merge.

I note that igraph already has the ability to select which graph to read
given such a multi-graph file (see the documentation for the "index"
parameter for igraph_read_graph_graphml()).

I hope that helps,

[1] http://xmlstar.sourceforge.net/

Cheers,

Matthew

Marco wrote:
    

Hi all!

In the simulations I write I have to save very often the graph, with
all the attributes.
If I save it using gml or something like this, I end up with many
(say: thousands) of very small files, which are not very practical to
carry around and tend to full the filesystem very quickly.

I am looking for a way to save graphs (even not single graphs, but
rather more graphs in the same file) which is a little more clever.

So far I have been looking at pickling the whole thing, but I find it
rather unsatisfactory.
I have heard of hdf (or something like this). Looks to me that the
overhead to use it is rather big.

Someone has highlights, remarks, advice?

Thank you in advance,

marco

--
restituiremo il riso ai salici


_______________________________________________
igraph-help mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/igraph-help


      

_______________________________________________
igraph-help mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/igraph-help

    

_______________________________________________
igraph-help mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/igraph-help