[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [igraph] Saving Graphs

From: Matthew Walker
Subject: Re: [igraph] Saving Graphs
Date: Thu, 03 Dec 2009 11:15:32 -0500
User-agent: Thunderbird (X11/20090812)


If you are concerned about the performance of re-reading the data, then yes, I agree, a compressed XML file might not be your best solution.

Have you looked at SQLite [1]?  It is a library that allows you to read and write information to/from a database file using SQL.  This would have no problem with such quantities of data, and the read access time would be very fast.  I do not know the best way to store your data in such a database, but at worst you could store the XML representation. 

There exist Python libraries that interface with SQLite.

[1] http://www.sqlite.org/

Good luck!


Marco wrote:
Hi Matthew,

thank you for your input!

I forgot to mention a little problem that makes me wonder if this
approach (which I thought of) is practicable.

The files (of the graphs) I write (in the number of thousands, as I
said, and totaling various gigabytes for each "species") have then to
be reread and analyzed several times (by another piece of code).

This makes me wonder if the overhead of compressing/decompressing so
much data each time is something I want to deal with: I'd say, having tried
just once, that the amount of time needed to do such
compression/decompression is longer than the actual time for the

Being this true "always", would  it be worth the effort?

Thank you again!


restituiremo il riso ai salici

On Thu, Dec 3, 2009 at 4:10 PM, Matthew Walker
<address@hidden> wrote:
Hi Marco,

I am no igraph nor python expert, but maybe you could try writing your
graphs into one large XML file.  You can easily compress such a file once
you've finished adding graphs.  I would even suspect that python would have
the ability to compress the file on the fly.

I think GraphML (an XML format) would be ideal for this.  You could write
out a graph to a temporary file (using igraph_write_graph_graphml(), or its
python equivalent) and then merge the temporary file into a main multi-graph
file (also in GraphML).  Just how to best do this, I am not sure.  I imagine
python already has strong abilities to manipulate XML files.  If that's not
for you, then there is a useful command-line utility called xmlstar [1] that
could help you do such a merge.

I note that igraph already has the ability to select which graph to read
given such a multi-graph file (see the documentation for the "index"
parameter for igraph_read_graph_graphml()).

I hope that helps,

[1] http://xmlstar.sourceforge.net/



Marco wrote:
Hi all!

In the simulations I write I have to save very often the graph, with
all the attributes.
If I save it using gml or something like this, I end up with many
(say: thousands) of very small files, which are not very practical to
carry around and tend to full the filesystem very quickly.

I am looking for a way to save graphs (even not single graphs, but
rather more graphs in the same file) which is a little more clever.

So far I have been looking at pickling the whole thing, but I find it
rather unsatisfactory.
I have heard of hdf (or something like this). Looks to me that the
overhead to use it is rather big.

Someone has highlights, remarks, advice?

Thank you in advance,


restituiremo il riso ai salici

igraph-help mailing list

igraph-help mailing list


igraph-help mailing list


reply via email to

[Prev in Thread] Current Thread [Next in Thread]