Re: [igraph] Large graphs with igraph

[Gábor Csárdi]

On Sun, Apr 6, 2014 at 9:43 AM, Bian, Jiang <address@hidden> wrote:

Dear all,

I have quite a few big networks (brain connectivity networks, if you care the context) that I need to analysis. On average, each graph has about 50k to 60k nodes, and about 1 billion edges (or more). So, these are not really sparse networks.

Well, that is an almost complete network. igraph is optimized for sparse networks. It might happen that the best representation for your networks is simply adjacency matrices, and then you can work with matrix operations.

 

Looks like igraph can’t really handle graphs at this scale. e.g., It took over two days to calculate the betweenness centrality (I killed the process, it didn’t finish) on a quad-core machine with 32G ram. I’m running the python binding of igraph, but I doubt it would be too much faster if I change to use the c portion of igraph directly.

Quad core does not matter, because igraph uses one core only. 

igraph can handle these networks, but its algorithms scale differently, and betweenness is quadratic (it is log n times n^2 to be exact). Other algorithms scale better or worse.

I did look into other libraries especially those are built for processing large graphs on a cluster such as graphlab, Spark’s GraphX, Giraph, etc. None of them really has all the algorithms implemented as complete as igraph or NetworkX...

Any suggestions?

Use adjacency matrices, and do your calculations in parallel. 32GB RAM might be an issue if you have weighted graphs, and one copy of your matrix is about 20GB.

Gabor

 

Thanks,

Jiang

----------------------------------------------------------------------
Confidentiality Notice: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.

_______________________________________________
igraph-help mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/igraph-help