[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [igraph] reading large, sparse graph with read.graph
From: |
David Hunkins |
Subject: |
Re: [igraph] reading large, sparse graph with read.graph |
Date: |
Tue, 27 Apr 2010 15:42:17 -0700 |
So I am trying both of your suggestions (thanks):
Tamas: I created a test file that's a subset of the very large file
I'm working with. I had already converted it to Pajek format. The
read.graph function reads it into memory in about 15 seconds. But it
is taking a very long time to write out the lgl file (now thirty
minutes already). The memory footprint doesn't change, and the cpu is
maxed at 100%. I would think that the more 'compact' format could be
written out quite fast, so I think something is wrong.
---------
g <- read.graph("email_1m.csv_dg", format="pajek")
write(proc.time(),file="")
write.graph(g, "email_1m_lgl", format="lgl") <- Hangs here
----------
Gabor, I tried using graph.edgelist to create a graph from a matrix
that was scanned from a file, and (consistent with the docs), edgelist
doesn't want any weights; it only accepts two-column input. What did I
miss?
> el <- scan("test_edgelist", what = list("","",0))
Read 9 records
> el
[[1]]
[1] "231" "231" "234" "235" "236" "237" "238" "232" "235"
[[2]]
[1] "232" "233" "235" "236" "234" "238" "239" "235" "239"
[[3]]
[1] 1 1 1 1 1 1 1 1 1
> dg <- graph.edgelist(el,directed=TRUE)
Error in graph.edgelist(el, directed = TRUE) :
graph.edgelist expects a matrix with two columns
The graph I eventually want to read in is directed, weighted and has
millions of edges and vertices. Thanks in advance for your help.
David Hunkins
On Apr 27, 2010, at 1:35 AM, Gábor Csárdi wrote:
On Tue, Apr 27, 2010 at 10:00 AM, Tamas Nepusz <address@hidden>
wrote:
Heloo David,
I would have thought that the time it would take read.graph to
load into memory would be O(|V|+|E|).
That might approximately be true for simple edge lists and other
similar formats, but it is definitely not true for Pajek files
because Pajek files are much more complicated to parse. If you need
only the edges and the weights from your file (i.e. there are no
other attributes to keep), I'd try to load the Pajek file first,
then save it in LGL format, which can be read back faster.
Actually, I think the Pajek format should be linear in time, as well.
But anyway, an easy way to speed up the import is to convert the file
to a three-column edge list, and then read it in with the scan() R
function. The conversion itself should be pretty simple with the
standard Unix text tools, the Pajek format is usually pretty close to
an edge list. (But not always, the Pajek format has different
subformats.)
Tell us if you need more help on this. Best,
Gabor
If it helps to answer, I am contemplating going outside R, and
using the igraph library for Ruby (in alpha). Would graphs load
significantly faster outside of R?
I don't think it would really matter, all the graph IO calls go
down into the C layer of igraph, which is the same for igraph-R and
igraph-Ruby.
--
Tamas
_______________________________________________
igraph-help mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/igraph-help
--
Gabor Csardi <address@hidden> UNIL DGM
_______________________________________________
igraph-help mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/igraph-help
- [igraph] reading large, sparse graph with read.graph, David Hunkins, 2010/04/26
- Re: [igraph] reading large, sparse graph with read.graph, Tamas Nepusz, 2010/04/27
- Re: [igraph] reading large, sparse graph with read.graph, Gábor Csárdi, 2010/04/27
- Re: [igraph] reading large, sparse graph with read.graph,
David Hunkins <=
- Re: [igraph] reading large, sparse graph with read.graph, Gábor Csárdi, 2010/04/28
- Re: [igraph] reading large, sparse graph with read.graph, Gábor Csárdi, 2010/04/28
- Re: [igraph] reading large, sparse graph with read.graph, David Hunkins, 2010/04/30