Re: [igraph] reading large, sparse graph with read.graph

[David Hunkins]

So I am trying both of your suggestions (thanks):

Tamas: I created a test file that's a subset of the very large file  
I'm working with. I had already converted it to Pajek format. The  
read.graph function reads it into memory in about 15 seconds. But it  
is taking a very long time to write out the lgl file (now thirty  
minutes already). The memory footprint doesn't change, and the cpu is  
maxed at 100%. I would think that the more 'compact' format could be  
written out quite fast, so I think something is wrong.
---------

g <- read.graph("email_1m.csv_dg", format="pajek")
write(proc.time(),file="")
write.graph(g, "email_1m_lgl", format="lgl")  <- Hangs here
----------

Gabor, I tried using graph.edgelist to create a graph from a matrix  
that was scanned from a file, and (consistent with the docs), edgelist  
doesn't want any weights; it only accepts two-column input. What did I  
miss?
> el <- scan("test_edgelist", what = list("","",0))
Read 9 records
> el
[[1]]
[1] "231" "231" "234" "235" "236" "237" "238" "232" "235"

[[2]]
[1] "232" "233" "235" "236" "234" "238" "239" "235" "239"

[[3]]
[1] 1 1 1 1 1 1 1 1 1
> dg <- graph.edgelist(el,directed=TRUE)
Error in graph.edgelist(el, directed = TRUE) :
  graph.edgelist expects a matrix with two columns

The graph I eventually want to read in is directed, weighted and has  
millions of edges and vertices. Thanks in advance for your help.

David Hunkins


On Apr 27, 2010, at 1:35 AM, Gábor Csárdi wrote:

> On Tue, Apr 27, 2010 at 10:00 AM, Tamas Nepusz <address@hidden>  
> wrote:
> > Heloo David,
> >
> > > I would have thought that the time it would take read.graph to  
> > > load into memory would be O(|V|+|E|).
> > That might approximately be true for simple edge lists and other  
> > similar formats, but it is definitely not true for Pajek files  
> > because Pajek files are much more complicated to parse. If you need  
> > only the edges and the weights from your file (i.e. there are no  
> > other attributes to keep), I'd try to load the Pajek file first,  
> > then save it in LGL format, which can be read back faster.
>
> Actually, I think the Pajek format should be linear in time, as well.
> But anyway, an easy way to speed up the import is to convert the file
> to a three-column edge list, and then read it in with the scan() R
> function. The conversion itself should be pretty simple with the
> standard Unix text tools, the Pajek format is usually pretty close to
> an edge list. (But not always, the Pajek format has different
> subformats.)
>
> Tell us if you need more help on this. Best,
> Gabor
>
> > > If it helps to answer, I am contemplating going outside R, and  
> > > using the igraph library for Ruby (in alpha). Would graphs load  
> > > significantly faster outside of R?
> > I don't think it would really matter, all the graph IO calls go  
> > down into the C layer of igraph, which is the same for igraph-R and  
> > igraph-Ruby.
> >
> > --
> > Tamas
> >
> >
> >
> > _______________________________________________
> > igraph-help mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/igraph-help
>
>
>
> --
> Gabor Csardi <address@hidden>     UNIL DGM
>
>
> _______________________________________________
> igraph-help mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/igraph-help