[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[igraph] Error pickling (very) large graphs: OSError: [Errno 22] Invalid

From: Nick Eubank
Subject: [igraph] Error pickling (very) large graphs: OSError: [Errno 22] Invalid argument
Date: Tue, 18 Apr 2017 19:25:24 +0000

Hello all,

I'm trying to pickle a very large graph (23 million vertices, 152 million edges, two vertex attributes), but keep getting an `OSError: [Errno 22] Invalid argument` error. However, I think that's erroneous, as if I subsample the graph and save with exact same code I have no problems. Here's the traceback:

Out[8]: 'IGRAPH UN-- 23331862 152099394 -- \n+ attr: name (v), pt (v)'

g.write_pickle(fname='graphs/with_inferred/vz_inferred3_sms{}_voz{}.pkl'.format(sms, voz))
Traceback (most recent call last):

 File "<ipython-input-9-6b5409a79251>", line 1, in <module>
   g.write_pickle(fname='graphs/with_inferred/vz_inferred3_sms{}_voz{}.pkl'.format(sms, voz))

 File "/Users/Nick/anaconda/lib/python3.5/site-packages/igraph/__init__.py", line 1778, in write_pickle
   result=pickle.dump(self, fname, version)

OSError: [Errno 22] Invalid argument


g.write_pickle(fname='graphs/with_inferred/vz_inferred3_sms{}_voz{}.pkl'.format(sms, voz))


The graph takes up about 10gb in memory, and the pickle command expands Python's memory footprint to about 35gb before the exception gets thrown, but I'm on a machine with 80gb ram, so that's not the constraint. 

Any suggestions as to what might be going on / is there a work around for saving? 



reply via email to

[Prev in Thread] Current Thread [Next in Thread]