Re: File size limits in textread()?

[Stefan van der Walt]

The attached program should help a bit with speed.  It's just a quick
hack, hasn't been tested too thoroughly.  Can probably be made a lot
quicker -- I am doing millions of cell-array resizes in there.  One
good way to improve its speed would be to estimate the cell array size
beforehand, based on the number of lines in the file first.

Regards
Stefan

On Fri, Jun 10, 2005 at 12:53:57PM +1000, address@hidden wrote:
> G'day all,
> 
> I'm using textread() to read in mixed string and integer data from a space 
> delimited text file. I found it worked okay with one data file, but then 
> hangs when reading another,  larger data file. Hitting ctrl-c brings back the 
> octave prompt.
> 
> Any ideas as to why textread() doesn't like the larger data file? This file 
> has about 6000 rows and 10 columns, 4 of which are strings, the others are 
> integers.
> 
> I'm now using a loop with fgetl() and split(), but it's very slow.
> 
> The data file is generated from a database query. I might use 2 queries, one 
> for integer data and one for string data. That way at least I can use load or 
> dlmread() to quickly pull in the integer data, and then deal with the string 
> data using textread().
> 
> -------
> 
> BTW, the data file is generated from a database query. I had earlier asked 
> about interfacing octave to databases (postgreSQL) as I had problems with 
> some interface code from Dirk Eddelbeuttel. I believe that was because the 
> postgresql version I'm using, 7.2.1, was built using g++ 2.95, i.e. shared 
> libraries for c++ interfacing had unresolved symbols for this reason--lack of 
> agreed ABI. I'll try again at some stage with 7.4, but I have a 5GB database 
> that will have to be converted before I can do that...takes ages.
> 
> Cheers,
> 
> Paul Koufalas
> Adelaide, S. Australia
>

tread.cc
Description: Text document