octave-bug-tracker
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #49630] importdata very slow when read large f


From: Philip Nienhuis
Subject: [Octave-bug-tracker] [bug #49630] importdata very slow when read large files of text containing data
Date: Thu, 17 Nov 2016 12:25:26 +0000 (UTC)
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:49.0) Gecko/20100101 Firefox/49.0

Update of bug #49630 (project octave):

              Item Group:    Matlab Compatibility => Performance            
                  Status:                    None => Confirmed              
                 Release:                   4.0.0 => dev                    
                 Summary: importdata cannot read large files of text
containing data, whereas matlab does => importdata very slow when read large
files of text containing data

    _______________________________________________________

Follow-up Comment #1:

The command works fine in the development version. But I agree that
importdata.m should work a bit more efficient.

A little debug session shows that in L. 168 and up it doesn't read just "a
first few rows" but rather reads all rows.  I waited 5 minutes (no response,
Octave busy reading) and then had a lunch break. After that Octave had
finished reading 217952 labels.
Then, in L.208 & up it again reads & parses the entire file.
I haven't tried beyond L.235 and up, as I noted that the input file is to be
read another time (w. dlmread) & parsed again.

FYI, it is much more efficient to read these files using textscan

>> fid = fopen ("imageLabels.txt")
fid =  3
>> tic; a = textscan (fid, "%s"); toc
Elapsed time is 4.40244 seconds.
>> fclose (fid)
ans = 0
>> more on
>> a{1}
ans =
{
  [1,1] = roofTops
  [2,1] = forest
  [3,1] = forest
:



(Title, Item group & release adapted)

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?49630>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/




reply via email to

[Prev in Thread] Current Thread [Next in Thread]