[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Octave-bug-tracker] [bug #30468] dlmread - performance problems

From: Alexander Renz
Subject: [Octave-bug-tracker] [bug #30468] dlmread - performance problems
Date: Sat, 17 Jul 2010 09:53:50 +0000
User-agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv: Gecko/20100625 Firefox/3.6.6


                 Summary: dlmread - performance problems 
                 Project: GNU Octave
            Submitted by: veloci
            Submitted on: Sa 17 Jul 2010 09:53:49 GMT
                Category: None
                Severity: 3 - Normal
              Item Group: None
                  Status: None
             Assigned to: None
         Originator Name: 
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any
                 Release: 3.2.4
        Operating System: Microsoft Windows



The dlmread function seems to have a performace lack, if used for analysing
large data files (number of rows > 100000). The processing time increases
exponetially, compared to the time needed for lesser data sets. 
In my case i've used dlmread for data files with a fixed number of header
rows followed by >300000 rows/ 5 columns separated by tabs. From these i've
read only columns 2-5 starting after the header rows and continuing to the end
of the file. For that i calculate a range vector in advance and allocate
memory in a zero matrix (data = zeros(datarange)). The process is started

data = dlmread(filename,'\t',r); 

I have tried this approach in Octave and Matlab. In Matlab i get results
after a couple of moments for the whole data set. In Octave the process is
started and doesn't seem to end. 
I suspected having a problem with dlmread, so i tried to change the
parameters- separator, range vector. The result was, that dlmread of Octave
works fast for smaller data sets.
Now i'm using a crutch for being able to read the big data sets: I process
the data in smaller parts and combine it piece by piece in the preallocated
data matrix. 



Reply to this item at:


  Nachricht geschickt von/durch Savannah

reply via email to

[Prev in Thread] Current Thread [Next in Thread]