[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Reading array slow ( and dlmread bug )
From: |
peter |
Subject: |
Reading array slow ( and dlmread bug ) |
Date: |
Sun, 20 Feb 2005 19:11:35 +0000 |
User-agent: |
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.3) Gecko/20041007 Debian/1.7.3-5 |
I am writing a small utility program to compare
files containing numerical data.
I found that reading the files (with octave) was very slow
compared to perl. The format of the data files is shown
below.
38157 10872 33592 26089
47591 14720 45843 16639
.....
......
55002 29660 45918 52316
36063 56511 31388 36950
I have enclosed a small function to generate
the data (generate.m)
I tried to use "load" and "dlmread". The execution speed
is show below (slightly edited output).
Sun Feb 20 18:40:50 GMT 2005
octave dlmread
ans = 100000 11 <=== incorrect size of array returned by "dlmread"
Sun Feb 20 18:40:56 GMT 2005
Octave load
ans = 100000 10
Sun Feb 20 18:41:17 GMT 2005
perl
Sun Feb 20 18:41:18 GMT 2005
In this case "dlmread" took 6 seconds and "load" took 21
seconds. The perl program took 1 second .....
I think there is a bug in "dlmread" as empty space followed
by "\n" is seen as a zero value. The size for the array is therefore
reported incorrectly.
I think the difference is remarkable, given that
the octave program is essentialy executing "C"
code.
Please find enclosed the code snippets I used.
rdiff.sh
-----------------------
#!/bin/bash
date
echo "octave dlmread"
./rdiff.m
date
echo "Octave load"
./rdiff_1.m
date
echo "perl"
./rdiff.pl
date
rdiff.m
---------------------
#!/usr/bin/octave -q
a = dlmread("new.out"," ") ;
size(a)
rdiff_1.m
--------------------
#!/usr/bin/octave -q
a = load("new.out") ;
size(a)
rdiff.pl.
-------------------------
#!/usr/bin/perl
open(fid,"new.out");
while($record = <fid>) {
@number = split(/[ \t]+/,$record) ;
#print("@number\n");
}
close(fid);
generate.m
-------------------------------
#!/usr/bin/octave -q
1 ;
function save_array(name,x)
fid = fopen(name,"wt") ;
for n=1:rows(x)
fprintf(fid,"%8d ",x(n,:)) ;
fprintf(fid,"\n") ;
endfor
fclose(fid) ;
endfunction
N = 100000 ;
a = round(rand(N,10)*2^16) ;
save_array("new.out",a) ;
-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.
Octave's home on the web: http://www.octave.org
How to fund new projects: http://www.octave.org/funding.html
Subscription information: http://www.octave.org/archive.html
-------------------------------------------------------------
- Reading array slow ( and dlmread bug ),
peter <=